LINK to KAFKA when KAFKA in Confluent Cloud

There exists cluster API key and API secret.
Which form can have LINK ?

1 Like

Please elaborate a bit. I’m not sure I understand the question.

1 Like

Hi @hanson, sure.
I mean CREATE LINK.
In examples with KAFKA credentials in docs missed a security protocol with key/secret and I not found way to do it right.
Thanks!

1 Like

Hi @ira,

Are you referring to Confluent Cluster Linking?

1 Like

Hi @jprice , if I understood right there talks about relation between different Kafka clusters, but I need access to Kafka cluster on Confluent Cloud from SingleStore, thanks.

1 Like

Problem that we not found yet way to receive pem file from Confluent Cloud.
I receive there code snippet with security.protocol=SASL_SSL, e.g.:

bootstrap.servers=pkc-ymrq7.us-east-2.aws.confluent.cloud:9092
security.protocol=SASL_SSL
sasl.mechanisms=PLAIN
sasl.username={{ CLUSTER_API_KEY }}
sasl.password={{ CLUSTER_API_SECRET }}

But when I try to use it in LINK (or PIPELINE) with security.protocol=SASL_SSL or SASL_PLAINTEXT, I receive next:
Error Code: 1933. Cannot get source metadata for pipeline. Could not fetch Kafka metadata; are the Kafka brokers reachable from the Master Aggregator?
ssl.ca.location is missing. Kafka error Local: Broker transport failure.

My code:
CREATE LINK DATA_KAFKA AS KAFKA
CREDENTIALS ‘{“sasl.password”:“my pwd”}’
CONFIG ‘{“security.protocol”:“sasl_ssl”,“sasl.mechanism”:“PLAIN”,“sasl.username”:“my uname”}’
DESCRIPTION ‘blablabla’;

OK

CREATE PIPELINE kafka_test_pp AS LOAD DATA
LINK DATA_KAFKA ‘my_cluster_id.us-east-2.aws.confluent.cloud:9092/my_topic’
INTO TABLE kafka_test;

ERROR 1933 (see above)

Thanks!

1 Like

Can you please share the details of your set up and the code (Kafka and SingleStore) that you’re using with any identifiable pieces removed for security?

1 Like

Sorry, @jprice, see above edited comment.

1 Like

Hi @ira, thanks for trying out SingleStore pipeline with Confluent Cloud.

Are you using them in a SingleStore Managed Service cluster? Or are you running the cluster yourself?

Confluent Kafka uses SSL encryption, so it presents a server certificate that needs to be validated with a certificate authority. On the managed service vms, the location for all top-level CA bundles is /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem.

This means that you should be able to specify that path in your LINK DDL and then create the pipeline as you did.

CREATE LINK DATA_KAFKA AS KAFKA
CREDENTIALS ‘{“sasl.password”:“my pwd”}’
CONFIG ‘{“security.protocol”:“sasl_ssl”,“sasl.mechanism”:“PLAIN”,“sasl.username”:“my uname”,“ssl.ca.location”: “/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem”}’
DESCRIPTION ‘blablabla’;

Hi @m_k,
Thanks,
We running the cluster on premise.
“Confluent Kafka uses SSL encryption, so it presents a server certificate” - way to receive a server certificate not found in Confluent Cloud ((

By Confluent Cloud documentation we can connect to cluster having on client side (on SingleStore master server) the config.properties file with next properties:
bootstrap.servers=my_cluster.us-east-2.aws.confluent.cloud:9092
ssl.endpoint.identification.algorithm=https
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="{{ CLUSTER_API_KEY }}" password="{{ CLUSTER_API_SECRET }}";

It not asks server sertificate, but CREATE LINK talks that params ssl.endpoint.identification.algorithm and sasl.jaas.config are wrong for config

The Confluent Cloud broker will present the certificate to the client when it attempts an SSL connection, so there’s no need to download anything from the portal.

The jaas configs usually mean that those are properties for a Java client. SingleStore on the other hand is using the native C++ librdkafka client. So, the most appropriate set of properties should look something like the following. Although, you currently do need to add the ssl.ca.location as well.

# Kafka
bootstrap.servers=pkc-4nym6.us-east-1.aws.confluent.cloud:9092
security.protocol=SASL_SSL
sasl.mechanisms=PLAIN
sasl.username={{ CLUSTER_API_KEY }}
sasl.password={{ CLUSTER_API_SECRET }}

@m_k , thanks, and what is a ssl.ca.location for on premise SingleStore installation ?

We found single certificates location on our server: /etc/ssl/certs/ with single file .crt, then we converted this file to ca-certificates.pem.
So, we use ssl.ca.location /etc/ssl/certs/ca-certificates.pem - and receives next error:
ERROR 1933 ER_EXTRACTOR_EXTRACTOR_GET_LATEST_OFFSETS: Cannot get source metadata for pipeline. Could not fetch Kafka metadata; are the Kafka brokers reachable from the Master Aggregator? Is ssl.ca.location correct (tried ‘/etc/ssl/certs/ca-certificates.pem’)? Kafka error Local: Broker transport failure

Please help

Hi Again,
Ok, we found strange performance:

  1. If Kafka cluster was created via Confluent Cloud site then we cannot connect to this cluster even from python client (Auth error).
  2. If Kafka cluster created via AWS marketplace we can connect to the cluster from python client.
    But we still cannot connect from Singlestore pipeline (same result as in prev comment). Our Singlestore dev cluster installed on AWS.

Can somebody comment it ?

On-prem location of the CA bundle will be dependent on your Linux distribution. You should check that /etc/ssl/certs/ca-certificates.pem file is readable by the user running the singlestore process (might be memsqld) ? Also, it needs to be present on all nodes in the cluster, in the same consistent location.

You have another option to do some debugging. You can turn on the global variable pipelines_extractor_debug_logging to ON, and try to create the pipeline again, and after an error run SHOW WARNINGS;. That generally includes the librdkafka log, which should have further clues if something went wrong with SSL or SASL layer.

1 Like

@m_k, thank you very much, problem was in access to CA bundle

1 Like