Pipeline using Confluent schema registry (avro format schema)

Help
1

Hello.
I’m working on a test to insert a Kafka message to a singlestore record, using confluent schema registry.
I used the following pipeline statement to send records to a simple test table with 2 columns of (id, name).

CREATE PIPELINE kafka
AS LOAD DATA KAFKA ‘bootstrapserver:9092/topicname’
SKIP DUPLICATE KEY ERRORS
INTO TABLE ppline
FORMAT AVRO
SCHEMA REGISTRY “schemaregistry:8081”
(ppline.id ← %::id, ppline.name ← %::name);

I had no problem creating and starting the pipeline, but the data is all coming in as NULL.
I was wondering if I should specify the topic name of the schema-value in the CONFIG value or make any other additional settings.

Thanks & Regards.

2

Hello Iris,

can you try to fix the syntax around this line:

CREATE OR REPLACE PIPELINE test
AS LOAD DATA KAFKA ‘’
CONFIG ‘{
“schema.registry.ssl.ca.location”: “<location”,
“schema.registry.username”: “schema_registry_name”,
“sasl.username”: “username”,
“sasl.mechanism”: “sasl meachnism”,
“security.protocol”: “SASL_SSL”
}’
CREDENTIALS ‘{
“ssl.key.password”: “password”,
“schema.registry.password”: “<schema_registry_password>”,
“sasl.password”: “”
}’
DISABLE OUT_OF_ORDER OPTIMIZATION
INTO table <Table_Name>
FORMAT AVRO
SCHEMA REGISTRY ‘Schema_registry_endpoint’
(

)

1 Like
3

Thank you so much, Kumar.