Unable to digest messages in Kafka Pipeline

Help Connectors and Pipelines
1

Hi Team,

I am unable to see messages digested from Kafka into our tables through Pipeline Procedures…

Observations done:

  1. The topic used contains huge data.(Pipeline wokring fine when data is less in TOPIC)
  2. Used default Batch Interval.
  3. Batch State is shown as INPROGRESS, always.
  4. Some times for huge data during the first run of the pipeline - it results in data population in destination table, but in second run on the same topic(same pipeline), Batch state is in INPROGRESS and always ended up in no data. - It is in consistent…

Is there any thing that i am missing here, requesting your help as i am nascent to Memsql.

1 Like
2

Hello @abdulmeeravali, thanks for trying out Kafka pipelines!

Can you confirm some details about your use case:

  • Did create pipeline work without an error?
  • Can you provide the create pipeline statement?
  • Can you run TEST PIPELINE <pipeline_name> and show us the output?
  • Can you provide the rows of information_schema.PIPELINES_BATCHES for the relevant pipeline?