Processing CDC events from KAFKA to SingleStore

Help Connectors and Pipelines
1

Dear Team,
We have requirement to load the CDC events from source to KAFKA(it can be using GG/HVR etc) and from KAFKA to SingleStore.
Please could you confirm,

  1. How to process the CDC events from KAFKA topics to SingleStore? Can we do using SingleStore Pipelines?
  2. How the updates applied on SingleStore if a records already available on SingelStore?
  3. Can we apply these changes from one SingleStore data base to other SingleStore data base and how?
  4. Please suggest different available approach to apply changes from one data base to other data base and what are the complexities involved.
  5. It would be great if you provide usecase(with relevant screenshots) to process CDC events from KAFKA to SingleStore.

Thanks,
Naga S
+966-558036261

2

I can answer on #1 and #2:

1 Like
3
  1. Can we apply these changes from one SingleStore data base to other SingleStore data base and how?

Currently to apply changes from one SingleStore database to another SingleStore database you can use SingleStore Replicate tool and use of delta snapshots for the same. Details here. This will require license for using the delta snapshot capability of the tool. However, this will require a timestamp based column to be used in the database.

An alternative would be to select SingleStore data into S3 in CSV format and then using pipelines to move data back into the other SingleStore.

  1. Please suggest different available approach to apply changes from one data base to other data base and what are the complexities involved.

Can you please clarify if the 2 database are same or different. If same you can refer to the #3 above replied by Arnaud. If they are different then you can do it by yourself, using tools such as HVR or Arcion (Webinar Recap: Migrating Oracle and MySQL Databases With SingleStoreDB & Arcion)

Regards,
Manish Kumar

1 Like