Kafka pipeline - accessing message metadata (partitions, offset)

Hey, when using kafka pipeline into stored procedure, is there a way to get each message partition or offset, and insert it into the target table?


Hi yarden! :wave: Thanks for your question. Would you be running on our managed service or self-hosted and your version number please?

Hey Maria, Thanks for responding,
self hosted,
version 7.5.12

can upgrade if needed for this feature

Hi Maria, same question self hosted, version 7.5.12

Hi yarden & will.smith!!

All metadata about pipelines, including partition & offset information, can be found in the following information schema tables.

For this particular use case:

I hope this helps. Would love to hear how this works out for ya’ll! :smile_cat: