JSON parse errors in pipeline

Help Connectors and Pipelines
1

Hi,
We’re ingesting JSON formatted data from kafka with a pipeline.
Is there a way to protect from parsing errors? A malformed message causes

start pipeline sm_filtering_accesses_pipeline foreground
ERROR 2328 ER_REPLAY_CATCHING_UP: Leaf Error (mem2-worker03:3306): Leaf Error (mem2-worker03:3306): JSON syntax error at byte 35 of object 1. Error: Missing a colon after a name of object member. Contents just before that byte: {"6c59b0724fdb450699066e1eaaf80b86".

I have tried changing

set global pipelines_stop_on_error=OFF;

but the result is the same.
We’re using SingleStore 7.6.7 and have no schema validator to take care of this.

2

Hello @smargareto
Thank you for reaching out.

Regarding the error you are referring to, we do not have a way to skip individual records of malformed JSON. However you can have an entire batch of records being skipped in case of error. Please note: one risk associated with it will be some correctly formed JSON will also be skipped.

@@pipelines_stop_on_error = off allows them to skip it on a per-batch basis for normal background pipelines

We are currently investigating how we can fix the issue and happy to share the plan for the new feature once we have it ready.

1 Like