We have JSON messages that have size limit of 800 KB, while processing this through stored procedure, the message when retrieved from topic looks truncated at 65 KB limit.
When we do a test pipeline, the message is truncated at that moment and hence throw error when we process in proc. The message is getting truncated at 65 KB limit and it does not have any special characters so as to break at that location.
This is what I see in Singlestore Documentation for JSON message. Could you please confirm if 65KB is the limit for JSON messages.
Unicode Support
JSON strings can contain any of the 65536 Unicode characters in the Basic Multilingual Plane, either directly as themselves or escaped with JSON’s \uABCD notation. JSON normalization converts all such ASCII escape sequences into UTF-8 byte sequences, except for those characters that must remain \u-encoded in order to keep the string valid JSON.