Ingesting Partitioned Parquet data from pipeline

SingleStore Development Connectors and Pipelines
,
#1

Pipeline is not reading data from partitioned Parquet files

#2

Hi kshastry, welcome to the SingleStore forums!

We’d need more information to try to help you with this. For starters, check that the pipeline is created, started, and that the system can read the data from the data source, and that the data file schemas and target table schemas match up.

#3

Hi hanson,

The sample parquet structure is

Data.parquet/DATE=2020-12-25/part-00000-8e301b7c-9739-481f-8794-f4dbc2a8b008.c000.snappy.parquet

My table has column DATE. But since the parquet is partitioned on DATE, it is not present in the parquet file. And hence the data is not ingested.
The pipeline is created. The pipeline can read the data from the source.