Pipeline is not reading data from partitioned Parquet files
Hi kshastry, welcome to the SingleStore forums!
We’d need more information to try to help you with this. For starters, check that the pipeline is created, started, and that the system can read the data from the data source, and that the data file schemas and target table schemas match up.
Hi hanson,
The sample parquet structure is
Data.parquet/DATE=2020-12-25/part-00000-8e301b7c-9739-481f-8794-f4dbc2a8b008.c000.snappy.parquet
My table has column DATE. But since the parquet is partitioned on DATE, it is not present in the parquet file. And hence the data is not ingested.
The pipeline is created. The pipeline can read the data from the source.