Know what the current file name the pipeline is currently processing

Help Connectors and Pipelines
1

Lets say I have two files in an s3 bucket/folder: file a and file b with columns name, age
I make a s3 pipeline

I want to put them into a memsql table with columns
name, age, file_name
Issac 5 file a
Austin 10 file b

Is there a way to know what the file_name is during the pipeline process or when I create a pipeline?

2

Welcome to the SingleStore Forum! When you get a moment, introduce yourself in the “Introductions” category. As for your question, the information_schema.PIPELINES_FILES table will give you the file name and the loading status for each pipeline. Here’s the docs page on that table:

3

Thanks for the reply! I actually found out what I was looking for, I probably should’ve worded my question better

I wanted to know, when a pipeline is executing, what is the current file being processed and something like does the trick

set source_file = pipeline_source_file();
2 Likes