AEP : Automate Ingestion of multiple files (daily, faile name has date and file number) based on a pattern

[AbhiRL]

1. We have a source feed which splits huge files into multiple smaller files on a daily basis. I'm trying to figure out how can we automate a schedule to pick up all files for that day (file name has Date) from the source location and ingest into AEP.

2. Our SFTP location has files of different category and how to pick files based on a pattern (or a wild card)  that are meant to be loaded into a dataset.

Sample files:

Testfile_20230701_1.csv

Testfile_20230701_2.csv

Anotherfile_20230701_1.csv

Anotherfile_20230701_2.csv

[ssj · Accepted solution]

when the data flow is scheduled on certain frequency, then AEP will pick up all the new files and will ingest. and on this scenario even the larger file is split into multiple small files, all those files will be picked up based on files timestamp. the date in the filename is not much relevant in this case.

would recommend to place different category files in different folder and create data flow to ingest into a dataset pointing to a folder. 

Regards

Samuel

[AbhiRL]

Thank you Samuel. Will give it a try .