Expand my Community achievements bar.

SOLVED

AEP : Automate Ingestion of multiple files (daily, faile name has date and file number) based on a pattern

Avatar

Level 1

1. We have a source feed which splits huge files into multiple smaller files on a daily basis. I'm trying to figure out how can we automate a schedule to pick up all files for that day (file name has Date) from the source location and ingest into AEP.

 

2. Our SFTP location has files of different category and how to pick files based on a pattern (or a wild card)  that are meant to be loaded into a dataset.

 

Sample files:

Testfile_20230701_1.csv

Testfile_20230701_2.csv

Anotherfile_20230701_1.csv

Anotherfile_20230701_2.csv

1 Accepted Solution

Avatar

Correct answer by
Employee
Employee

when the data flow is scheduled on certain frequency, then AEP will pick up all the new files and will ingest. and on this scenario even the larger file is split into multiple small files, all those files will be picked up based on files timestamp. the date in the filename is not much relevant in this case.

 

would recommend to place different category files in different folder and create data flow to ingest into a dataset pointing to a folder. 

 

Regards

Samuel

 

View solution in original post

2 Replies

Avatar

Correct answer by
Employee
Employee

when the data flow is scheduled on certain frequency, then AEP will pick up all the new files and will ingest. and on this scenario even the larger file is split into multiple small files, all those files will be picked up based on files timestamp. the date in the filename is not much relevant in this case.

 

would recommend to place different category files in different folder and create data flow to ingest into a dataset pointing to a folder. 

 

Regards

Samuel