1. We have a source feed which splits huge files into multiple smaller files on a daily basis. I'm trying to figure out how can we automate a schedule to pick up all files for that day (file name has Date) from the source location and ingest into AEP.
2. Our SFTP location has files of different category and how to pick files based on a pattern (or a wild card) that are meant to be loaded into a dataset.
Sample files:
Testfile_20230701_1.csv
Testfile_20230701_2.csv
Anotherfile_20230701_1.csv
Anotherfile_20230701_2.csv
Solved! Go to Solution.
when the data flow is scheduled on certain frequency, then AEP will pick up all the new files and will ingest. and on this scenario even the larger file is split into multiple small files, all those files will be picked up based on files timestamp. the date in the filename is not much relevant in this case.
would recommend to place different category files in different folder and create data flow to ingest into a dataset pointing to a folder.
Regards
Samuel
when the data flow is scheduled on certain frequency, then AEP will pick up all the new files and will ingest. and on this scenario even the larger file is split into multiple small files, all those files will be picked up based on files timestamp. the date in the filename is not much relevant in this case.
would recommend to place different category files in different folder and create data flow to ingest into a dataset pointing to a folder.
Regards
Samuel
Thank you Samuel. Will give it a try .
Views
Likes
Replies
Views
Likes
Replies