Adobe's best practice to import previous data for disabled SFTP source setup

[mdajaz]

Hi,

Recently, I found that one of my dataflow which manages the SFTP source setup has been disabled by one of the marketing team member. It was originally setup to import daily. I want to import the missing files which came in the past (while the dataflow was disabled) and restart the daily import. Whats the best practice to handle/fix this scenario?

I am confused with On-demand Ingestion which takes Ingestion start time, date range start time and date range end time. Screenshot attached. Can this option be used where I re-enable the dataflow and start on demand Ingestion and change the starting date to be the date that the dataflow was disabled?

Other viable option can be to create a new dataflow and enable the backfill.

Thoughts and guidance please. Thanks!

[NimashaJain]

@arijitg @birthariyamayank 

[hariprasathg]

Enable the "Back Fill" option in your dataflow, which will ingest data from the date it was ingested last. Keep in mind that all the missing file should be available in your source SFTP location.

hope this will help!

[mdajaz]

Thanks for the response. I am confused with On-demand Ingestion which takes Ingestion start time, date range start time and date range end time. Screenshot attached. Which use case this can be used?

[arijitg · Accepted solution]

Hi @mdajaz 

To answer your questions -->

"I want to import the missing files which came in the past (while the dataflow was disabled) and restart the daily import. Whats the best practice to handle/fix this scenario?" You can enable the dataflow again and all the missing runs till date will start running one by one. So for example you disabled after 24th October run and if you enable today on 27th October you will see 3 immediate runs for 25th, 26th & 27th and it will process files placed on those days in sequence.

"I am confused with On-demand Ingestion which takes Ingestion start time, date range start time and date range end time. Screenshot attached. Which use case this can be used?" This can be used when there is a defined timeline for example you decide to move from SFTP to S3 next year in that case rather than depending on someone to manually disable the dataflow you can schedule an end date of 31st December so system will take care of it's own. 

[mdajaz]

Hi @arijitg ,

Thanks for the detailed clarification. 

So to conclude, on-demand Ingestion which takes Ingestion start time, date range start time and date range end time is not for the use case which I mentioned originally. 

Thanks!