Adobe's best practice to import previous data for disabled SFTP source setup | Community
Skip to main content
M
Md__AjazAn
Level 3
October 26, 2023
Solved

Adobe's best practice to import previous data for disabled SFTP source setup

  • October 26, 2023
  • 3 replies
  • 1518 views

Hi,

Recently, I found that one of my dataflow which manages the SFTP source setup has been disabled by one of the marketing team member. It was originally setup to import daily. I want to import the missing files which came in the past (while the dataflow was disabled) and restart the daily import. Whats the best practice to handle/fix this scenario?

 

I am confused with On-demand Ingestion which takes Ingestion start time, date range start time and date range end time. Screenshot attached. Can this option be used where I re-enable the dataflow and start on demand Ingestion and change the starting date to be the date that the dataflow was disabled?

 

Other viable option can be to create a new dataflow and enable the backfill.

 

Thoughts and guidance please. Thanks!

 

 

 

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by arijitg

Hi @md__ajazan 

 

To answer your questions -->

 

"I want to import the missing files which came in the past (while the dataflow was disabled) and restart the daily import. Whats the best practice to handle/fix this scenario?" You can enable the dataflow again and all the missing runs till date will start running one by one. So for example you disabled after 24th October run and if you enable today on 27th October you will see 3 immediate runs for 25th, 26th & 27th and it will process files placed on those days in sequence.

 

"I am confused with On-demand Ingestion which takes Ingestion start time, date range start time and date range end time. Screenshot attached. Which use case this can be used?" This can be used when there is a defined timeline for example you decide to move from SFTP to S3 next year in that case rather than depending on someone to manually disable the dataflow you can schedule an end date of 31st December so system will take care of it's own. 

 

 

3 replies

NimashaJain
Adobe Employee
NimashaJainAdobe Employee
Adobe Employee
October 26, 2023

@arijitg @birthariya 

 
H
hariprasathg
Level 2
October 26, 2023

Enable the "Back Fill" option in your dataflow, which will ingest data from the date it was ingested last. Keep in mind that all the missing file should be available in your source SFTP location.

 

hope this will help!

 

M
Md__AjazAnAuthor
Level 3
October 26, 2023

Thanks for the response. I am confused with On-demand Ingestion which takes Ingestion start time, date range start time and date range end time. Screenshot attached. Which use case this can be used?

arijitg
Adobe Employee
arijitgAdobe EmployeeAccepted solution
Adobe Employee
October 27, 2023

Hi @md__ajazan 

 

To answer your questions -->

 

"I want to import the missing files which came in the past (while the dataflow was disabled) and restart the daily import. Whats the best practice to handle/fix this scenario?" You can enable the dataflow again and all the missing runs till date will start running one by one. So for example you disabled after 24th October run and if you enable today on 27th October you will see 3 immediate runs for 25th, 26th & 27th and it will process files placed on those days in sequence.

 

"I am confused with On-demand Ingestion which takes Ingestion start time, date range start time and date range end time. Screenshot attached. Which use case this can be used?" This can be used when there is a defined timeline for example you decide to move from SFTP to S3 next year in that case rather than depending on someone to manually disable the dataflow you can schedule an end date of 31st December so system will take care of it's own. 

 

 

M
Md__AjazAnAuthor
Level 3
October 27, 2023

Hi @arijitg ,

Thanks for the detailed clarification. 

So to conclude, on-demand Ingestion which takes Ingestion start time, date range start time and date range end time is not for the use case which I mentioned originally. 

Thanks!

    Terms & ConditionsAccessibility statement

    Loading footer...