Hello, I am wondering if there is a setting in the SFTP source connector to enable only net new records to be ingested. Im noticing that every time the data flows runs, it ingests the old records as well as any new records, which will inevitably lead to millions of records to be ingested over the long run. Is this a setting in RTCDP or the source (SFMC).
It is overwriting the current data in the data extension and then being extracted to the same file on the SFTP location.
Hi @ryanmoravick,
In the process of overwriting the data, the last modified time of the file changes and the system considers it as a new file.
The system is designed in such a way that it checks the last execution time and any file placed after its last execution time will be picked.
To overcome this problem, limit your export to run once daily.
If you cannot limit the writing process, write the file to a staging folder during the entire day and copy the file to the scheduled folder once daily.
Regards,
Kumar Saurabh
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
