Help shape the future of AI assistance by participating in this quick card sorting activity. Your input will help create a more effective system that better serves your needs and those of your colleagues.
SOLVED
Redundant data in dataset.
1 Accepted Solution
Correct answer by
Hi @arpan-garg, that says the reason for duplication.
Since the file is getting modified, it loads the file again and ingests the records since you cannot filter based on a field. See the note below:
The best way to handle this is as follows:
- Select the folder instead of the file
- Select the backfill option to load the existing files from the folder; otherwise, skip.
- Add a new file with the increment changes in the folder
- The job will automatically pick the new files (based on the file modification timestamp and the job's last run time).
Hi @arpan-garg, quick questions :
- In your data selection step, did you select a particular file or folder?
- How is the delta data fed into S3 - Does a new file in a folder or an existing file gets updated?
- What's the value of backfill?
Here's an important thing to look for - https://experienceleague.adobe.com/docs/experience-platform/sources/ui-tutorials/dataflow/cloud-stor...
Thanks,
Chetanya
Correct answer by
Hi @arpan-garg, that says the reason for duplication.
Since the file is getting modified, it loads the file again and ingests the records since you cannot filter based on a field. See the note below:
The best way to handle this is as follows:
- Select the folder instead of the file
- Select the backfill option to load the existing files from the folder; otherwise, skip.
- Add a new file with the increment changes in the folder
- The job will automatically pick the new files (based on the file modification timestamp and the job's last run time).
