Expand my Community achievements bar.

Check out the 3rd Edition of the AEP Community Lens for all the latest releases, resources, and Community updates
SOLVED

Redundant data in dataset.

Avatar

Level 4

Hi,

 

I am ingesting some data from Amazon S3 bucket every 15 mins and i noticed that the data which is already ingested is getting ingested again and again as a separate row in the dataset.

 

Is there a way to make sure that data already ingested is not ingested again? 

 

1 Accepted Solution

Avatar

Correct answer by
Community Advisor

Hi @arpang16406580, that says the reason for duplication. 

 

Since the file is getting modified, it loads the file again and ingests the records since you cannot filter based on a field. See the note below:

ChetanyaJain_0-1681951913435.png

 

The best way to handle this is as follows:

  • Select the folder instead of the file
  • Select the backfill option to load the existing files from the folder; otherwise, skip.
  • Add a new file with the increment changes in the folder
  • The job will automatically pick the new files (based on the file modification timestamp and the job's last run time).
5 Replies

Avatar

Community Advisor

Hi @arpang16406580, quick questions :

  • In your data selection step, did you select a particular file or folder?
  • How is the delta data fed into S3 - Does a new file in a folder or an existing file gets updated?
  • What's the value of backfill?

Here's an important thing to look for - https://experienceleague.adobe.com/docs/experience-platform/sources/ui-tutorials/dataflow/cloud-stor... 

ChetanyaJain_0-1681855495564.png

Thanks,

Chetanya

Avatar

Level 4

Hi @ChetanyaJain - Yes, i selected a particular file while ingesting data.

An existing file gets updated with the new values but it also contains the old values.

One things to notice is when ingesting data via S3 bucket we don't see an option for selecting a incremental load field so it backfills all the data again.

Avatar

Correct answer by
Community Advisor

Hi @arpang16406580, that says the reason for duplication. 

 

Since the file is getting modified, it loads the file again and ingests the records since you cannot filter based on a field. See the note below:

ChetanyaJain_0-1681951913435.png

 

The best way to handle this is as follows:

  • Select the folder instead of the file
  • Select the backfill option to load the existing files from the folder; otherwise, skip.
  • Add a new file with the increment changes in the folder
  • The job will automatically pick the new files (based on the file modification timestamp and the job's last run time).

Avatar

Level 4

Thank you, @ChetanyaJain , for the information provided. I will definitely give it a try. Based on the information, it appears that data cleaning should be handled separately, and only newly modified or updated data should be included in the new file. If a file with new timestamp contains the old data as well, AEP will ingest everything again.

Avatar

Community Advisor

That's right, any record that comes in it will re-ingest. So there are times when you must plan the events ingestion very well. Once added, they cannot be modified.

page footer