Redundant data in dataset.

Forum|Forum|2 years ago
April 18, 2023
2 replies
1419 views

Hi,

I am ingesting some data from Amazon S3 bucket every 15 mins and i noticed that the data which is already ingested is getting ingested again and again as a separate row in the dataset.

Is there a way to make sure that data already ingested is not ingested again?

Experience Platform

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

Best answer by ChetanyaJain-1

Hi @arpan-garg, that says the reason for duplication.

Since the file is getting modified, it loads the file again and ingests the records since you cannot filter based on a field. See the note below:

The best way to handle this is as follows:

Select the folder instead of the file
Select the backfill option to load the existing files from the folder; otherwise, skip.
Add a new file with the increment changes in the folder
The job will automatically pick the new files (based on the file modification timestamp and the job's last run time).

ChetanyaJain-1

Community Advisor

Hi @arpan-garg, quick questions :

In your data selection step, did you select a particular file or folder?
How is the delta data fed into S3 - Does a new file in a folder or an existing file gets updated?
What's the value of backfill?

Here's an important thing to look for - https://experienceleague.adobe.com/docs/experience-platform/sources/ui-tutorials/dataflow/cloud-storage.html?lang=en#schedule-ingestion-runs

Thanks,

Chetanya

arpan-garg

Author

Community Advisor

Hi @chetanyajain-1 - Yes, i selected a particular file while ingesting data.

An existing file gets updated with the new values but it also contains the old values.

One things to notice is when ingesting data via S3 bucket we don't see an option for selecting a incremental load field so it backfills all the data again.

ChetanyaJain-1

Accepted solution

Community Advisor

Hi @arpan-garg, that says the reason for duplication.

Since the file is getting modified, it loads the file again and ingests the records since you cannot filter based on a field. See the note below:

The best way to handle this is as follows:

Select the folder instead of the file
Select the backfill option to load the existing files from the folder; otherwise, skip.
Add a new file with the increment changes in the folder
The job will automatically pick the new files (based on the file modification timestamp and the job's last run time).

N

Nirmalkumar_Ses

That's right, any record that comes in it will re-ingest. So there are times when you must plan the events ingestion very well. Once added, they cannot be modified.

So data preparation/preprocessing is necessary before ingesting the data? Dear Chetanya, is there any recommended approach to automate this? Any tools that could find the diff and create the necessary input?

Sign up

Login with SSO

Login to the community

Login with SSO

Scanning file for viruses.

This file cannot be downloaded