Incremental load data discrepancy in New Profile Fragments and Existing Profile Fragments. | Community
Skip to main content
S
sandip_surse
Level 3
April 26, 2023
Solved

Incremental load data discrepancy in New Profile Fragments and Existing Profile Fragments.

  • April 26, 2023
  • 1 reply
  • 2035 views

Hi team,
Recently I did the batch data ingestion with enabling the incremental load. Once the incremental load work as on daily basis for each batch ID against the records get ingested into the dataset but all records only reflect under the New Profile Fragments as new records and Zero records reflect in Existing Profile Fragments as existing record or modified existing record. (Specific to XDM-ExperienceEvent Class Dataset)
As per source daily new updated records count is low but in AEP getting the different count what could be the reason.
All mapping is correct.

Below snaps for your reference.

 

Regards,

Sandip Surse

 

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by nnakirikanti

Hi nnakirikanti,


Please find the answers here as you requested.

1. We use snowflake source connector for batch ingestion.
2. For incremental data we use timestamp column.
3. We are doing the data preparation during the mapping for '_id' field and map with uuid() function.


aha there you go, if you are bringing in only incremental data then your updated existing profile fragments will be always zero becoz as you are ingesting deltas, if you want to update existing data then you need offload the uuid generation at source level and persist as attribute/field at record level, this way when you load historical data with updates then system not gonna treat they are new... and you will get to see that updated profile fragments count..

 

Test this with simple file of ten records with uuid mocked and ingest, do mock file with five records of old (with old uuid's) and 5 new records, now you will see the numbers showing up for existing profile count as 5 and new profile fragments as 5.

 

Let me know if you have more concerns.

 

 

1 reply

arpan-garg
Community Advisor
arpan-gargCommunity Advisor
Community Advisor
April 27, 2023

Hi @sandip_surse - Which field have you selected for incremental load? As per my understanding while data ingestion AEP will check if the timestamp of this field is newer than the timestamp of last batch ingestion. If it is new then it will ingest this record else ignore this record.

 

Can you check the values of this field?

nnakirikanti
Community Advisor
nnakirikantiCommunity Advisor
Community Advisor
May 4, 2023

@sandip_surse Few questions in here,

1) Did you use the batch ingestion api or batch source connector? If it is source connector what type is it?

2) How you are handling incremental data consideration column vs file generation?

3) is the source generation the required _id (uuid ot guid :unique string identifier for the event) for experience event or you are doing the with data prep?

 

I have faced this issue and want to recommend based on above clarifications....

S
sandip_surseAuthor
Level 3
May 4, 2023

Hi nnakirikanti,


Please find the answers here as you requested.

1. We use snowflake source connector for batch ingestion.
2. For incremental data we use timestamp column.
3. We are doing the data preparation during the mapping for '_id' field and map with uuid() function.

Terms & ConditionsAccessibility statement

Loading footer...