Need clarity - Deduplication in AEP to a destination

[mdajaz]

Hello,

An confused with the data behaviour in AEP when Deduplication is set to a AEP destination. The deduplication is enabled on ownership-id and has multiple Profile records with same ownership-id imported in following order:

User 1

email: test@test.com | ownership-id: 999999999 | First Name: A-1 | Last Name: B-1 | Creation Date: 1/1/2022

User 2

email: test@test.com | ownership-id: 999999999 | First Name: A-2 | Last Name: B-2  |Creation Date: 2//2/2023

What is ideally expected from the data export to the destination? As per AEP docs (https://experienceleague.adobe.com/docs/experience-platform/destinations/ui/activate/activate-batch-... it should be "Profile 2 is the latest one that qualified for the audience, so it is the only one getting exported.". So I should get User 2 in the data export.

But, I see different results. Any reason for this behaviour? Appreciate any pointers.

Thanks!

[abhinavbalooni]

Hey @mdajaz 

When you say you observe different results, can you provide example of what you see at your end. Also, creation date above ? Is that the membership qualification time ? You need to look at the lastqualificationtime for the segment membership and not the creation date of the record highlighted above to see if User1 records should reflect or User2 records should get exported.

For eg., even though User2 was created more recently, if User1 qualifies for a segment on a later date (more recent) you are trying to export against, User1 record will get exported.

Hope the above helps.

Cheers,

Abhinav

[abhinavbalooni]

Hey @mdajaz 

Did the above response help? Let me know if you need any further help.

Cheers,

Abhinav

[mdajaz]

Thanks @abhinavbalooni for your response. I have the related segments set at "Streaming" frequency which implies that User 2 (second record ) will get qualified later (most recent). In that case, I should have User-2 exported to my destination. Is that right understanding?

To answer your second question, I see user-1 exported. 

[abhinavbalooni]

Hey @mdajaz 

Do you have the lastQualification time for the segment for both User 1 & 2 ? What seems like is that based on your segment definition, User 1 is qualifying more recently and then you get that as an export.

Do one thing, under profiles, there is an option to view JSON which should give you lastqualificationtime of the segment u r looking at for both User 1 & 2. I am pretty sure, it would show that User 1 has more recently qualified for the segment and hence exported.

You can also use Profile API if you are more comfortable that ways to get the segment qualification for both users.

[mdajaz]

Additionally, what happens when I already have more than 2 Records for a customer and all of the records gets qualified for a Segment (assume Batch Evaluation and more than 5 records for the user from different data sources) used for Destination export. 

Now, all the records will be qualified for the Segment as part of the same Evaluation run and hence will have same timestamp. What is the expected behaviour in this case? Which record will be sent to the destination?

I need to know since I am in this use case and need to take appropriate design decision based on product behaviour. Thanks!

[abhinavbalooni]

@mdajaz That sounds interesting. Though I havent been in this situation but if the lastqualificationtime is the same for all records then all should get exported. May be you can try it out. But all the records should get exported.

Out of curiosity, you mentioned 5 records of the same person ? Why won't these be merged ? Are the records on different identities ?

[mdajaz]

Yes, few are on different identities like email, CustomerID, phone etc coming from different systems. How should I tackle this scenario? 

I believe, I need to set secondary identities accordingly so that they get merged in Unified profile datalake before Segmentation picks up that data. Thoughts?

[abhinavbalooni · Accepted solution]

Hi @mdajaz 

Yeah you should be defining these fields as identities, whether primary or secondary would depend on the system where the data comes from. For instance, data from CRM would have CRMId as a primary identity and then rest as identities.

You would also need to ensure that you have some form of a common identity across sources so that AEP knows that data coming from multiple sources belongs to one person and hence needs to be merged into one profile.

Hope the above helps.

Cheers.

[mdajaz]

Maybe last question on this thread, suppose if I have email as Primary Identity, these these two records (shown below) will not get collapsed/merged to the same CDP. Assuming, both the records meet the segment qualification criteria, implies that both the records will qualify approx 24 hours (worst case - Batch evaluation) after the record creation (which are an year apart). Implies both the records co-exists in Unified Profile data lake.

Now, if I have deduplication enabled on ownership-id (which matches for both records), which records will get exported?

User 1

email: testABC@test.com | ownership-id: 999999999 | First Name: A-1 | Last Name: B-1 | Creation Date: 1/1/2022

User 2

email: test@test.com | ownership-id: 999999999 | First Name: A-2 | Last Name: B-2  |Creation Date: 2//2/2023