Ensuring Latest Information Prevails in AEP Streaming Ingestion

[JeanBaro_]

Hello there,

We’re encountering a significant issue with data consistency in AEP during streaming ingestion of consent data from a single server-to-server source. Our expectation is that the most recent consent status will replace older information in the CDP/AEP Profile. However, instead, the older consent data is being retained. This data inconsistency is concerning, especially given that reliable, up-to-date data in AEP is crucial for many users.

I’m aware of the upcoming Customer Timestamp Ordering feature which could help resolve this by using source timestamps to prioritize the latest data. However, this feature isn’t available for production yet, has several limitations, and lacks an ETA.

Here’s an example of what we’re seeing:

UserID Status Time

1	Skipped	14:25:23
1	Accepted	14:38:01

Despite the second record being newer by 13 minutes (based on the source’s wall clock), the profile sometimes ends up with the "Skipped" status instead of the expected "Accepted" status. This inconsistency is problematic as we end up with outdated consent information in the Unified Profile Store.

Given the critical nature of data integrity, what are the best practices or alternative solutions available to address this currently?

Any guidance would be greatly appreciated.

Thank you in advance!

[DavidRoss91]

Hi @JeanBaro_ 

What is the current merge policy used in your instance of AEP? The difference between using timestamp or dataset precedence may help solve your use case. I have attached the documentation below.

https://experienceleague.adobe.com/en/docs/experience-platform/profile/merge-policies/overview#datas...

[JeanBaro_]

I’ll check out the link you shared! Thank you!

From what I understand, as long as we have the appropriate merge policies configured, the ingestion order in AEP STREAMING INGESTION should ensure the most recent data prevails in the profile (unless there's a race condition with events for the same ID arriving simultaneously).

So, am I correct in thinking that with the proper timestamp-based or dataset precedence rules, we should avoid seeing outdated information retained in the profile?

This would mean that the issue we’re experiencing with older data overriding newer data shouldn’t actually occur if everything is set up correctly?

Thanks again for your help!

[JeanBaro_]

@DavidRoss91 

Just to make sure this is clear, we are using Streaming Ingestion.

[DavidRoss91]

Hi @JeanBaro_ 

Yes, if everything is setup correctly this should not occur. Are you using timebased or dataset precedence? If timebased, that should prioritize the most recent data based on timestamps. If using dataset precedence, then whichever dataset has the highest ranking can overwrite fields that may occur within multiple datasets (i.e. identity values like CTID, custguid, etc)

[kautuk_sahni]

@JeanBaro_ Were these suggestions helpful? If you need further clarification, feel free to ask. Otherwise, marking the answer as correct can guide others. If you've resolved the issue on your own, please share your solution to benefit the community. Thank you!