For streaming ingestion into Adobe Experience Platform using an Individual Profile class schema, which timestamp or field determines how a profile record is ingested and evaluated
Hi Community,
I’m looking for some guidance on a streaming ingestion scenario in Adobe Experience Platform.
We have a use case where multiple events are received for a single transaction from an upstream system via streaming ingestion, and the data is ingested into an Individual Profile–enabled dataset. The expectation is that the latest record should be reflected on the profile.
However, the challenge is that multiple events for the same transaction:
- Arrive within milliseconds of each other
- Belong to the same streaming batch
- End up having the same ingestion timestamp
Because of this, AEP often retains only the first record, and the profile is not consistently updated with the latest event.
To address this, we mapped the source system’s published date to the lastUpdatedDate field (OOTB External Source System field group). This approach works when there are up to two events, but in some cases, the source system sends more than four events for the same transaction, and the profile still fails to reflect the latest record—even though the merge policy is timestamp-based.
Adobe Support has indicated that since multiple updates to the same profile fragment occur within milliseconds, there is no guarantee that the latest record will be deterministically applied.
My questions are:
- Is there a recommended way to handle this scenario within AEP?
- Are there any best practices around merge policies, timestamp fields, or schema design that could help ensure the latest record consistently wins?
- Is this an inherent limitation of streaming ingestion, or is there something configurable within AEP to improve reliability in this case?
Any suggestions or insights would be greatly appreciated
From the below screen shot we have same ingestion time but the published date is mapped to external source system filed will that be able to decide the merge policy ?
the profile should have the data from 1st event which is at 52:40:251
but in the RTCDP it is showing the one from 52:21:788.
how can this be handled in AEP ?
is there a way
This Document Explains that lastUpdateddate can be used for defining the priority but at what point ?
is this true for individual class schema ?
any suggestions would be helpful
Thanks
Reshmika