Using the Real Time CDP as a source of truth for Cusomer Data in a separate system.

[IanFi2]

Is is possible to use the RT CDP as a source of truth for Customer data in another system.

In particular how would new/updated/deleted profiles be kept in synch in the destination system?

How would reconciliation be done?

[Devyendar · Accepted solution]

@IanFi2 you can use 'Profile-Snapshot-Export-<UUID>' dataset to export all Profiles data from AEP into your Destination system, it can act as AEP source of truth for customer data.

Profile snapshot is typically updated once a day and has only the most recent snapshot of Profile (updated around your AEP Sandbox Batch Segmentation schedule data). It doesn't explicitly show new/update/delete, we need to decipher it comparing data between loads e.g.  previous day's load vs. current days data, you might also need to do some additional data preparation and data modelling in your destination system to know datetime records ingested compare and decipher new/update/delete status and any other meta data needed.

[IanFi2]

Thanks Devyendar,

For new/update/delete  is it possibke to use event streaming or the Entities API using the "startTime" parameter?

[Devyendar]

@IanFi2 Event streaming can not be used for this purpose. Simplest approach could be.

1. Do a dataset export of the Profile snapshot dataset to a Cloud storage system like S3, Blob storage etc.,
   • Make sure to select Incremental load while exporting the data in Destination connections for (AWS S3/Azure blob/Google cloud storage)
2. Ingest this data in your own database (all or limited fields as your require)
3. Incremental load allows you to export full data at the first time and then incrementally daily new data modified in AEP Profile snapshot will be exported.
   • Based on the incremental load of data you can run a comparison queries on the first export to determine if new Ids are added/update of profile or any profile missing probably deleted.

This is doing a new batch vs. existing full data reconciliation daily to determine the record modifications. 

[IanFi2]

Got it, thanks Devyendar