How to Achieve Seamless Data Integration and Performance in Real-Time CDPs Under High Concurrency?

[Betterhu]

How can a real-time Customer Data Platform (CDP) effectively integrate heterogeneous data from multiple sources—such as structured CRM data, semi-structured website clickstream data, unstructured social media data, and third-party advertising platforms—while ensuring data consistency and real-time performance, particularly in high-concurrency scenarios like e-commerce promotional events, where processing delays could impact personalized recommendations and marketing automation?

[itsMeTechy · Accepted solution]

AEP/RTCDP is meant for this purpose. There are wide range of OOTB source connectors which can bring data from sources like CRM, Cloud storage, databases, social media, adobe native applications and so on. you can check list of such connectors in the below link.

https://experienceleague.adobe.com/en/docs/experience-platform/sources/home

You can ingest data in csv, json or parquet. You can also stream your click stream data using webSDK and mobileSDK.

Regarding concurrency you can check the guardrails of streaming ingestion which can be found in the below link. Based on specific requirement you can also request Adobe and look out for any possibility for enhancing the concurrency.

https://experienceleague.adobe.com/en/docs/experience-platform/ingestion/guardrails

[GigiCotruta]

Hi @Betterhu 

Basically, there are few ways to integrate data in RT. You either use one of the available SDKs (web or mobile) or you can use a Streaming connection like HTTP API.

In order to structure all the data, you can always use schemas, I would highly recommend using as much prebuild Adobe field groups as possible as those generally work better for the integrations between Adobe products (Adobe Journey Optimizer, Analytics, Target, etc.)

Now, once you have ingested this data, you can create segments and send those to your RT connections like Adobe Target, Adobe Commerce, Custom personalization, social media, etc.

If you need a more in-detail answer, feel free to describe your use case so we can try and help you.