Individual Schema data ingestion from two different source

[gprime84]

I have 2 external systems that do data ingestion on AEP, the two systems are:
1. System1 (id=system1_id)

2. System2 (id=system2_id)

Can I create two individual schemas SYSTEM1 and SYSTEM2? Or can I create a single Individual Schema that contains both the system1_id and system2_id keys as identity attribute? Obviously System2 does not have the system1_id id and vice versa.

[jayakrishnaaparthasarathy · Accepted solution]

Hi @gprime84  - 

I recommend to create a separate schema for each external system to make the proper profile stitching and unification from the different external system ( Assuming both are external system with different data source and meant for unifying the customer profile). Make sure you have the primary identifier on both the external system to identify the user and make them in to the identity graph as a single customer.
Have a look on the below document for your better understanding on the identity graph and it purpose : https://experienceleague.adobe.com/docs/experience-platform/identity/namespaces.html

As you mentioned, System2 does not have the system1_id id and vice versa, then unification and profile stitching not gonna happen unless the both the schemas have primary identifier. Otherwise, both those external system data live in separate silos under AEP data lake. Therefore, strongly suggest to revisit your schema creation, create required identity namespace, apply the necessary primary identifier accordingly. 

Hope, I made it clear and let me know in case if you have any clarification.

Note: In case if you create the single schema for both external system then you would be able to assign only one primary identifier and there are other complication while you work on the data preparation. I also don't think so you can utilize single schema for the multiple dataset on this specific scenario.

[gprime84]

So if I set up 2 individual schemas and 2 event schemas, I will have to set up two different datastreams, it's correct? because the datastream is tied to the dataset. and consequently I will have to create a new properties tag, correct? because the web sdk configuration under the property tag binds directly to the datastream

[jayakrishnaaparthasarathy]

Hi @gprime84  - Obviously, it is yes. Assuming (Hopefully), you have two different web site and / or mobile app/other domain to have the two separate property tag to create two DataStream. Respective event and profile level schema also be tied with DataStream to pass the data via edge network to AEP data lake. 

Also, wondering that you mentioned two external system and talking about web sdk which is a another in house platform.