Expand my Community achievements bar.

Nomination window for the Adobe Community Advisor Program, Class of 2025, is now open!

How to bring in new data data for updates made to existing schema

Avatar

Level 3

Hi, 

 

Client has multiple sub-businesses (about 4 or 5) and was earlier bringing in 1-Business data into AEP RTCDP. Then, very recently client wanted to bring in 2nd Businessses data. To do that I updated the exisiting schema as per client request and added fields which for 2nd Business. 

 

For data Ingestion, I used exisiting Scheduled daily dataflow and added 2nd business or new fields into it. And created another dataset which had Schema PK + 2nd business fields to do a one-time load to bring in historical data for 2nd business. Client also wanted to use this data in CJA, due to two datasets and very less overlapping data between then, it will be an issue.

 

To fix this, I have two approaches, 

  • Approach 1: Using existing dataset - 1-Time-Historical load will be done to Existing dataset; scheduled data flow will run as it is.  (New dataset that was created earlier for 2nd business, will be deleted). How should I perform this 1-Time-Load? Will it contain all fields (1st business + 2nd business fields) or it should only have 2nd business fields?
  • Approach 2: Using a new dataset - To this Historical load was done earlier. As next step, a new scheduled dataflow will be setup to the new dataset & a On-Demand Load will be done for the difference between 1st One time load and Next when a scheduled dataflow is created. 

 

Please let me know the correct and best approach. 

Topics

Topics help categorize Community content and increase your ability to discover relevant content.

1 Reply

Avatar

Level 2

Hello, @SahuSa1,

 

Could you please clarify this part: "Client also wanted to use this data in CJA. Due to two datasets and very little overlapping data between them, it will be an issue."? It's not entirely clear what your question is.

 

That said, is the dataset (DS) you used for the second business enabled for profiles? If so, deleting it will also remove all associated profile data.

 

For a one-time load, I recommend ingesting everything into a dataset that is not profile-enabled first. Then, use a query to transfer the data to a profile-enabled dataset, ensuring it adheres to the XDM schema of the target dataset. This approach allows you to safely delete the non-profile-enabled dataset afterward.

Let me know if you need further clarification!