Expand my Community achievements bar.

Nomination window for the Adobe Community Advisor Program, Class of 2025, is now open!
SOLVED

How to bring in new data data for updates made to existing schema

Avatar

Level 3

Hi, 

 

Client has multiple sub-businesses (about 4 or 5) and was earlier bringing in 1-Business data into AEP RTCDP. Then, very recently client wanted to bring in 2nd Businessses data. To do that I updated the exisiting schema as per client request and added fields which for 2nd Business. 

 

For data Ingestion, I used exisiting Scheduled daily dataflow and added 2nd business or new fields into it. And created another dataset which had Schema PK + 2nd business fields to do a one-time load to bring in historical data for 2nd business. Client also wanted to use this data in CJA, due to two datasets and very less overlapping data between then, it will be an issue.

 

To fix this, I have two approaches, 

  • Approach 1: Using existing dataset - 1-Time-Historical load will be done to Existing dataset; scheduled data flow will run as it is.  (New dataset that was created earlier for 2nd business, will be deleted). How should I perform this 1-Time-Load? Will it contain all fields (1st business + 2nd business fields) or it should only have 2nd business fields?
  • Approach 2: Using a new dataset - To this Historical load was done earlier. As next step, a new scheduled dataflow will be setup to the new dataset & a On-Demand Load will be done for the difference between 1st One time load and Next when a scheduled dataflow is created. 

 

Please let me know the correct and best approach. 

Topics

Topics help categorize Community content and increase your ability to discover relevant content.

1 Accepted Solution

Avatar

Correct answer by
Level 3

Hello, @SahuSa1,

 

Could you please clarify this part: "Client also wanted to use this data in CJA. Due to two datasets and very little overlapping data between them, it will be an issue."? It's not entirely clear what your question is.

 

That said, is the dataset (DS) you used for the second business enabled for profiles? If so, deleting it will also remove all associated profile data.

 

For a one-time load, I recommend ingesting everything into a dataset that is not profile-enabled first. Then, use a query to transfer the data to a profile-enabled dataset, ensuring it adheres to the XDM schema of the target dataset. This approach allows you to safely delete the non-profile-enabled dataset afterward.

Let me know if you need further clarification!

View solution in original post

4 Replies

Avatar

Correct answer by
Level 3

Hello, @SahuSa1,

 

Could you please clarify this part: "Client also wanted to use this data in CJA. Due to two datasets and very little overlapping data between them, it will be an issue."? It's not entirely clear what your question is.

 

That said, is the dataset (DS) you used for the second business enabled for profiles? If so, deleting it will also remove all associated profile data.

 

For a one-time load, I recommend ingesting everything into a dataset that is not profile-enabled first. Then, use a query to transfer the data to a profile-enabled dataset, ensuring it adheres to the XDM schema of the target dataset. This approach allows you to safely delete the non-profile-enabled dataset afterward.

Let me know if you need further clarification!

Avatar

Level 3

Hi @GigiCotruta , Thanks for your reply. 

 

Let me clarify the "little overlapping data" point. We used two datasets- 

  • one which was already present for 1st business, Updated the exisiting scheduled dataflow with new fields from 2nd business.
  • A new dataset was created in which only 2nd buisness data was ingested as one time load. 
  • Now, as one time load was done on a certain day and scheduled run was also updated the same day, the common data in both datasets would be very less. This was creating an issue in CJA, as per my understanding there is Inner join in CJA between profile type of datasets.
  • Hence we have planned for using a separate dataset only  for 2nd business so that it works for both AEP and CJA.

 

Could you please provide me with any documentation link, that you may have handy for achieving the ingest via Query service.

 

Thanks,

Sambit 

Avatar

Level 3

You probably need this docu: SQL Syntax in Query Service | Adobe Experience Platform

SELECT
_ELCORTEINGLES.CO_PAIRWISE AS CO_PAIRWISE,
ARRAY_AGG(
STRUCT(
_ELCORTEINGLES.AUDIENCES.CO_TRATAMIENTO AS CO_TRATAMIENTO,
_ELCORTEINGLES.AUDIENCES.DT_INICIO AS DT_INICIO,
_ELCORTEINGLES.AUDIENCES.ID_AUDIENCIA AS ID_AUDIENCIA,
CASE
WHEN _ELCORTEINGLES.AUDIENCES.ID_AUDIENCIA = '8889' THEN 'Eci audiencia 2'
ELSE _ELCORTEINGLES.AUDIENCES.NA_AUDIENCIA
END AS NA_AUDIENCIA,
_ELCORTEINGLES.AUDIENCES.NA_TEXTO_1 AS NA_TEXTO_1,
_ELCORTEINGLES.AUDIENCES.NA_TEXTO_2 AS NA_TEXTO_2,
_ELCORTEINGLES.AUDIENCES.NA_TEXTO_3 AS NA_TEXTO_3,
_ELCORTEINGLES.AUDIENCES.NA_TEXTO_4 AS NA_TEXTO_4,
_ELCORTEINGLES.AUDIENCES.NA_TEXTO_5 AS NA_TEXTO_5,
_ELCORTEINGLES.AUDIENCES.TS_TACADA AS TS_TACADA
)
) AS audienceECI
FROM
audiences

 

This part of the query is made for this part of the Schema: 

GigiCotruta_0-1739372470495.png

 

Avatar

Level 3

Thanks @GigiCotruta