Newbie - Please Explain How Data Sets Work in the flow of Data Ingestion

[MichaelJo13]

Hi All,

I'm trying to learn about AEP etc. What I'm confused about is when you create an XDM schema, you have to create a Data Set from it. What confuses me is, for example, if that data set has a count of # of times a widget was used, then will the data set automatically get updated when the widget is used again, or does it get updated when a data view is created using that data set? If not, when is that data updated and how? 

Thanks!

[Pradeep_Kumar_Srivastav · Accepted solution]

Hi @MichaelJo13 ,

The dataset gets updated based on the data ingestion type e.g. Batch or Streaming. Please have a look at the document https://experienceleague.adobe.com/en/docs/experience-platform/ingestion/home to get more details on how the data gets ingested into the platform.

Data Ingestion and Updates

• Data Ingestion:

  • Data ingestion refers to the process of bringing data into AEP from various sources.
  • Data can be ingested in batch mode or in real-time streaming mode.
  • When data is ingested, it populates the data set according to the defined XDM schema.

• Updating Data Sets:

  • Data sets do not automatically update when an event (such as a widget being used) occurs unless the data ingestion process is triggered.
  • To update the data set with new events, you need to ensure that data is continuously ingested.

How DataSet Gets Updated

1. Event-Driven Updates:

   • For real-time updates, you can use streaming data ingestion.
   • For example, when a widget is used, an event can be sent to AEP through a data streaming service, which then updates the data set in real-time.

2. Batch Updates:

   • For periodic updates, you can use batch ingestion.
   • Data is collected over a period (e.g., daily) and then uploaded to AEP in a batch.

Hope this helps.

[MichaelJo13]

Thank you very much!