Newbie - Please Explain How Data Sets Work in the flow of Data Ingestion | Community
Skip to main content
M
MichaelJo13
Level 4
August 8, 2024
Solved

Newbie - Please Explain How Data Sets Work in the flow of Data Ingestion

  • August 8, 2024
  • 1 reply
  • 784 views

Hi All,

I'm trying to learn about AEP etc. What I'm confused about is when you create an XDM schema, you have to create a Data Set from it. What confuses me is, for example, if that data set has a count of # of times a widget was used, then will the data set automatically get updated when the widget is used again, or does it get updated when a data view is created using that data set? If not, when is that data updated and how? 

 

Thanks!

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by Pradeep_Kumar_Srivastav

Hi @michaeljo13 ,

 

The dataset gets updated based on the data ingestion type e.g. Batch or Streaming. Please have a look at the document https://experienceleague.adobe.com/en/docs/experience-platform/ingestion/home to get more details on how the data gets ingested into the platform.

 

Data Ingestion and Updates

  • Data Ingestion:

    • Data ingestion refers to the process of bringing data into AEP from various sources.
    • Data can be ingested in batch mode or in real-time streaming mode.
    • When data is ingested, it populates the data set according to the defined XDM schema.

  • Updating Data Sets:

    • Data sets do not automatically update when an event (such as a widget being used) occurs unless the data ingestion process is triggered.
    • To update the data set with new events, you need to ensure that data is continuously ingested.

How DataSet Gets Updated

  1. Event-Driven Updates:

    • For real-time updates, you can use streaming data ingestion.
    • For example, when a widget is used, an event can be sent to AEP through a data streaming service, which then updates the data set in real-time.

  2. Batch Updates:

    • For periodic updates, you can use batch ingestion.
    • Data is collected over a period (e.g., daily) and then uploaded to AEP in a batch.

Hope this helps.

 

 

 

1 reply

Pradeep_Kumar_Srivastav
Community Advisor
Pradeep_Kumar_SrivastavCommunity AdvisorAccepted solution
Community Advisor
August 8, 2024

Hi @michaeljo13 ,

 

The dataset gets updated based on the data ingestion type e.g. Batch or Streaming. Please have a look at the document https://experienceleague.adobe.com/en/docs/experience-platform/ingestion/home to get more details on how the data gets ingested into the platform.

 

Data Ingestion and Updates

  • Data Ingestion:

    • Data ingestion refers to the process of bringing data into AEP from various sources.
    • Data can be ingested in batch mode or in real-time streaming mode.
    • When data is ingested, it populates the data set according to the defined XDM schema.

  • Updating Data Sets:

    • Data sets do not automatically update when an event (such as a widget being used) occurs unless the data ingestion process is triggered.
    • To update the data set with new events, you need to ensure that data is continuously ingested.

How DataSet Gets Updated

  1. Event-Driven Updates:

    • For real-time updates, you can use streaming data ingestion.
    • For example, when a widget is used, an event can be sent to AEP through a data streaming service, which then updates the data set in real-time.

  2. Batch Updates:

    • For periodic updates, you can use batch ingestion.
    • Data is collected over a period (e.g., daily) and then uploaded to AEP in a batch.

Hope this helps.

 

 

 

    M
    MichaelJo13Author
    Level 4
    August 8, 2024

    Thank you very much!

      Terms & ConditionsAccessibility statement

      Loading footer...