Batch Ingestion and Profile creation | Community
Skip to main content
A
AEP_guy
Level 2
June 11, 2025
Solved

Batch Ingestion and Profile creation

  • June 11, 2025
  • 3 replies
  • 709 views

Hi ,
I have an question:

If ingest a data using Batch ingestion and just to avoid unwanted data and error before profiling ,i don't enabled the dataset for profile. so if i enabled the dataset for profile after few days of Data ingestion.
How much time it takes to do profiling done of that data ?
or is it perfect approach to do?

Best answer by shivampokhriyal

Hi,

If data is ingested into a dataset before it is enabled for Profile, that data will be stored in the Adobe Experience Platform Data Lake, but it will not be available in the Real-Time Customer Profile.
Once the dataset is later enabled for Profile, only the data ingested after the enablement will automatically flow into the Profile store.

Regards,
Shivam

3 replies

shivampokhriyal
Adobe Employee
shivampokhriyalAdobe EmployeeAccepted solution
Adobe Employee
June 11, 2025

Hi,

If data is ingested into a dataset before it is enabled for Profile, that data will be stored in the Adobe Experience Platform Data Lake, but it will not be available in the Real-Time Customer Profile.
Once the dataset is later enabled for Profile, only the data ingested after the enablement will automatically flow into the Profile store.

Regards,
Shivam

    jayakrishnaaparthasarathy
    Community Advisor
    jayakrishnaaparthasarathyCommunity Advisor
    Community Advisor
    June 11, 2025

    @aep_guy  - This is a good practice to avoid ingestion error for the batch data. I would request you to disable the profile for dataset as you mentioned and monitor the data being ingested, if all looks good then delete the respective dataset and dataflow create a new one by enabling the profile. This approach may allow us to fix any ingestion related error. However, it doesn't ingest the data to profile store if we haven't enable the profile option while testing the ingestion error. So, best delete dataset & dataflow once tested then create new one by enabling the profile.

    Thank you,
    Jayakrishnaa P.

      K
      Karuppiah_Sakthivel
      Level 4
      June 16, 2025

      Hi,

      Great question — yes, your approach is valid and actually aligns with Adobe’s best practices.

      To answer your questions:

      1. How long does profiling take after enabling Profile on a dataset post-ingestion?
        Once you enable the dataset for Profile, the profiling process for that data typically completes within ~24 hours. However, this can vary based on system load and data volume. You can monitor progress in the Batch End-to-End Flow dashboard in AEP
        https://experienceleaguecommunities.adobe.com/t5/real-time-customer-data-platform/time-to-profile-latency/m-p/677732 

      2. Is it the right approach to delay enabling Profile to avoid bad/unwanted data?
        Yes — Adobe recommends this sequence:

        • Ingest your data

        • Fix any mapping or validation errors

        • Then enable Profile on the dataset

        • Optionally, re-ingest earlier data if needed

        https://experienceleague.adobe.com/en/docs/platform-learn/getting-started-for-data-architects-and-data-engineers/enable-profiles 

         

        Thanks
        Karuppiah Sakthivel
        Terms & ConditionsAccessibility statement

        Loading footer...