Batch Ingestion and Profile creation

[AEP_guy]

Hi ,
I have an question:

If ingest a data using Batch ingestion and just to avoid unwanted data and error before profiling ,i don't enabled the dataset for profile. so if i enabled the dataset for profile after few days of Data ingestion.
How much time it takes to do profiling done of that data ?
or is it perfect approach to do?

[shivampokhriyal]

Hi,

If data is ingested into a dataset before it is enabled for Profile, that data will be stored in the Adobe Experience Platform Data Lake, but it will not be available in the Real-Time Customer Profile.
Once the dataset is later enabled for Profile, only the data ingested after the enablement will automatically flow into the Profile store.

Regards,
Shivam

[jayakrishnaaparthasarathy]

@AEP_guy  - This is a good practice to avoid ingestion error for the batch data. I would request you to disable the profile for dataset as you mentioned and monitor the data being ingested, if all looks good then delete the respective dataset and dataflow create a new one by enabling the profile. This approach may allow us to fix any ingestion related error. However, it doesn't ingest the data to profile store if we haven't enable the profile option while testing the ingestion error. So, best delete dataset & dataflow once tested then create new one by enabling the profile.

Thank you,
Jayakrishnaa P.

[Karuppiah_Sakthivel]

Hi,

Great question — yes, your approach is valid and actually aligns with Adobe’s best practices.

To answer your questions:

1. How long does profiling take after enabling Profile on a dataset post-ingestion?
   Once you enable the dataset for Profile, the profiling process for that data typically completes within ~24 hours. However, this can vary based on system load and data volume. You can monitor progress in the Batch End-to-End Flow dashboard in AEP
   https://experienceleaguecommunities.adobe.com/t5/real-time-customer-data-platform/time-to-profile-la... 

2. Is it the right approach to delay enabling Profile to avoid bad/unwanted data?
   Yes — Adobe recommends this sequence:

   • Ingest your data

   • Fix any mapping or validation errors

   • Then enable Profile on the dataset

   • Optionally, re-ingest earlier data if needed

   https://experienceleague.adobe.com/en/docs/platform-learn/getting-started-for-data-architects-and-da... 
   Thanks
   Karuppiah Sakthivel