Data Lake updates | Community
Skip to main content
Silvio6
Silvio6
Level 5
August 19, 2025
Solved

Data Lake updates

  • August 19, 2025
  • 2 replies
  • 1791 views

Hi,


Why does it take almost 30 minutes for the AEP Data Lake to update datasets? I need to monitor bounce events, but the data isn't available in real time.

Also, could you clarify why in Adobe Campaign Standard those values are available in real time, while in AEP they are delayed?

 

Thanks!

Best answer by SatheeskannaK

@silvio6 Adobe Campaign Standard (ACS) runs on a cloud-based PostgreSQL database, storing data in a table, while AEP has a different approach.

Even though events like bounces, opens, and clicks are recorded almost instantly in Adobe Journey Optimizer, they are processed into the AEP Data Lake in micro-batches rather than in real-time. Typically, Adobe’s SLA for Data Lake ingestion falls within the 15–30 minute range, influenced by load and pipeline latency. Additionally, when data arrives in the Data Lake, AEP performs schema validation (XDM), enrichments, and identity stitching, which adds some processing time before the profile dataset becomes available. It's important to note that the data is stored in the Data Lake using Parquet format, and metadata indexes are updated, which takes longer than a direct stream.

2 replies

SatheeskannaK
Community Advisor
SatheeskannaKCommunity AdvisorAccepted solution
Community Advisor
August 19, 2025

@silvio6 Adobe Campaign Standard (ACS) runs on a cloud-based PostgreSQL database, storing data in a table, while AEP has a different approach.

Even though events like bounces, opens, and clicks are recorded almost instantly in Adobe Journey Optimizer, they are processed into the AEP Data Lake in micro-batches rather than in real-time. Typically, Adobe’s SLA for Data Lake ingestion falls within the 15–30 minute range, influenced by load and pipeline latency. Additionally, when data arrives in the Data Lake, AEP performs schema validation (XDM), enrichments, and identity stitching, which adds some processing time before the profile dataset becomes available. It's important to note that the data is stored in the Data Lake using Parquet format, and metadata indexes are updated, which takes longer than a direct stream.

Thanks, Sathees
    Devyendar
    Devyendar
    Level 6
    September 4, 2025

    @satheeskannak does ACS has a native source connector to AEP? Is the approach of onboarding ACS data through Cloud storage > Batch ingestion or HTTP API ingestion into AEP?

    K
    Adobe Employee
    KevinFosterDXAdobe Employee
    Adobe Employee
    September 4, 2025

    As mentioned above, streaming data is batched before it's written to the data lake. But it is ingested immediately into Profile and Identity Services, so there's not much reason to worry about the intervals between batches in the data lake. Nothing should be trying to read from those datasets in the data lake in anything approaching real-time... or even same day normally.

    Terms & ConditionsAccessibility statement

    Loading footer...