Data Lake updates

[Silvio6]

Hi,

Why does it take almost 30 minutes for the AEP Data Lake to update datasets? I need to monitor bounce events, but the data isn't available in real time.

Also, could you clarify why in Adobe Campaign Standard those values are available in real time, while in AEP they are delayed?

Thanks!

[SatheeskannaK · Accepted solution]

@Silvio6 Adobe Campaign Standard (ACS) runs on a cloud-based PostgreSQL database, storing data in a table, while AEP has a different approach.

Even though events like bounces, opens, and clicks are recorded almost instantly in Adobe Journey Optimizer, they are processed into the AEP Data Lake in micro-batches rather than in real-time. Typically, Adobe’s SLA for Data Lake ingestion falls within the 15–30 minute range, influenced by load and pipeline latency. Additionally, when data arrives in the Data Lake, AEP performs schema validation (XDM), enrichments, and identity stitching, which adds some processing time before the profile dataset becomes available. It's important to note that the data is stored in the Data Lake using Parquet format, and metadata indexes are updated, which takes longer than a direct stream.

[Devyendar]

@SatheeskannaK does ACS has a native source connector to AEP? Is the approach of onboarding ACS data through Cloud storage > Batch ingestion or HTTP API ingestion into AEP?

[KevinFosterDX]

As mentioned above, streaming data is batched before it's written to the data lake. But it is ingested immediately into Profile and Identity Services, so there's not much reason to worry about the intervals between batches in the data lake. Nothing should be trying to read from those datasets in the data lake in anything approaching real-time... or even same day normally.