Time to profile latency

[renatoz28]

Hi,

I want to measure the time that passes between a batch ingestion in the data lake and the amount of time it takes to get to the profile. I've been searching for documentation about how to get the time in which the data from a dataset got to the profile, but couldn't find any related API. 

Any possible way to accomplish that?

Best,

Renato

[DavidRoss91]

Hi @renatoz28 

For a profile enabled dataset & schema:

Streaming ingestion - the profile attributes will update in real-time within seconds.

Batch Ingestion - Once the ingestion to the data lake is complete, there are internal jobs that will simultaneously begin (this cannot be triggered via API). This process can take up to ~24 hours to complete.

You can also view the progress in the monitoring dashboard within AEP --> batch end-to-end

[renatoz28]

Hi David,

Thank you for your time for replying this thread. So from what I understood, there's no way to measure the time it takes for batch ingestion to go from the data lake to the profile store? Like getting the time the batch progressed until getting to the profile. I've seen batch end-to-end, but that dashboard doesn't provide that time.

Best,

Renato

[DavidRoss91 · Accepted solution]

Hi @renatoz28 

Within the monitoring tab, you can also navigate to the Profiles section --> select Source --> Dataflows --> Click to expand the dataflow, and within this you will see a start and end time of the write to profile update. It will provide a number of profiles updated via that specific batch. Let me know if this is helpful.

Thanks,

David