Time to profile latency | Community
Skip to main content
renatoz28
renatoz28
Level 2
May 28, 2024
Solved

Time to profile latency

  • May 28, 2024
  • 1 reply
  • 1527 views

Hi,

 

I want to measure the time that passes between a batch ingestion in the data lake and the amount of time it takes to get to the profile. I've been searching for documentation about how to get the time in which the data from a dataset got to the profile, but couldn't find any related API. 

 

Any possible way to accomplish that?

 

Best,

Renato

Best answer by DavidRoss91

Hi @renatoz28 

 

Within the monitoring tab, you can also navigate to the Profiles section --> select Source --> Dataflows --> Click to expand the dataflow, and within this you will see a start and end time of the write to profile update. It will provide a number of profiles updated via that specific batch. Let me know if this is helpful.

 

Thanks,

David

1 reply

DavidRoss91
Community Advisor
DavidRoss91Community Advisor
Community Advisor
June 4, 2024

Hi @renatoz28 

 

For a profile enabled dataset & schema:

 

Streaming ingestion - the profile attributes will update in real-time within seconds.

 

Batch Ingestion - Once the ingestion to the data lake is complete, there are internal jobs that will simultaneously begin (this cannot be triggered via API). This process can take up to ~24 hours to complete.

 

You can also view the progress in the monitoring dashboard within AEP --> batch end-to-end

    renatoz28
    renatoz28Author
    Level 2
    June 4, 2024

    Hi David,

     

    Thank you for your time for replying this thread. So from what I understood, there's no way to measure the time it takes for batch ingestion to go from the data lake to the profile store? Like getting the time the batch progressed until getting to the profile. I've seen batch end-to-end, but that dashboard doesn't provide that time.

     

    Best,

    Renato

      DavidRoss91
      Community Advisor
      DavidRoss91Community AdvisorAccepted solution
      Community Advisor
      June 4, 2024

      Hi @renatoz28 

       

      Within the monitoring tab, you can also navigate to the Profiles section --> select Source --> Dataflows --> Click to expand the dataflow, and within this you will see a start and end time of the write to profile update. It will provide a number of profiles updated via that specific batch. Let me know if this is helpful.

       

      Thanks,

      David

        Terms & ConditionsAccessibility statement

        Loading footer...