What are some ways to reduce daily segmentation job processing time? | Community
Skip to main content
A
akwankl
Level 6
July 18, 2024
Solved

What are some ways to reduce daily segmentation job processing time?

  • July 18, 2024
  • 1 reply
  • 1447 views

Hi all,

 

Currently we have around 9M profiles, and 500 segments. The daily process is taking around 6 hours to complete, unfortunately our upper stream data doesn't update and get to us till around 5am PT, and that means our daily segmentation job doesn't complete till around 1pm PT which is really late for the business.

 

We have plans to reduce the number of segments, but is there anything else we can do?

 

Thanks.

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by brekrut

Hi @brekrut, 

 

Thanks for the clarification, sorry still a bit foggy for me on this concept.

 

Let's say if I convert the some of our data sources to streaming ingestion, and convert our segments to streaming as well. But wouldn't those streaming segments still go through daily segmentation? So let's say out of 600 segments, even if I convert 200 to streaming, wouldn't the daily segmentation job still evaluate the 600 segments and hence the time would be the same?

 

Thanks.


Hello @akwankl 

 

We need to consider three phases within the Adobe Experience Platform (AEP) when it comes to data.  There is ingestion, segmentation, and activation of the audience.  

 

All segments in AEP will be processed by the daily segmentation job, but streaming segments will be evaluated upon the incoming hit of data.  An item to consider is the segmentation logic when creating a segment.  Segments which are created with a timeframe longer than 7 days will be evaluated in batch. 

 

What is the use case you are attempting to achieve? Or is the goal to see if you can reduce the segmentation evaluation time?  What are the systems you are activating data into?

1 reply

Anil_Umachigi
Adobe Employee
Anil_UmachigiAdobe Employee
Adobe Employee
July 19, 2024

@akwankl Nothing wihin the UI, however is there a way you can split the upstream data and steram/batch them as they become available?  

 

Anil 

    A
    akwanklAuthor
    Level 6
    July 23, 2024

    Hi @anil_umachigi,

     

    Can you elaborate on how that will work? Say if I have 10 data sources currently all using batch API, switch 5 of them to streaming API, are you suggesting that will that improve the daily segmentation processing time? 

     

    Thanks.

      A
      akwanklAuthor
      Level 6
      July 24, 2024

      @akwankl  As mentioned above the batch segmentation job will process all segments within the Adobe Experience Platform sandbox.  

       

      As more segments get built out the longer the processing time will be for the segmentation service, but I would not recommending using the segmentation service to rely upon same day campaigns.  Another item to note use arrays to store data and when used in segmentation logic also can slow down the batch segmentation process.  

       

      If your looking to speed up the evaluation process of segmentation for audiences you may want to consider a means other than batch ingestion to bring data into Adobe Experience Platform.


      Hi @brekrut, 

       

      Thanks for the clarification, sorry still a bit foggy for me on this concept.

       

      Let's say if I convert the some of our data sources to streaming ingestion, and convert our segments to streaming as well. But wouldn't those streaming segments still go through daily segmentation? So let's say out of 600 segments, even if I convert 200 to streaming, wouldn't the daily segmentation job still evaluate the 600 segments and hence the time would be the same?

       

      Thanks.

      Terms & ConditionsAccessibility statement

      Loading footer...