What are some ways to reduce daily segmentation job processing time?

[akwankl]

Hi all,

Currently we have around 9M profiles, and 500 segments. The daily process is taking around 6 hours to complete, unfortunately our upper stream data doesn't update and get to us till around 5am PT, and that means our daily segmentation job doesn't complete till around 1pm PT which is really late for the business.

We have plans to reduce the number of segments, but is there anything else we can do?

Thanks.

[Anil_Umachigi]

@akwankl Nothing wihin the UI, however is there a way you can split the upstream data and steram/batch them as they become available?  

Anil 

[akwankl]

Hi @Anil_Umachigi,

Can you elaborate on how that will work? Say if I have 10 data sources currently all using batch API, switch 5 of them to streaming API, are you suggesting that will that improve the daily segmentation processing time? 

Thanks.

[Anil_Umachigi]

@akwankl Sure, this would be a approach which would require design update but worth a look, you could investigate of all the data coming in today (via batch) if some of it can be coverted to streaming type ( example 40% of current attributes can be converted to streaming data using either HTTP API or out of box connector). 

If you could identify the key attributes which are required for real time or use cases where streaming would help, you could push towards making these attributes available via streaming method instead of batch 

Hope that helps 

Anil 

[akwankl]

Hi @Anil_Umachigi ,

Thanks for the clarification. So if my daily job takes around 6 hours to complete, and I convert say x% of attributes to streaming, I should see some kind of decrease from 6 hours? Do the segments need to be converted to streaming too?

Thanks.

[Anil_Umachigi]

@akwankl right so i should have mentioned, so this will reduce the amount of data which will be evaluated via batch and hopefully the time to complete, additionally based on what attributes ( rather usecases) are now streaming you can create segments with these for faster evaluation. 

Anil

[brekrut]

@akwankl  As mentioned above the batch segmentation job will process all segments within the Adobe Experience Platform sandbox.  

As more segments get built out the longer the processing time will be for the segmentation service, but I would not recommending using the segmentation service to rely upon same day campaigns.  Another item to note use arrays to store data and when used in segmentation logic also can slow down the batch segmentation process.  

If your looking to speed up the evaluation process of segmentation for audiences you may want to consider a means other than batch ingestion to bring data into Adobe Experience Platform.

[akwankl]

Hi @brekrut, 

Thanks for the clarification, sorry still a bit foggy for me on this concept.

Let's say if I convert the some of our data sources to streaming ingestion, and convert our segments to streaming as well. But wouldn't those streaming segments still go through daily segmentation? So let's say out of 600 segments, even if I convert 200 to streaming, wouldn't the daily segmentation job still evaluate the 600 segments and hence the time would be the same?

Thanks.

[brekrut · Accepted solution]

Hello @akwankl 

We need to consider three phases within the Adobe Experience Platform (AEP) when it comes to data.  There is ingestion, segmentation, and activation of the audience.  

All segments in AEP will be processed by the daily segmentation job, but streaming segments will be evaluated upon the incoming hit of data.  An item to consider is the segmentation logic when creating a segment.  Segments which are created with a timeframe longer than 7 days will be evaluated in batch. 

What is the use case you are attempting to achieve? Or is the goal to see if you can reduce the segmentation evaluation time?  What are the systems you are activating data into?