Dataset prioritisation for processing events

[ButhpurKiran]

Hi,

We have a scenario of data(events) ingestion into RTCDP with 20k requests per second, However the processing Throughput is 1500 req/sec as per standard allocation from Adobe platform which can increase based on availability of platform resources. In our case, it's gone up to 15K. Thanks to Adobe's robust allocation algorithms. However, we have seen significant drop(to 1500) sometimes (Attached snapshot for reference). Just wanted to know if anyone experienced this issue and have a solution.

One approach i could find is through designing Ingestion architecture by creating priority based data pipelines where we can create priorities for datasets and ensure High priority datasets are allocated with resources and stream events on real time, Medium and Low priority datasets can be queued based on the data ingestion pipeline orchestration. 

Any insights or direction can greatly help us. Thanks in advance.

Kind regards,

Kiran Buthpur 

[Devyendar · Accepted solution]

AEP’s streaming capacity is elastic but not guaranteed, so the most reliable approach is to split events by use case and route them accordingly.

P0 - Edge

P1 - Streaming

P2 - Micro-batches with 15 minutes data ingestion intervals.

Use the Edge Network for P0 to power sub-second decisions and edge segmentation. Move P1 for near real-time streaming segmentation related use cases and P2 for Batch segmentation or CJA focused analytics use cases.

The exact allocations and SLAs depend on whether you’re primarily using CJA analytics, RTCDP real-time offers, or AJO orchestration.

Work with Adobe Solution Consultants and Product Managers to finalize entitlements, routing, and design that meets your needs and SLAs. Best Regards

[ButhpurKiran]

Thanks Devyendar,

This approach can help. We have listed all datasets and working on analysis to identify the load on each dataset based on peak times. We are currently carrying out the activity by manually verifying the reports and categorising. Just curious on to understand if this can get into easy lines with existing feature to filter?

Kind regards,

Kiran Buthpur  

[Abie]

Hi @ButhpurKiran ,

This 1500 requests/per second not even per dataflow, its the capacity of the entire system. So, I would not rely on system capabilities but design something custom to handle ingestion. We had a similar issue and we ended up designing wrapper API layer which also takes care of the throttling and regulating incoming requests.

Thanks!

[Devyendar]

@ButhpurKiran adding a Event throttling or let the source itself throttle the events can help but it will impact your real-time use cases if any. If there are not real-time use cases instead of throttle diverting events to batch endpoints is also wise as it frees up AEP resources and lets your key real-time use cases still possible.

cc @Abie 

[Devyendar]

Hello @ButhpurKiran , Hope the earlier responses helped clarify your question, if so, kindly mark the most appropriate response as the Accepted Answer to help future readers.

[ButhpurKiran]

Thanks Abie, Appreciate the idea. Had a thought about it, throttling requests can be helpful in managing resources, however it impacts our real-time ingestions as there are few campaigns and activities that are expected to trigger just in seconds when player kicks in goal. And, approx 25M users will be qualified to receive notifications and emails.

Thanks again.

Kind regards,

Kiran Buthpur

[KevinFosterDX]

Looking at the original image, it appears that your processing throughput increased on September 13th and since then has been nearly matching your actual ingestion rate?

If true and that's still not sufficient bandwidth then I'd explore this method of reducing ingestion into Profile Service:

https://experienceleaguecommunities.adobe.com/t5/adobe-experience-platform-blogs/multi-forked-event-...