Dataset prioritisation for processing events

[ButhpurKiran]

Hi,

We have a scenario of data(events) ingestion into RTCDP with 20k requests per second, However the processing Throughput is 1500 req/sec as per standard allocation from Adobe platform which can increase based on availability of platform resources. In our case, it's gone up to 15K. Thanks to Adobe's robust allocation algorithms. However, we have seen significant drop(to 1500) sometimes (Attached snapshot for reference). Just wanted to know if anyone experienced this issue and have a solution.

One approach i could find is through designing Ingestion architecture by creating priority based data pipelines where we can create priorities for datasets and ensure High priority datasets are allocated with resources and stream events on real time, Medium and Low priority datasets can be queued based on the data ingestion pipeline orchestration. 

Any insights or direction can greatly help us. Thanks in advance.

Kind regards,

Kiran Buthpur 

[Devyendar]

AEP’s streaming capacity is elastic but not guaranteed, so the most reliable approach is to split events by use case and route them accordingly.

P0 - Edge

P1 - Streaming

P2 - Micro-batches with 15 minutes data ingestion intervals.

Use the Edge Network for P0 to power sub-second decisions and edge segmentation. Move P1 for near real-time streaming segmentation related use cases and P2 for Batch segmentation or CJA focused analytics use cases.

The exact allocations and SLAs depend on whether you’re primarily using CJA analytics, RTCDP real-time offers, or AJO orchestration.

Work with Adobe Solution Consultants and Product Managers to finalize entitlements, routing, and design that meets your needs and SLAs. Best Regards