Dataset prioritisation for processing events | Community
Skip to main content
ButhpurKiran
ButhpurKiran
Level 4
September 29, 2025
Solved

Dataset prioritisation for processing events

  • September 29, 2025
  • 3 replies
  • 626 views

Hi,

 

We have a scenario of data(events) ingestion into RTCDP with 20k requests per second, However the processing Throughput is 1500 req/sec as per standard allocation from Adobe platform which can increase based on availability of platform resources. In our case, it's gone up to 15K. Thanks to Adobe's robust allocation algorithms. However, we have seen significant drop(to 1500) sometimes (Attached snapshot for reference). Just wanted to know if anyone experienced this issue and have a solution.

One approach i could find is through designing Ingestion architecture by creating priority based data pipelines where we can create priorities for datasets and ensure High priority datasets are allocated with resources and stream events on real time, Medium and Low priority datasets can be queued based on the data ingestion pipeline orchestration. 

Any insights or direction can greatly help us. Thanks in advance.

 

Kind regards,

Kiran Buthpur 

Best answer by Devyendar

AEP’s streaming capacity is elastic but not guaranteed, so the most reliable approach is to split events by use case and route them accordingly.

 

P0 - Edge

P1 - Streaming

P2 - Micro-batches with 15 minutes data ingestion intervals.

 

Use the Edge Network for P0 to power sub-second decisions and edge segmentation. Move P1 for near real-time streaming segmentation related use cases and P2 for Batch segmentation or CJA focused analytics use cases.

 

The exact allocations and SLAs depend on whether you’re primarily using CJA analytics, RTCDP real-time offers, or AJO orchestration.

 

Work with Adobe Solution Consultants and Product Managers to finalize entitlements, routing, and design that meets your needs and SLAs. Best Regards

3 replies

Devyendar
DevyendarAccepted solution
Level 6
September 30, 2025

AEP’s streaming capacity is elastic but not guaranteed, so the most reliable approach is to split events by use case and route them accordingly.

 

P0 - Edge

P1 - Streaming

P2 - Micro-batches with 15 minutes data ingestion intervals.

 

Use the Edge Network for P0 to power sub-second decisions and edge segmentation. Move P1 for near real-time streaming segmentation related use cases and P2 for Batch segmentation or CJA focused analytics use cases.

 

The exact allocations and SLAs depend on whether you’re primarily using CJA analytics, RTCDP real-time offers, or AJO orchestration.

 

Work with Adobe Solution Consultants and Product Managers to finalize entitlements, routing, and design that meets your needs and SLAs. Best Regards

    ButhpurKiran
    ButhpurKiranAuthor
    Level 4
    October 6, 2025

    Thanks Devyendar,

    This approach can help. We have listed all datasets and working on analysis to identify the load on each dataset based on peak times. We are currently carrying out the activity by manually verifying the reports and categorising. Just curious on to understand if this can get into easy lines with existing feature to filter?

    Kind regards,

    Kiran Buthpur  

    A
    Abie
    Level 4
    October 2, 2025

    Hi @buthpurkiran ,

     

    This 1500 requests/per second not even per dataflow, its the capacity of the entire system. So, I would not rely on system capabilities but design something custom to handle ingestion. We had a similar issue and we ended up designing wrapper API layer which also takes care of the throttling and regulating incoming requests.

     

    Thanks!

      Devyendar
      Devyendar
      Level 6
      October 6, 2025

      @buthpurkiran adding a Event throttling or let the source itself throttle the events can help but it will impact your real-time use cases if any. If there are not real-time use cases instead of throttle diverting events to batch endpoints is also wise as it frees up AEP resources and lets your key real-time use cases still possible.

       

      cc @abie 

      Devyendar
      Devyendar
      Level 6
      October 6, 2025

      Hello @buthpurkiran , Hope the earlier responses helped clarify your question, if so, kindly mark the most appropriate response as the Accepted Answer to help future readers.

      K
      Adobe Employee
      KevinFosterDXAdobe Employee
      Adobe Employee
      October 6, 2025

      Looking at the original image, it appears that your processing throughput increased on September 13th and since then has been nearly matching your actual ingestion rate?

       

      If true and that's still not sufficient bandwidth then I'd explore this method of reducing ingestion into Profile Service:

      https://experienceleaguecommunities.adobe.com/t5/adobe-experience-platform-blogs/multi-forked-event-expiration-strategy-using-datastream-in-aep/ba-p/593224

       

        ButhpurKiran
        ButhpurKiranAuthor
        Level 4
        October 13, 2025

        Thanks @kevinfosterdx 
        The multi-forking concept is quite helpful. We have upcoming integrations with new systems. This approach can ease our designs and allow to segregate the events into multi datastreams. 

        Thanks a lot again.

         

        Kind regards,

        Kiran Buthpur 

        Terms & ConditionsAccessibility statement

        Loading footer...