Can AEM be run with Dispatcher (Apache) in Worker (MPM) mode

[bcop]

Is it possible to run Apache in "Worker" Multi-Processing Modules (MPMs) mode?

Apache Prefork vs Worker MPM - Stack Overflow

https://www.garron.me/en/blog/apache2-mpm-worker-prefork-php.html

Just wondering if this presents any particular problems for AEM in a typical Author-Publisher-Dispatcher setup (one Author, 4 Publishers, 4 Dispatchers), we are hoping to boost performance of our servers as it is failing to serve 1000 consecutive customers in JMeter performance tests and we are weeks away from launching. We will not be approved with this poor performance and our AEM project will be at risk.

• Apache 2.4
• AEM 6.1

Need any other info to assess, please let me know... this is fairly urgent to get a response to confirm or deny this being a possibility.

If its not possible, please let me know why it may not work well (I guess we'd be giving up Thread safety for performance).

[kautuk_sahni]

Jörg Hoh​ Any Help here?

~kautuk

[smacdonald2008]

Watch this Ask the AEM Community Experts that covers these types of setup from real community experts:

"This session will share large scale architectures from the author's experiences with various companies like Cisco, Symantec, and EMC and compare and contrast the architecture across: Infrastructure Architecture Scaling Ecommerce integrations and migration approach from legacy into Adobe Experience Manager, Digital Marketing Cloud Integrations such as personalization, analytics, and DMP."

ATACE 102516 Comparative Architecture Analysis of large scale AEM Installations

[smacdonald2008]

I have asked internal team for more information about this. AEM runs many sites so it can handle this. If you do not hear back - also consider opening a performance ticket - you may need a hotfix. 

[kaushal_mall]

With 4 publishers and dispatchers, 1000 consecutive requests should be a breeze. Are you sure your requests are being cached appropriately? Are you activating pages while running these tests? This might be an issue further down the request chain and not something that should be fixed on the web server level.

[MC_Stuff]

Hi,

Yes, Dispatcher is compatible (worker binary of apache vs the prefork,the new event MPM in Apache 2.4:)   the dispatcher modifies only local, per-thread state, but all global variables are read-only, initialized once and are therefore thread-safe.

Thanks,

[bcop]

Thanks for all the responses they definitely help, we probably need to speak to an Adobe Engineer for Performance Tuning our AEM instance before go-live, its coming up fairly soon.

smacdonald2008​

Can you point me to any hotfixes we might have missed around Performance? As far as I know we have everything up to SP2 (AEM 6.1) and every hotfix except the March 24, 2017 one for OAK here: AEM 6.1 hotfixes​ (but suprisingly enough the JCR queries doesn't seem to be our bottleneck, no single query is even taking more than 400ms, I was hoping to see some smoking guns in the AEM monitoring/diagnostics tools but nothing).

@kaushal_mall

No, we are not really activating many pages. Our content won't be updated in PROD that often so its only a handful of tiny changes going on simultaneously with the Performance Testing. We are not bothering to Performance Test our Author at all since we won't have that many users in there (< 50). We had a caching problem with static images some how bad quotes got into the "Cache-Control" header, but its been fixed and image caching seems to be fully working again. Don't suppose you have any other ideas? If it helps, we have a lot of external dependencies on 3rd party APIs due to the nature of our applications, that's all our Thread Dumps are telling Adobe right now, but we are sure there's more to the story, hitting the APIs in isolation is not performing as badly as the end-to-end test cases.

We are in an Active-Active (Primary Data Center & Backup Data Center) arrangement with 2 Dispatchers, 2 Publishers in each, and a single Author with a cold backup Author ready to get flipped on.

Our performance troubles have been across the board, Authors are complaining it is slow and the website is just barely meeting SLAs without any Performance Testing scripts running. Once we fire up JMeter towards our (I thought relatively modest) concurrent threads/users goal of 2500, it croaks before hitting 1000.

[edgarn27224967]

As someone mentioned earlier - how much caching is being done with dispatcher's disk cache?

You really want most of the requests to hit the cache and not have to hit AEM itself more than once.

And of course I assume apache has been tuned for number of children/workers, etc.

I'm assuming AEM is using TarMK and not mongo for the repository as that would be a big performance hit to AEM w/o a fair amount of tuning.

[bcop]

edgarn27224967​
The caching was broken during our first round of Performance Testing due to a bad character in Cache-Control header, but luckily we noticed it and fixed that issue, just before I initially started this forum discussion.

It should be working fine now but am no AEM engineering/performance optimization expert. Definitely the Publish instance CPU availability has improved, yet the log still shows tons of errors. I'm not sure about tuning Apache either for number of children/workers but I was suspecting the thread pool settings for "oak-observation" that currently say "Min Pool Size" 20 and "Max Pool Size" 20 for the "Apache Sling Thread Pool Configuration" seemed odd and could be a limiting factor.

Yes, AEM is using TarMK and not MongoDB.

[Opkar_Gill]

Hi there,

it seems you have already been in touch with daycare, so you may well have already tried my suggestions:

Have you determined what is the bottleneck on your instances? CPU, IO, Memory?

What are your slowest requests using rlog.jar, you said performance is poor on the author instance as well, this should help to identify why.

When you say you are calling external API's is this from the browser or from the publish instance?

You could also benchmark base AEM installs using tough day to see how much traffic a base install can handle.

Regards,

Opkar

[0] Best Practices for Performance Testing - docs.adobe.com