Author performance issues

[boser87]

Hi,

I am running some performance load test on AEM Author instance (AEM6+SP3). The performance test consists of 250 concurrent users performing login, three basic component editing operations into an existing page and page view.

Hardware is the following: 8 CPU, 12 GB (8 GB allocated for AEM).

What happens is the following: even when the number of active threads is 30, we have response time of 18 seconds (.html page/component view) and the response time continues to grow rapidly. When the number of threads goes up to 250, we reach an OutOfMemory.

My doubt is: what is the limit of concurrent authors performing such kind of operations that AEM can sustian with reasonable response time (2, 3 seconds?) independently by the application code?

Thank you.

[Opkar_Gill]

Hi,

I usually provide AEM 50% of the available RAM to start with. With AEM, TarMK can utilise the available memory to run in the repository in memory which can speed things up.

Also, the pages, components that you are testing, are they basic or are you doing any complex in them which would slow the response.

I'd attach a profiler to see what is going on, also take some thread dumps[1] to analyse further to see if it is a product or custom code issue. Did you enable 

-XX:+HeapDumpOnOutOfMemoryError' ?

Regards,

Opkar

[1]https://docs.adobe.com/docs/en/aem/6-0/administer/operations/troubleshoot.html

[2] https://helpx.adobe.com/experience-manager/kb/AnalyzeMemoryProblems.html

[boser87]

Hi Opkar and thank you so much for your insights.

Actually I forgot the off-heap memory feature of TarMK and leaving more space outside of the heap will be another tuning step for me.

Is the 50% setting some kind of heuristic of your specific case, or is that a general common best practice?

Actually I still do not know into the details each component/template, but trying to figure out their complexity (and possible performance leaks) would be the next step.

I will enable the heap dump on memory error as well.

[Jörg_Hoh]

Hi,

how large is your repository? I agree to Opkar, that you should spend more RAM to the machine (not to the heap!) to cache more agressvily on an Oak layer. This should improve your performance massively.

kind regards,
Jörg

[Opkar_Gill]

Hi,

the 50% is a guideline, remember you will not get this right first time. It is a starting point from which to start your performance testing, which will determine the correct value for your situation.

Another thing to try is to run your tests against the OOTB pages and components, to see if you have identified an issue in the product or whether it is an issue with the code.

I take it you are adding a wait time between each invocation of a request to simulate real users?

Regards,

Opkar

[boser87]

Hi guys,

actually I thought to use Tough Day by Adobe to measure the performance on the OOTB code and to understand of the hardware sizing is proper. I would not have thought to use Tough Day to detect AEM 6 performance leaks :-) but just for wrong hardware sizing.

The size of the repository is 230 G.

The test counts 250 concurrent users with a ramp-up period of 250 seconds. I know this is quite aggressive as a test, but we are trying to find an upper limit of concurrent users for the author where the response time is reasonable. Until now, I have tested page creation and component editing. I still have to test assets ingestion and replication.

These are averages between all the concurrent 250 users; time to browse the page to edit 1867 ms, first bunch of editing operations is 2274 ms, second 5507 ms, third 2429 ms, fourth is 2201 ms, last one is 1728 ms. Editing operations are just a matter of adding the component to the page and edit it.

Above are the results after memory tuning as you suggested.

[Opkar_Gill · Accepted solution]

Hi,

I would run your tests in phases, start with 10 run your full tests, then 20, then 30 and so on. That way you can determine at which point performance starts dropping off. Starting off with 250 seems aggressive for a single instance that has not been tuned.

Here is a paper on some performance testing a partner ran for AEM 6(although it was as a comparison to 5.6)

Regards,

Opkar

[1] http://labs.6dglobal.com/blog/2015-09-02/cq-56-tipping-point/#.VeeugmBY7mM.twitter