AEM author heap size error

[narendiran_ravi]

We are using AEM 6.2 author instance and we ran into lot of performance issues lately. From the logs we found that the error is related to heap memory so we requested for a dump from server team so the we can analyse the memory leak issue

Hardware configurations:

Repository size: 250 GB

Free space : 400 GB

RAM: 64 GB

Heap was 12 GB and we decided to increase to 14GB as we assumed the out of memory is because of the heap size but still we faced slowness in AEM author and CPU load is increasing rapidly.

While we analyse the heap dump, I would like to know how to determine the size requirement of the RAM for the machine and Heap memory for AEM.

Also any suggestions/ links which we can follow to determine the root cause of slowness would be much appreciated. Please let me know if more details are required

[Kiran_Vedantam · Accepted solution]

Hi @narendiran_ravi,

For hardware requirements please check this link. 

Regarding the system performance, please perform the maintenance activities listed here as per your requirement.

Also, regarding the performance, please check your code for the below issues.

1. Reduce the session leaks
   1. You can achieve this by reducing the use of the JCR session
2. Closing the ResourceResolver whenever it is opened
3. Prefer the ResourceResolver over the JCR Session
4. Do not share nodes, resources, and pages between threads and requests

Hope this helps.

Thanks,

Kiran Vedantam

[narendiran_ravi]

Thanks Kiran for your answer

[Jörg_Hoh]

Hi,

it's hard to give proper recommendations based on the given information. 12G of heap is plenty for most usecases, and not enough if you deal with very large assets. So increasing the heap size might or might not improve the situation, that totally depends on your application and your workload.

To understand the root cause for slowness, I would use the following approach:

* enable Garbage Collection logging on the JVM and monitor the GC behavior.

* quantify "slowness": Determine "slow" requests and make it actionable (what makes a slow request "slow"? Response times over 1 second?)

* check the periodic threaddump and search for patterns; correlate it with observed slowness (slow requests). It's definitely best if you could reproduce this in a lower environment and attach a profile to it to understand where the time is spend.

There are tons of ways, how and why requests get slow; a few of them:

* JCR queries, which are not served by an index (you will see warnings in the logs)

* overly complex logic with lots of unnecessary repository access

* requests, which trigger itself external requests to 3rd party systems.

HTH,

Jörg

[narendiran_ravi]

Hi , Thanks for your reply.

[narendiran_ravi]

I have posted more details . Please check if you can give suggestions based on that. Thanks in Advance

[narendiran_ravi]

Adding more information :

We are using below features in our instance

• Lot of sling jobs running
• Product data imported from xml and pages are created based on this. This happens during off business hours.
• Asset ingestion is high both from content authors and through auto upload
• Assets are auto published as we are autotagging them based on some business rules
• As mentioned heap size  before heap size and direct memory allocated for AEM repository is  10 g earlier it was 8G -Xmx14g -Xms14g -XX:MaxDirectMemorySize=10g

I came across an article -https://dekesmith.info/index.php/2015/03/26/think-outside-the-heap-with-aem-6-and-jackrabbit-oak-tar...  which explains that segmenstore loads a part of it into direct memory so our segmenstore is 250 GB but having only 10G allocated in RAM is it fine ? Is there a thumb rule based on which we can balance this heap and directmemory for repository. In all articles I have checked I couldn't get more information on this.

Also we have total 64 GB of RAM and 8 CPU cores currently in the server and we have a spring application(for search)and solr also running in the same server. We understand if we change the heap then we may need to increase these parameters as well but I would like more inputs from experts here how to determine this.

[Jörg_Hoh]

Parsing large XML files with a SAX parser can definitely be a memory hog and consume large amount of heap. 

You need to understand much better, when you are running short of memory, and based on that you can start with both heap analysis and more testing. 

And regarding performance: Yes, you should have an amount of free RAM on system which is not consumed by Java heap, and which can be used as cache. But only a profiler run (or a good observation on an OS level) can tell if this is really a bottleneck. You are performing a number of operations, which all could be implemented sub-par and which could cause the observed behavior, that is impossible to diagnose here. I can only share some recommendations how you can find out yourself.

regards,

Jörg