Heap dump analysis on aem server

[apolu]

Hi All,

Frequently we are observing one of the publisher is going down with java.lang.OutOfMemoryError: Java heap space error.

From the HeapDump analysis using Eclipse MAT tool, we got below Problem Suspects stack trace..

Problem Suspect 1

One instance of "org.apache.jackrabbit.oak.segment.CachingSegmentReader" loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x743115ad0" occupies 251,720,416 (11.62%) bytes. The memory is accumulated in one instance of "org.apache.jackrabbit.oak.cache.CacheLIRS$Segment[]" loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x7425fd5e0".

Keywords
org.apache.jackrabbit.oak.segment.CachingSegmentReader
org.apache.jackrabbit.oak.cache.CacheLIRS$Segment[]
org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x743115ad0
org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x7425fd5e0

Problem Suspect 2

One instance of "org.apache.jackrabbit.oak.cache.CacheLIRS" loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x76d76f610" occupies 220,491,096 (10.18%) bytes. The memory is accumulated in one instance of "org.apache.jackrabbit.oak.cache.CacheLIRS$Segment[]" loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x76d76f610".

Keywords
org.apache.jackrabbit.oak.cache.CacheLIRS
org.apache.jackrabbit.oak.cache.CacheLIRS$Segment[]
org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x76d76f610

Details »

Problem Suspect 3

17,374 instances of "org.apache.jackrabbit.oak.segment.Segment", loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x743115ad0" occupy 231,538,184 (10.69%) bytes. These instances are referenced from one instance of "com.google.common.cache.LocalCache$Segment[]", loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x74301f6f0"

Keywords
org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x74301f6f0
com.google.common.cache.LocalCache$Segment[]
org.apache.jackrabbit.oak.segment.Segment
org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x743115ad0

Problem Suspect 4

15,729 instances of "org.apache.jackrabbit.oak.segment.Segment", loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x76d86aba0" occupy 220,611,808 (10.18%) bytes. These instances are referenced from one instance of "com.google.common.cache.LocalCache$Segment[]", loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x74301f6f0"

Keywords
org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x74301f6f0
com.google.common.cache.LocalCache$Segment[]
org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x76d86aba0
org.apache.jackrabbit.oak.segment.Segment

Can someone please go through above and provide me the suggestion what exactly caused from the above stack trace, i mean need detailed explanation to convey with dev team

Thanks,

ANR

[Phoenics]

Hello ANR,

The information you have shared below is insufficient. However based on  the information shared below i could suggest a few things,

1. As the issue you are facing is only on on instance & Not all . Hence it may be worth doing a revision cleanup  on this instance.

2. The Leak suspects don't generally reveal component / human readable issue information. A rather easier way of root cause analysis using the heapdumps is to  use the dominator tree which gives a much detailed level info on the component wise issue root cause.

[arturl43391132]

Hello @Phoenics,

You told that dominator tree can say much more than these errors:

We've checked the dominator tree:

Could you please suggest or tell if there something understandable what area should be fixed?

Thanks in advance.

[berliant · Accepted solution]

MAT did not point to the right problems. org.apache.jackrabbit.oak.segment.CachingSegmentReader is a standard AEM activity. It constantly accesses data*.tar files, hence MAT pointed to those the most common operations.

Your heap dump needs a deeper analysis. You need to open a support case.

[apolu]

Hi @berliant, thanks for the update, we found memory consumption exhausted due to large size of images uploaded into system and the details we got from error.log and as well some clue from dominator tree view shows the largest objects in the heap dump.

Thanks,

ANR