Memory leak:Getting heap dump at my AEM publisher

Avatar

Avatar
Give Back 3
Level 2
swapnilu7465684
Level 2

Likes

4 likes

Total Posts

5 posts

Correct reply

0 solutions
Top badges earned
Give Back 3
Give Back
Ignite 1
Boost 3
Boost 1
View profile

Avatar
Give Back 3
Level 2
swapnilu7465684
Level 2

Likes

4 likes

Total Posts

5 posts

Correct reply

0 solutions
Top badges earned
Give Back 3
Give Back
Ignite 1
Boost 3
Boost 1
View profile
swapnilu7465684
Level 2

08-01-2019

Hi Team,

I am getting heap dump and GC on my AEM server when doing soak test the following is the report.

Problem Suspect 1

One instance of "org.apache.sling.models.impl.ModelAdapterFactory" loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoaderJava5 @ 0x5c7e3e280" occupies 214,336,312 (29.68%) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentHashMap$Node[]" loaded by "<system class loader>".

Keywords
java.util.concurrent.ConcurrentHashMap$Node[]
org.apache.felix.framework.BundleWiringImpl$BundleClassLoaderJava5 @ 0x5c7e3e280
org.apache.sling.models.impl.ModelAdapterFactory

Details »

Problem Suspect 2

10,992 instances of "org.apache.jackrabbit.oak.segment.Segment", loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoaderJava5 @ 0x5c1c84e10" occupy 79,999,744 (11.08%) bytes. These instances are referenced from one instance of "com.google.common.cache.LocalCache$Segment[]", loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoaderJava5 @ 0x5c1c77e70"

Keywords
org.apache.felix.framework.BundleWiringImpl$BundleClassLoaderJava5 @ 0x5c1c77e70
com.google.common.cache.LocalCache$Segment[]
org.apache.felix.framework.BundleWiringImpl$BundleClassLoaderJava5 @ 0x5c1c84e10
org.apache.jackrabbit.oak.segment.Segment

Can someone tell me how to proceeds

Replies

Avatar

Avatar
Give Back 10
Level 3
wimsymons
Level 3

Likes

14 likes

Total Posts

49 posts

Correct reply

7 solutions
Top badges earned
Give Back 10
Give Back 5
Give Back 3
Give Back
Ignite 3
View profile

Avatar
Give Back 10
Level 3
wimsymons
Level 3

Likes

14 likes

Total Posts

49 posts

Correct reply

7 solutions
Top badges earned
Give Back 10
Give Back 5
Give Back 3
Give Back
Ignite 3
View profile
wimsymons
Level 3

09-01-2019

According to what I see, is that you only have 800mb of heap space. Seems to me you did not pass Xms and Xmx parameters to your publish instance. A production instance needs more than 1gb ram. Give it some more breathing room. For example 4gb.

Avatar

Avatar
Give Back 3
Level 2
swapnilu7465684
Level 2

Likes

4 likes

Total Posts

5 posts

Correct reply

0 solutions
Top badges earned
Give Back 3
Give Back
Ignite 1
Boost 3
Boost 1
View profile

Avatar
Give Back 3
Level 2
swapnilu7465684
Level 2

Likes

4 likes

Total Posts

5 posts

Correct reply

0 solutions
Top badges earned
Give Back 3
Give Back
Ignite 1
Boost 3
Boost 1
View profile
swapnilu7465684
Level 2

09-01-2019

Hi Thanks for your response

we have started JVM as per below

java -server -Xms8192m -Xmx8192m -XX:MaxPermSize=256M -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -XX:+UnlockDiagnosticVMOptions -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -Xloggc:/mountpoint2/aem/crx-quickstart/logs/gc-2019_01_05-09_08.log -Dsling.run.modes=publish,uat,standalone,nosamplecontent,crx3,crx3tar -jar crx-quickstart/app/cq-quickstart-6.3.0-standalone-quickstart.jar start -c crx-quickstart -i launchpad -p 61003 -Dsling.properties=conf/sling.properties

Avatar

Avatar
Give Back 3
Level 2
swapnilu7465684
Level 2

Likes

4 likes

Total Posts

5 posts

Correct reply

0 solutions
Top badges earned
Give Back 3
Give Back
Ignite 1
Boost 3
Boost 1
View profile

Avatar
Give Back 3
Level 2
swapnilu7465684
Level 2

Likes

4 likes

Total Posts

5 posts

Correct reply

0 solutions
Top badges earned
Give Back 3
Give Back
Ignite 1
Boost 3
Boost 1
View profile
swapnilu7465684
Level 2

09-01-2019

Kindly tell me if anything need to change in above start up script.

Avatar

Avatar
Give Back 10
Level 3
wimsymons
Level 3

Likes

14 likes

Total Posts

49 posts

Correct reply

7 solutions
Top badges earned
Give Back 10
Give Back 5
Give Back 3
Give Back
Ignite 3
View profile

Avatar
Give Back 10
Level 3
wimsymons
Level 3

Likes

14 likes

Total Posts

49 posts

Correct reply

7 solutions
Top badges earned
Give Back 10
Give Back 5
Give Back 3
Give Back
Ignite 3
View profile
wimsymons
Level 3

09-01-2019

Strange. Looks legit to me.

Can you take a look at http://<youraemhost>:4502/system/console/memoryusage (the URL, not the path) and paste the content?

That should show your current memory configuration and free space.

Avatar

Avatar
Give Back 3
Level 2
swapnilu7465684
Level 2

Likes

4 likes

Total Posts

5 posts

Correct reply

0 solutions
Top badges earned
Give Back 3
Give Back
Ignite 1
Boost 3
Boost 1
View profile

Avatar
Give Back 3
Level 2
swapnilu7465684
Level 2

Likes

4 likes

Total Posts

5 posts

Correct reply

0 solutions
Top badges earned
Give Back 3
Give Back
Ignite 1
Boost 3
Boost 1
View profile
swapnilu7465684
Level 2

09-01-2019

Please see

1663607_pastedImage_0.png

Avatar

Avatar
Give Back 10
Level 3
wimsymons
Level 3

Likes

14 likes

Total Posts

49 posts

Correct reply

7 solutions
Top badges earned
Give Back 10
Give Back 5
Give Back 3
Give Back
Ignite 3
View profile

Avatar
Give Back 10
Level 3
wimsymons
Level 3

Likes

14 likes

Total Posts

49 posts

Correct reply

7 solutions
Top badges earned
Give Back 10
Give Back 5
Give Back 3
Give Back
Ignite 3
View profile
wimsymons
Level 3

09-01-2019

Looks totally fine. (We have similar settings).

But your suspects are strange as they consume so little heap space. (214 MB for suspect 1, 79 MB for suspect 2).

Are you sure you didn't miss something else?

When we have OOM's on AEM, we find suspects holding 6 GB or more.

Avatar

Avatar
Give Back 3
Level 2
swapnilu7465684
Level 2

Likes

4 likes

Total Posts

5 posts

Correct reply

0 solutions
Top badges earned
Give Back 3
Give Back
Ignite 1
Boost 3
Boost 1
View profile

Avatar
Give Back 3
Level 2
swapnilu7465684
Level 2

Likes

4 likes

Total Posts

5 posts

Correct reply

0 solutions
Top badges earned
Give Back 3
Give Back
Ignite 1
Boost 3
Boost 1
View profile
swapnilu7465684
Level 2

09-01-2019

That is why I have to see whether it is a really need to worry and look the solution for it.  The full GC report are as per below

Class NameShallow HeapRetained Heap
org.apache.sling.models.impl.ModelAdapterFactory
  @ 0x5c7ee6ac8
96214336312
org.apache.felix.framework.BundleWiringImpl$BundleClassLoaderJava5
  @ 0x5c948db38
9647600992
org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker
  @ 0x5c24d1dc8
4036948344
org.apache.jackrabbit.oak.cache.CacheLIRS
  @ 0x5c25478a0
6432506632
org.apache.jackrabbit.oak.segment.file.FileStore
  @ 0x5c1de28c8
11229710512
org.apache.jackrabbit.oak.cache.CacheLIRS
  @ 0x5c1e23f30
6421838752
org.apache.felix.framework.BundleWiringImpl$BundleClassLoaderJava5
  @ 0x5c2484e60
9612621072
com.adobe.granite.ui.clientlibs.impl.HtmlLibraryManagerImpl
  @ 0x5c37cc4d0
6410548568
org.apache.sling.scripting.sightly.js.impl.jsapi.SlyBindingsValuesProvider
  @ 0x5c9cb2378
4810136152
org.apache.jackrabbit.oak.segment.SegmentTracker
  @ 0x5c1de2938
247718544

Avatar

Avatar
Boost 50
Employee
WASIL
Employee

Likes

51 likes

Total Posts

386 posts

Correct reply

51 solutions
Top badges earned
Boost 50
Give Back 5
Give Back 3
Give Back 25
Give Back 10
View profile

Avatar
Boost 50
Employee
WASIL
Employee

Likes

51 likes

Total Posts

386 posts

Correct reply

51 solutions
Top badges earned
Boost 50
Give Back 5
Give Back 3
Give Back 25
Give Back 10
View profile
WASIL
Employee

09-01-2019

There have been couple of improvements included in AEM 6.3 with regards to ModelAdapterFactory(6.3.2.1)[2], so please try on latest AEM 6.3 SP + CFP, taking into account the latest Apache Sling Models Implementation[1]

Thanks,

Wasil

[1] Maven Repository: org.apache.sling » org.apache.sling.models.impl

[2] Adobe Experience Manager Help | Release Notes: AEM 6.3 Cumulative Fix Pack

The key highlights of the AEM Cumulative Fix Pack are:

  • The built-in repository (Apache Jackrabbit Oak) is updated to version 1.6.11.
  • Improved rendering of Parsys components in Classic UI.
  • Updated sling models bundles to include character set fix.
  • Made publish to Brand Portal async for all assets types.
  • Updated coralui-component-richtexteditor.git from 0.1.15 to 0.1.16
  • Fixes in show/hide functionality of drop-down component.
  • Enabled image flipping for the Core Image Component.
  • Updated felix http bundles to enable session attributes.
  • Removed cache=true on Sling Models due to memory consumption issues.