Your achievements

Level 1

0% to

Level 2

Tip /
Sign in

Sign in to Community

to gain points, level up, and earn exciting badges like the new
BedrockMission!

Learn More

View all

Sign in to view all badges

Heap dump analysis on aem server

Avatar

Avatar
Validate 1
Level 2
apolu
Level 2

Likes

6 likes

Total Posts

22 posts

Correct Reply

1 solution
Top badges earned
Validate 1
Ignite 3
Ignite 1
Give Back
Boost 5
View profile

Avatar
Validate 1
Level 2
apolu
Level 2

Likes

6 likes

Total Posts

22 posts

Correct Reply

1 solution
Top badges earned
Validate 1
Ignite 3
Ignite 1
Give Back
Boost 5
View profile
apolu
Level 2

17-01-2020

Hi All,

 

Frequently we are observing one of the publisher is going down with java.lang.OutOfMemoryError: Java heap space error.

From the HeapDump analysis using Eclipse MAT tool, we got below Problem Suspects stack trace..

Problem Suspect 1

One instance of "org.apache.jackrabbit.oak.segment.CachingSegmentReader" loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x743115ad0" occupies 251,720,416 (11.62%) bytes. The memory is accumulated in one instance of "org.apache.jackrabbit.oak.cache.CacheLIRS$Segment[]" loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x7425fd5e0".

Keywords
org.apache.jackrabbit.oak.segment.CachingSegmentReader
org.apache.jackrabbit.oak.cache.CacheLIRS$Segment[]
org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x743115ad0
org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x7425fd5e0

Problem Suspect 2

One instance of "org.apache.jackrabbit.oak.cache.CacheLIRS" loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x76d76f610" occupies 220,491,096 (10.18%) bytes. The memory is accumulated in one instance of "org.apache.jackrabbit.oak.cache.CacheLIRS$Segment[]" loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x76d76f610".

Keywords
org.apache.jackrabbit.oak.cache.CacheLIRS
org.apache.jackrabbit.oak.cache.CacheLIRS$Segment[]
org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x76d76f610

Details »

Problem Suspect 3

17,374 instances of "org.apache.jackrabbit.oak.segment.Segment", loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x743115ad0" occupy 231,538,184 (10.69%) bytes. These instances are referenced from one instance of "com.google.common.cache.LocalCache$Segment[]", loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x74301f6f0"

Keywords
org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x74301f6f0
com.google.common.cache.LocalCache$Segment[]
org.apache.jackrabbit.oak.segment.Segment
org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x743115ad0

 

Problem Suspect 4

15,729 instances of "org.apache.jackrabbit.oak.segment.Segment", loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x76d86aba0" occupy 220,611,808 (10.18%) bytes. These instances are referenced from one instance of "com.google.common.cache.LocalCache$Segment[]", loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x74301f6f0"

Keywords
org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x74301f6f0
com.google.common.cache.LocalCache$Segment[]
org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x76d86aba0
org.apache.jackrabbit.oak.segment.Segment

 

Can someone please go through above and provide me the suggestion what exactly caused from the above stack trace, i mean need detailed explanation to convey with dev team

 

Thanks,

ANR

Accepted Solutions (1)

Accepted Solutions (1)

Avatar

Avatar
Give Back 50
Employee
berliant
Employee

Likes

207 likes

Total Posts

315 posts

Correct Reply

98 solutions
Top badges earned
Give Back 50
Give Back 5
Give Back 3
Give Back 25
Give Back 10
View profile

Avatar
Give Back 50
Employee
berliant
Employee

Likes

207 likes

Total Posts

315 posts

Correct Reply

98 solutions
Top badges earned
Give Back 50
Give Back 5
Give Back 3
Give Back 25
Give Back 10
View profile
berliant
Employee

17-01-2020

MAT did not point to the right problems. org.apache.jackrabbit.oak.segment.CachingSegmentReader is a standard AEM activity. It constantly accesses data*.tar files, hence MAT pointed to those the most common operations.

Your heap dump needs a deeper analysis. You need to open a support case.

Answers (1)

Answers (1)

Avatar

Avatar
Affirm 1
Level 1
Phoenics
Level 1

Likes

0 likes

Total Posts

3 posts

Correct Reply

1 solution
Top badges earned
Affirm 1
View profile

Avatar
Affirm 1
Level 1
Phoenics
Level 1

Likes

0 likes

Total Posts

3 posts

Correct Reply

1 solution
Top badges earned
Affirm 1
View profile
Phoenics
Level 1

17-01-2020

Hello ANR,

 

The information you have shared below is insufficient. However based on  the information shared below i could suggest a few things,

 

1. As the issue you are facing is only on on instance & Not all . Hence it may be worth doing a revision cleanup  on this instance.

2. The Leak suspects don't generally reveal component / human readable issue information. A rather easier way of root cause analysis using the heapdumps is to  use the dominator tree which gives a much detailed level info on the component wise issue root cause.

 

Screenshot 2020-01-17 at 14.09.14.png