Highlighted

Memory leak: CMS Old Gen Heap Issue

Avatar

Avatar

sandeepm744005

Avatar

sandeepm744005

sandeepm744005

04-06-2020

Hello,

Environment - AEM 6.4.8

 

Facing frequently CMS Old Gen Heap Issue in production publishers. By reading through the HeapDumps, it is always pointing to CacheLIRS which is taking around 65% of the space and causing issues. 

 

One instance of "org.apache.jackrabbit.oak.cache.CacheLIRS" loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x6c26bc280" occupies 2,666,504,552 (64.64%) bytes. The memory is accumulated in one instance of "org.apache.jackrabbit.oak.cache.CacheLIRS$Segment[]" loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x6c26bc280".

 

Keywords

org.apache.jackrabbit.oak.cache.CacheLIRS

org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x6c26bc280

org.apache.jackrabbit.oak.cache.CacheLIRS$Segment[]

 

HeapDumps & thread-dumps are pointing towards the below thread. 

 

  •    <Java Local> java.lang.Thread @ 0x6ea66f4e0 HealthCheck Sling/Granite Content Access Check Thread

Do we require the HealthChecks monitoring dashboard in Production Publishers? there are other agents like NewRelic are already installed. 

leaksuspect.JPGCacheLIRS.JPGhealthcheck.JPG

  

  

Replies

Highlighted

Avatar

Avatar

berliant

Employee

Avatar

berliant

Employee

berliant
Employee

04-06-2020

org.apache.jackrabbit.oak.cache.CacheLIRS - is not an issue at all.

I suggest opening a support case and share thread & heap dump along with log files

Highlighted

Avatar

Avatar

sandeepm744005

Avatar

sandeepm744005

sandeepm744005

04-06-2020

Thanks for your reply. Adobe Support Case has already been created for this issue, I thought of checking some of the experts opinion in this thread if there are any other pointers in terms of fixing the issue.

Highlighted

Avatar

Avatar

aemmarc

Employee

Avatar

aemmarc

Employee

aemmarc
Employee

04-06-2020

If you aren't, please ensure AEM is running with at least these 4 JVM flags : 
-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
 
Which will have Java write some GC information to disk :
gc.log, gc.log.1, gc.log.8.current etc
 
Provide those logs if you have them to the case you raised.
Highlighted

Avatar

Avatar

aemmarc

Employee

Avatar

aemmarc

Employee

aemmarc
Employee

04-06-2020

If you aren't, please ensure AEM is running with at least these 4 JVM flags : 
-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
 
Which will have Java write some GC information to disk :
gc.log, gc.log.1, gc.log.8.current etc
 
Provide those logs if you have them to the case you raised.
Highlighted

Avatar

Avatar

sandeepm744005

Avatar

sandeepm744005

sandeepm744005

04-06-2020

Thanks for your reply. Yes, those flags are present, since it is AMS platform, I do not have access to those logs. CSE has already provided those logs in the Adobe Case. I hope, this issue will be resolved soon.

Highlighted

Avatar

Avatar

Jörg_Hoh

Employee

Total Posts

3.0K

Likes

911

Correct Answer

1.0K

Avatar

Jörg_Hoh

Employee

Total Posts

3.0K

Likes

911

Correct Answer

1.0K
Jörg_Hoh
Employee

06-06-2020

Hi,

 

I am not totally sure if that's a normal behavior. Can you check in /system/console/jmx if the number of "SessionStatistics" entries is growing over time. In that case you have a session leak, which often goes together with memory issues like you described.

Highlighted

Avatar

Avatar

arturl43391132

Avatar

arturl43391132

arturl43391132

01-09-2020

Hello @Jörg_Hoh,

Could you please explain what do you mean by "he number of "SessionStatistics" entries is growing over time"? Now I've checked our number and it is 104. If it will increase in some hours, can it be session leak?

 

Thanks.

Highlighted

Avatar

Avatar

aemmarc

Employee

Avatar

aemmarc

Employee

aemmarc
Employee

01-09-2020

104 is fine.

 

This generally is not of concern until the jcr-session count goes into the thousands.