Your achievements

Level 1

0% to

Level 2

Tip /
Sign in

Sign in to Community

to gain points, level up, and earn exciting badges like the new
BedrockMission!

Learn More

View all

Sign in to view all badges

Memory leak: CMS Old Gen Heap Issue

Avatar

Avatar
Validate 10
Level 4
sandeepm744005
Level 4

Likes

19 likes

Total Posts

127 posts

Correct Reply

13 solutions
Top badges earned
Validate 10
Validate 1
Ignite 5
Ignite 3
Ignite 1
View profile

Avatar
Validate 10
Level 4
sandeepm744005
Level 4

Likes

19 likes

Total Posts

127 posts

Correct Reply

13 solutions
Top badges earned
Validate 10
Validate 1
Ignite 5
Ignite 3
Ignite 1
View profile
sandeepm744005
Level 4

04-06-2020

Hello,

Environment - AEM 6.4.8

 

Facing frequently CMS Old Gen Heap Issue in production publishers. By reading through the HeapDumps, it is always pointing to CacheLIRS which is taking around 65% of the space and causing issues. 

 

One instance of "org.apache.jackrabbit.oak.cache.CacheLIRS" loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x6c26bc280" occupies 2,666,504,552 (64.64%) bytes. The memory is accumulated in one instance of "org.apache.jackrabbit.oak.cache.CacheLIRS$Segment[]" loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x6c26bc280".

 

Keywords

org.apache.jackrabbit.oak.cache.CacheLIRS

org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @ 0x6c26bc280

org.apache.jackrabbit.oak.cache.CacheLIRS$Segment[]

 

HeapDumps & thread-dumps are pointing towards the below thread. 

 

  •    <Java Local> java.lang.Thread @ 0x6ea66f4e0 HealthCheck Sling/Granite Content Access Check Thread

Do we require the HealthChecks monitoring dashboard in Production Publishers? there are other agents like NewRelic are already installed. 

leaksuspect.JPGCacheLIRS.JPGhealthcheck.JPG

  

  

Replies

Avatar

Avatar
Give Back 50
Employee
berliant
Employee

Likes

207 likes

Total Posts

315 posts

Correct Reply

98 solutions
Top badges earned
Give Back 50
Give Back 5
Give Back 3
Give Back 25
Give Back 10
View profile

Avatar
Give Back 50
Employee
berliant
Employee

Likes

207 likes

Total Posts

315 posts

Correct Reply

98 solutions
Top badges earned
Give Back 50
Give Back 5
Give Back 3
Give Back 25
Give Back 10
View profile
berliant
Employee

04-06-2020

org.apache.jackrabbit.oak.cache.CacheLIRS - is not an issue at all.

I suggest opening a support case and share thread & heap dump along with log files

Avatar

Avatar
Validate 10
Level 4
sandeepm744005
Level 4

Likes

19 likes

Total Posts

127 posts

Correct Reply

13 solutions
Top badges earned
Validate 10
Validate 1
Ignite 5
Ignite 3
Ignite 1
View profile

Avatar
Validate 10
Level 4
sandeepm744005
Level 4

Likes

19 likes

Total Posts

127 posts

Correct Reply

13 solutions
Top badges earned
Validate 10
Validate 1
Ignite 5
Ignite 3
Ignite 1
View profile
sandeepm744005
Level 4

04-06-2020

Thanks for your reply. Adobe Support Case has already been created for this issue, I thought of checking some of the experts opinion in this thread if there are any other pointers in terms of fixing the issue.

Avatar

Avatar
Ignite 1
Employee
aemmarc
Employee

Likes

183 likes

Total Posts

243 posts

Correct Reply

92 solutions
Top badges earned
Ignite 1
Give Back 50
Give Back 5
Give Back 3
Give Back 25
View profile

Avatar
Ignite 1
Employee
aemmarc
Employee

Likes

183 likes

Total Posts

243 posts

Correct Reply

92 solutions
Top badges earned
Ignite 1
Give Back 50
Give Back 5
Give Back 3
Give Back 25
View profile
aemmarc
Employee

04-06-2020

If you aren't, please ensure AEM is running with at least these 4 JVM flags : 
-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
 
Which will have Java write some GC information to disk :
gc.log, gc.log.1, gc.log.8.current etc
 
Provide those logs if you have them to the case you raised.

Avatar

Avatar
Ignite 1
Employee
aemmarc
Employee

Likes

183 likes

Total Posts

243 posts

Correct Reply

92 solutions
Top badges earned
Ignite 1
Give Back 50
Give Back 5
Give Back 3
Give Back 25
View profile

Avatar
Ignite 1
Employee
aemmarc
Employee

Likes

183 likes

Total Posts

243 posts

Correct Reply

92 solutions
Top badges earned
Ignite 1
Give Back 50
Give Back 5
Give Back 3
Give Back 25
View profile
aemmarc
Employee

04-06-2020

If you aren't, please ensure AEM is running with at least these 4 JVM flags : 
-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
 
Which will have Java write some GC information to disk :
gc.log, gc.log.1, gc.log.8.current etc
 
Provide those logs if you have them to the case you raised.

Avatar

Avatar
Validate 10
Level 4
sandeepm744005
Level 4

Likes

19 likes

Total Posts

127 posts

Correct Reply

13 solutions
Top badges earned
Validate 10
Validate 1
Ignite 5
Ignite 3
Ignite 1
View profile

Avatar
Validate 10
Level 4
sandeepm744005
Level 4

Likes

19 likes

Total Posts

127 posts

Correct Reply

13 solutions
Top badges earned
Validate 10
Validate 1
Ignite 5
Ignite 3
Ignite 1
View profile
sandeepm744005
Level 4

04-06-2020

Thanks for your reply. Yes, those flags are present, since it is AMS platform, I do not have access to those logs. CSE has already provided those logs in the Adobe Case. I hope, this issue will be resolved soon.

Avatar

Avatar
Coach
Employee
Jörg_Hoh
Employee

Likes

1,081 likes

Total Posts

3,121 posts

Correct Reply

1,061 solutions
Top badges earned
Coach
Give back 600
Ignite 5
Ignite 3
Ignite 1
View profile

Avatar
Coach
Employee
Jörg_Hoh
Employee

Likes

1,081 likes

Total Posts

3,121 posts

Correct Reply

1,061 solutions
Top badges earned
Coach
Give back 600
Ignite 5
Ignite 3
Ignite 1
View profile
Jörg_Hoh
Employee

06-06-2020

Hi,

 

I am not totally sure if that's a normal behavior. Can you check in /system/console/jmx if the number of "SessionStatistics" entries is growing over time. In that case you have a session leak, which often goes together with memory issues like you described.

Avatar

Avatar
Validate 10
Level 3
arturl43391132
Level 3

Likes

18 likes

Total Posts

68 posts

Correct Reply

10 solutions
Top badges earned
Validate 10
Validate 1
Ignite 1
Give Back
Boost 5
View profile

Avatar
Validate 10
Level 3
arturl43391132
Level 3

Likes

18 likes

Total Posts

68 posts

Correct Reply

10 solutions
Top badges earned
Validate 10
Validate 1
Ignite 1
Give Back
Boost 5
View profile
arturl43391132
Level 3

01-09-2020

Hello @Jörg_Hoh,

Could you please explain what do you mean by "he number of "SessionStatistics" entries is growing over time"? Now I've checked our number and it is 104. If it will increase in some hours, can it be session leak?

 

Thanks.

Avatar

Avatar
Ignite 1
Employee
aemmarc
Employee

Likes

183 likes

Total Posts

243 posts

Correct Reply

92 solutions
Top badges earned
Ignite 1
Give Back 50
Give Back 5
Give Back 3
Give Back 25
View profile

Avatar
Ignite 1
Employee
aemmarc
Employee

Likes

183 likes

Total Posts

243 posts

Correct Reply

92 solutions
Top badges earned
Ignite 1
Give Back 50
Give Back 5
Give Back 3
Give Back 25
View profile
aemmarc
Employee

01-09-2020

104 is fine.

 

This generally is not of concern until the jcr-session count goes into the thousands.

Avatar

Avatar
Give Back
Level 1
narmadha_ganesan
Level 1

Likes

0 likes

Total Posts

1 post

Correct Reply

0 solutions
Top badges earned
Give Back
View profile

Avatar
Give Back
Level 1
narmadha_ganesan
Level 1

Likes

0 likes

Total Posts

1 post

Correct Reply

0 solutions
Top badges earned
Give Back
View profile
narmadha_ganesan
Level 1

27-04-2021

Did you get any solution for this?