Open sessions - JMX

Avatar

Avatar

Veena_Vikram

MVP

Avatar

Veena_Vikram

MVP

Veena_Vikram
MVP

03-08-2018

Hi All

    I am trying to figure out the code which has a session leak. I am following Adobe's documentation.Check and analyze if JCR session leaks in your AEM instance

But I have few models where a resourceResolver is injected using

@SlingObject

    private ResourceResolver resolver;

    I can see few class reference in JMX which has these classes mentioned in the stack trace. How will we handle these resolvers? Will these be closed automatically ?

TIA

Veena

Accepted Solutions (1)

Accepted Solutions (1)

Avatar

Avatar

Jörg_Hoh

Employee

Total Posts

3.0K

Likes

953

Correct Reply

1.0K

Avatar

Jörg_Hoh

Employee

Total Posts

3.0K

Likes

953

Correct Reply

1.0K
Jörg_Hoh
Employee

07-08-2018

I just checked [1] and I haven't found a way that the ResourceTypeResolver (which seems to leak in your case) is actually leaking. It is always closed (line 665ff).

I cannot explain the fact that a lot of these session are actually open (it would be good if you could verify that by a heapdump), and that it's not a problem of the garbage collection.

[1] sling-org-apache-sling-resourceresolver/ResourceResolverControl.java at org.apache.sling.resourceres...

Answers (14)

Answers (14)

Avatar

Avatar

Veena_Vikram

MVP

Avatar

Veena_Vikram

MVP

Veena_Vikram
MVP

12-09-2018

Thanks a lot Jorg . I was struggling to figure out those logs.

Avatar

Avatar

Jörg_Hoh

Employee

Total Posts

3.0K

Likes

953

Correct Reply

1.0K

Avatar

Jörg_Hoh

Employee

Total Posts

3.0K

Likes

953

Correct Reply

1.0K
Jörg_Hoh
Employee

05-09-2018

Below that (and not contained in the screenshot) there is a field "stacktrace", which should be filled with the stacktrace. This stacktraces are only created when more than 1000 sessions are opened (default), so you should check the youngest sessions, they should contain these stacks.

Avatar

Avatar

Veena_Vikram

MVP

Avatar

Veena_Vikram

MVP

Veena_Vikram
MVP

05-09-2018

There are few sessions like below (around 6k) opened in STAGE. How can I trace from where this sessions are opened ?

1564749_pastedImage_0.png

Avatar

Avatar

Veena_Vikram

MVP

Avatar

Veena_Vikram

MVP

Veena_Vikram
MVP

07-08-2018

current prod servers are 6.2 , we will be moving to 6.4 in may be 4 months.

Avatar

Avatar

Jörg_Hoh

Employee

Total Posts

3.0K

Likes

953

Correct Reply

1.0K

Avatar

Jörg_Hoh

Employee

Total Posts

3.0K

Likes

953

Correct Reply

1.0K
Jörg_Hoh
Employee

06-08-2018

what AEM version are you using?

Avatar

Avatar

Veena_Vikram

MVP

Avatar

Veena_Vikram

MVP

Veena_Vikram
MVP

06-08-2018

Thanks Jörg Hoh​ .

A little insight to the situation right now ,The situation is bit bad with our prod servers right now. Since business reported slowness , we went back to Adobe and they informed us to check JMX for open sessions and fix the code. The sessions and resourceResolvers in our code was already taken care. I checked JMX and yesterday alone there was >11000 Session objects (note that this is the count of sessions opened in last 24 hours ) tracked in JMX. I read that anything above 500 is bad. I checked first 50 manually and all showed the above stack trace. Then I checked randomly from all these 11000 and few didn't had a stack trace and the rest were all these and few from one old project which was still in prod (this will be deleted).

Avatar

Avatar

Jörg_Hoh

Employee

Total Posts

3.0K

Likes

953

Correct Reply

1.0K

Avatar

Jörg_Hoh

Employee

Total Posts

3.0K

Likes

953

Correct Reply

1.0K
Jörg_Hoh
Employee

06-08-2018

A little bit of background: A resource resolver internally holds 2 session! One "real" session through which all actions are performed, and this session is running with user privileges. And an additional one with elevated privileges which is used for resolving privileges and such (which is typically not always directly accesible by the user session).

I remember an issue with this additional session but I don't recall any details.

Also it could be temporary glitch with Garbage Collection and lazily unregistering MBeans, I saw reports on the sling JIRA as well. Maybe you want to check if all (well, kind of) sessions have the same threaddump for opening and then report it to Adobe support.

Avatar

Avatar

Veena_Vikram

MVP

Avatar

Veena_Vikram

MVP

Veena_Vikram
MVP

06-08-2018

Thanks Jörg Hoh​ . As you can see from the method signature, we are just passing a Page and String as arguments. But the real culprit is the method isResoureType() in Resource ("The Adobe AEM Quickstart and Web Application.") . If you see the method details in API, it is saying that

So if we go by above does that mean this method is opening a resourceResolver and is never closed ?

Avatar

Avatar

Jörg_Hoh

Employee

Total Posts

3.0K

Likes

953

Correct Reply

1.0K

Avatar

Jörg_Hoh

Employee

Total Posts

3.0K

Likes

953

Correct Reply

1.0K
Jörg_Hoh
Employee

06-08-2018

What version of AEM are you using?

This observation seems somehow related to [1], although this did not mention a leak of JCR sessions. IIRC there was another issue in that area which actually dealt with a leak loosing the ResourceTypeResolver RR, but I don't find it rightaway. How do you obtain the the resourceResolver you feed into this "isResourceType" call? I am asking because of [2].

[1] [SLING-3285] ResourceResolver.isResourceType() creates admin session for every call with super type,...

[2] [SLING-7591] Session leak when a ResourceResolver is created from a Session then gets cloned - ASF J...

Avatar

Avatar

Veena_Vikram

MVP

Avatar

Veena_Vikram

MVP

Veena_Vikram
MVP

06-08-2018

@Jörg Hoh

Any help here ?

Avatar

Avatar

Veena_Vikram

MVP

Avatar

Veena_Vikram

MVP

Veena_Vikram
MVP

06-08-2018

Thanks Arun , Jorg . The sessions and resourceResolver I opened is closed. But yet there are around 6500 open sessions in P1 which I am not sure what is causing. I tried manually opening each session detail given in JMX and tried to check the stack trace. Basically the stack trace points to me a class , where I have not opened any sessions implicitly, rather I can only see this line

1542958_pastedImage_0.png

So as per the stack trace page.getContentResource().isResourceType(resourceType) is one culprit in the chain whcih opened the session. Please see the stack trace below.

java.lang.Exception: The session was opened here:

at org.apache.jackrabbit.oak.jcr.session.SessionStats.<init>(SessionStats.java:84)

at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.<init>(SessionDelegate.java:150)

at org.apache.jackrabbit.oak.jcr.repository.RepositoryImpl$1.<init>(RepositoryImpl.java:301)

at org.apache.jackrabbit.oak.jcr.repository.RepositoryImpl.createSessionDelegate(RepositoryImpl.java:299)

at org.apache.jackrabbit.oak.jcr.repository.RepositoryImpl.login(RepositoryImpl.java:281)

at com.adobe.granite.repository.impl.CRX3RepositoryImpl.login(CRX3RepositoryImpl.java:94)

at com.adobe.granite.repository.impl.SlingRepositoryImpl$2.run(SlingRepositoryImpl.java:112)

at com.adobe.granite.repository.impl.SlingRepositoryImpl$2.run(SlingRepositoryImpl.java:104)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)

at com.adobe.granite.repository.impl.SlingRepositoryImpl.createAdministrativeSession(SlingRepositoryImpl.java:104)

at org.apache.sling.jcr.base.AbstractSlingRepository2.loginAdministrative(AbstractSlingRepository2.java:396)

at org.apache.sling.jcr.resource.internal.helper.jcr.JcrProviderStateFactory.createProviderState(JcrProviderStateFactory.java:82)

at org.apache.sling.jcr.resource.internal.helper.jcr.JcrResourceProvider.authenticate(JcrResourceProvider.java:267)

at org.apache.sling.jcr.resource.internal.helper.jcr.JcrResourceProvider.authenticate(JcrResourceProvider.java:78)

at org.apache.sling.resourceresolver.impl.providers.stateful.ProviderManager.authenticate(ProviderManager.java:161)

at org.apache.sling.resourceresolver.impl.providers.stateful.ProviderManager.getOrCreateProvider(ProviderManager.java:87)

at org.apache.sling.resourceresolver.impl.providers.stateful.ProviderManager.authenticateAll(ProviderManager.java:129)

at org.apache.sling.resourceresolver.impl.ResourceResolverImpl.createControl(ResourceResolverImpl.java:154)

at org.apache.sling.resourceresolver.impl.ResourceResolverImpl.<init>(ResourceResolverImpl.java:116)

at org.apache.sling.resourceresolver.impl.ResourceResolverImpl.<init>(ResourceResolverImpl.java:110)

at org.apache.sling.resourceresolver.impl.CommonResourceResolverFactoryImpl.getResourceResolverInternal(CommonResourceResolverFactoryImpl.java:258)

at org.apache.sling.resourceresolver.impl.CommonResourceResolverFactoryImpl.getAdministrativeResourceResolver(CommonResourceResolverFactoryImpl.java:141)

at org.apache.sling.resourceresolver.impl.helper.ResourceResolverControl.getResourceTypeResourceResolver(ResourceResolverControl.java:689)

at org.apache.sling.resourceresolver.impl.helper.ResourceResolverControl.getParentResourceType(ResourceResolverControl.java:713)

at org.apache.sling.resourceresolver.impl.ResourceResolverImpl.getParentResourceType(ResourceResolverImpl.java:1304)

at org.apache.sling.resourceresolver.impl.ResourceResolverImpl.getParentResourceType(ResourceResolverImpl.java:1293)

at org.apache.sling.resourceresolver.impl.ResourceResolverImpl.isResourceType(ResourceResolverImpl.java:1321)

at org.apache.sling.api.resource.AbstractResource.isResourceType(AbstractResource.java:123)

at com.<****>.aem.website.v1.core.utils.PageUtil.isResourceType(PageUtil.java:147)

Hope this will help more

Avatar

Avatar

smacdonald2008

Total Posts

12.7K

Likes

1.4K

Correct Reply

2.3K

Avatar

smacdonald2008

Total Posts

12.7K

Likes

1.4K

Correct Reply

2.3K
smacdonald2008

04-08-2018

NIce responses Joerg and Arun!

Avatar

Avatar

Jörg_Hoh

Employee

Total Posts

3.0K

Likes

953

Correct Reply

1.0K

Avatar

Jörg_Hoh

Employee

Total Posts

3.0K

Likes

953

Correct Reply

1.0K
Jörg_Hoh
Employee

03-08-2018

Applying Arun's statement to your question: Do not close sessions which are injected to your sling model.

Avatar

Avatar

Arun_Patidar

MVP

Total Posts

3.0K

Likes

1.1K

Correct Reply

840

Avatar

Arun_Patidar

MVP

Total Posts

3.0K

Likes

1.1K

Correct Reply

840
Arun_Patidar
MVP

03-08-2018

Hi,

If you open ResourceResolver then you need to close it.

Please check Closing Resource Resolver

Thanks

Arun