Open sessions - JMX

[VeenaVikraman]

Hi All

    I am trying to figure out the code which has a session leak. I am following Adobe's documentation.Check and analyze if JCR session leaks in your AEM instance

But I have few models where a resourceResolver is injected using

@SlingObject

    private ResourceResolver resolver;

    I can see few class reference in JMX which has these classes mentioned in the stack trace. How will we handle these resolvers? Will these be closed automatically ?

TIA

Veena

[arunpatidar]

Hi,

If you open ResourceResolver then you need to close it.

Please check Closing Resource Resolver

Thanks

Arun

[Jörg_Hoh]

Applying Arun's statement to your question: Do not close sessions which are injected to your sling model.

[smacdonald2008]

NIce responses Joerg and Arun!

[VeenaVikraman]

Thanks Arun , Jorg . The sessions and resourceResolver I opened is closed. But yet there are around 6500 open sessions in P1 which I am not sure what is causing. I tried manually opening each session detail given in JMX and tried to check the stack trace. Basically the stack trace points to me a class , where I have not opened any sessions implicitly, rather I can only see this line

So as per the stack trace page.getContentResource().isResourceType(resourceType) is one culprit in the chain whcih opened the session. Please see the stack trace below.

java.lang.Exception: The session was opened here:

at org.apache.jackrabbit.oak.jcr.session.SessionStats.<init>(SessionStats.java:84)

at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.<init>(SessionDelegate.java:150)

at org.apache.jackrabbit.oak.jcr.repository.RepositoryImpl$1.<init>(RepositoryImpl.java:301)

at org.apache.jackrabbit.oak.jcr.repository.RepositoryImpl.createSessionDelegate(RepositoryImpl.java:299)

at org.apache.jackrabbit.oak.jcr.repository.RepositoryImpl.login(RepositoryImpl.java:281)

at com.adobe.granite.repository.impl.CRX3RepositoryImpl.login(CRX3RepositoryImpl.java:94)

at com.adobe.granite.repository.impl.SlingRepositoryImpl$2.run(SlingRepositoryImpl.java:112)

at com.adobe.granite.repository.impl.SlingRepositoryImpl$2.run(SlingRepositoryImpl.java:104)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)

at com.adobe.granite.repository.impl.SlingRepositoryImpl.createAdministrativeSession(SlingRepositoryImpl.java:104)

at org.apache.sling.jcr.base.AbstractSlingRepository2.loginAdministrative(AbstractSlingRepository2.java:396)

at org.apache.sling.jcr.resource.internal.helper.jcr.JcrProviderStateFactory.createProviderState(JcrProviderStateFactory.java:82)

at org.apache.sling.jcr.resource.internal.helper.jcr.JcrResourceProvider.authenticate(JcrResourceProvider.java:267)

at org.apache.sling.jcr.resource.internal.helper.jcr.JcrResourceProvider.authenticate(JcrResourceProvider.java:78)

at org.apache.sling.resourceresolver.impl.providers.stateful.ProviderManager.authenticate(ProviderManager.java:161)

at org.apache.sling.resourceresolver.impl.providers.stateful.ProviderManager.getOrCreateProvider(ProviderManager.java:87)

at org.apache.sling.resourceresolver.impl.providers.stateful.ProviderManager.authenticateAll(ProviderManager.java:129)

at org.apache.sling.resourceresolver.impl.ResourceResolverImpl.createControl(ResourceResolverImpl.java:154)

at org.apache.sling.resourceresolver.impl.ResourceResolverImpl.<init>(ResourceResolverImpl.java:116)

at org.apache.sling.resourceresolver.impl.ResourceResolverImpl.<init>(ResourceResolverImpl.java:110)

at org.apache.sling.resourceresolver.impl.CommonResourceResolverFactoryImpl.getResourceResolverInternal(CommonResourceResolverFactoryImpl.java:258)

at org.apache.sling.resourceresolver.impl.CommonResourceResolverFactoryImpl.getAdministrativeResourceResolver(CommonResourceResolverFactoryImpl.java:141)

at org.apache.sling.resourceresolver.impl.helper.ResourceResolverControl.getResourceTypeResourceResolver(ResourceResolverControl.java:689)

at org.apache.sling.resourceresolver.impl.helper.ResourceResolverControl.getParentResourceType(ResourceResolverControl.java:713)

at org.apache.sling.resourceresolver.impl.ResourceResolverImpl.getParentResourceType(ResourceResolverImpl.java:1304)

at org.apache.sling.resourceresolver.impl.ResourceResolverImpl.getParentResourceType(ResourceResolverImpl.java:1293)

at org.apache.sling.resourceresolver.impl.ResourceResolverImpl.isResourceType(ResourceResolverImpl.java:1321)

at org.apache.sling.api.resource.AbstractResource.isResourceType(AbstractResource.java:123)

at com.<****>.aem.website.v1.core.utils.PageUtil.isResourceType(PageUtil.java:147)

Hope this will help more

[VeenaVikraman]

@Jörg Hoh

Any help here ?

[Jörg_Hoh]

What version of AEM are you using?

This observation seems somehow related to [1], although this did not mention a leak of JCR sessions. IIRC there was another issue in that area which actually dealt with a leak loosing the ResourceTypeResolver RR, but I don't find it rightaway. How do you obtain the the resourceResolver you feed into this "isResourceType" call? I am asking because of [2].

[1] [SLING-3285] ResourceResolver.isResourceType() creates admin session for every call with super type,...

[2] [SLING-7591] Session leak when a ResourceResolver is created from a Session then gets cloned - ASF J...

[VeenaVikraman]

Thanks Jörg Hoh​ . As you can see from the method signature, we are just passing a Page and String as arguments. But the real culprit is the method isResoureType() in Resource ("The Adobe AEM Quickstart and Web Application.") . If you see the method details in API, it is saying that

So if we go by above does that mean this method is opening a resourceResolver and is never closed ?

[Jörg_Hoh]

A little bit of background: A resource resolver internally holds 2 session! One "real" session through which all actions are performed, and this session is running with user privileges. And an additional one with elevated privileges which is used for resolving privileges and such (which is typically not always directly accesible by the user session).

I remember an issue with this additional session but I don't recall any details.

Also it could be temporary glitch with Garbage Collection and lazily unregistering MBeans, I saw reports on the sling JIRA as well. Maybe you want to check if all (well, kind of) sessions have the same threaddump for opening and then report it to Adobe support.

[VeenaVikraman]

Thanks Jörg Hoh​ .

A little insight to the situation right now ,The situation is bit bad with our prod servers right now. Since business reported slowness , we went back to Adobe and they informed us to check JMX for open sessions and fix the code. The sessions and resourceResolvers in our code was already taken care. I checked JMX and yesterday alone there was >11000 Session objects (note that this is the count of sessions opened in last 24 hours ) tracked in JMX. I read that anything above 500 is bad. I checked first 50 manually and all showed the above stack trace. Then I checked randomly from all these 11000 and few didn't had a stack trace and the rest were all these and few from one old project which was still in prod (this will be deleted).

[Jörg_Hoh]

what AEM version are you using?

[VeenaVikraman]

current prod servers are 6.2 , we will be moving to 6.4 in may be 4 months.

[Jörg_Hoh · Accepted solution]

I just checked [1] and I haven't found a way that the ResourceTypeResolver (which seems to leak in your case) is actually leaking. It is always closed (line 665ff).

I cannot explain the fact that a lot of these session are actually open (it would be good if you could verify that by a heapdump), and that it's not a problem of the garbage collection.

[1] sling-org-apache-sling-resourceresolver/ResourceResolverControl.java at org.apache.sling.resourceres...

[VeenaVikraman]

There are few sessions like below (around 6k) opened in STAGE. How can I trace from where this sessions are opened ?

[Jörg_Hoh]

Below that (and not contained in the screenshot) there is a field "stacktrace", which should be filled with the stacktrace. This stacktraces are only created when more than 1000 sessions are opened (default), so you should check the youngest sessions, they should contain these stacks.

[Mansi1]

My AEM version is 6.5 and oak core is 1.10.8, so is it that initstacktraces will show data only if opened sessions are > 1000? Currently initstacktrace is showing empty. Where this property "oak.sessionStats.initStackTraceThreshold" can be tuned?

[solis]

Same problem here. Did you manage to solve it?

[VeenaVikraman]

Thanks a lot Jorg . I was struggling to figure out those logs.

[NitroHazeDev]

Hi @VeenaVikraman  was it due to @slingObject way of injecting resource resolver?