We were facing some memory/garbage collection issues on one of our AEM 6.3 publishers. We did a heap dump analysis and the one suspect said:
243,406 instances of "org.eclipse.jetty.server.session.HashedSession", loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoaderJava5 @ 0x741a1fee8" occupy 1,421,271,104 (66.52%) bytes. These instances are referenced from one instance of "java.util.concurrent.ConcurrentHashMap$Node", loaded by "<system class loader>"
This lead me to this article: AEM instance crashes with OutOfMemoryError due to Jetty Sessions . (Even though that article was for 6.0 and 6.1, it fit our issue pretty well.) We've had issues with our Jetty sessions before - the Session Timeout was still set to the default 0, so we set it to 20. But when we changed it, the parent setting didn't change; it stayed at the default 0. Instead, it created a child entry that has the new value.
The top one, Apache Felix Jetty Based Http Service has this:
The bottom one, org.apache.felix.http.47b40c8f-1ab9-4b26-9d8a-1b4fa044beb0, has this:
And if I go straight to /system/console/configMgr/org.apache.felix.http, it shows the entry that is set to 0. Therefore, I suspect that the system is using the parent that won't change, which is then causing (or contributing to) our of memory/garbage collection issues.
So my questions are:
Is the system using the parent entry or the child entry?
How do I change the top/parent entry (and not have a child entry)?
When there are multiple entries on a setting, how do we know which setting the system uses?
Is the system using the parent entry or the child entry? > Not certain how to confirm -- cannot guarantee if parent/child is picking in the environment
How do I change the top/parent entry (and not have a child entry)? > You should be able to click the Apache Felix Jetty Based Http Service row within /system/console/configMgr (the row with the checkmark on the left and Edit,Unbind, and Trash buttons on the right of the row) and make changes and save accordingly. This item will be saved into a .config file which you should see in CRXDE Lite ; e.g. /apps/system/config/org.apache.felix.http.config (OOTB location upon save) > You can verify be searching CRXDE Lite for org.apache.felix.http.config and confirm the org.apache.felix.http.config file has your changes intended
When there are multiple entries on a setting, how do we know which setting the system uses? > I could not find a way to visually confirm if the factory Apache Felix Jetty Based Http Service configuration (org.apache.felix.http.47b40c8f-1ab9-4b26-9d8a-1b4fa044beb0) ever picked up. I was able to confirm org.apache.felix.http.config picks up by checking /system/console/status-httpservice which shows 'Runtime Properties' property value pair for org.apache.felix.http.session.invalidate and org.apache.felix.http.session.uniqueid
Let us know if this helps -- suggest not to use the Apache Felix Jetty Based Http Service factory configurations unless the factory config will apply for different Ports (HTTP / HTTPs) (e.g. ) -- when same port is used in multiple configurations - not certain which is applying