'Out of Memory: Metaspace'

[ruchim71073425]

Hello,

We had an incident on one of our Publish instance where the environment was completely down.

The error in the logs:

org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Uncaught SlingExceptionjava.lang.Exception: Wrapping Throwable: java.lang.OutOfMemoryError: Metaspace at org.apache.sling.scripting.jsp.JspScriptEngineFactory$JspScriptEngine.eval(JspScriptEngineFactory.java:614) [org.apache.sling.scripting.jsp:2.3.6]at org.apache.sling.scripting.core.impl.DefaultSlingScript.call(DefaultSlingScript.java:388) [org.apache.sling.scripting.core:2.3.2]at

We checked the thread dumps and lot of waiting threads at:

org.apache.sling.scripting.sightly.impl.engine.compiled.SlingHTLMasterCompiler.compileSource(SlingHTLMasterCompiler.java:392)
at org.apache.sling.scripting.sightly.impl.engine.compiled.SlingHTLMasterCompiler.compileHTLScript(SlingHTLMasterCompiler.java:245)
at org.apache.sling.scripting.sightly.impl.engine.SightlyScriptEngine.compile(SightlyScriptEngine.java:58)

We have also been seeing a 'Missing Compilation Support' error after some deployments lately. Only a restart fixes that.

Wondering if these are related and if anyone has faced such issue earlier and found a root cause and solution.

Thanks

Ruchi

[Rohan_Garg]

Hi @ruchim71073425 

java.lang.OutOfMemoryError implies JVM is unable to allocate an object due to insufficient space in the Java heap. 

• Check your servlet/custom code for any possible memory leaks.
• If it is not due to memory leak, increasing the heap size would be best recommendation. 
  What is the memory in your publish server and how much is free for AEM.

[tusharbias · Accepted solution]

JVM memory has the following regions:

• a. Young Generation
• b. Old Generation
• c. Metaspace
• d. Others region

When you encounter ‘java.lang.OutOfMemoryError: Metaspace’, it indicates that the Metaspace region in the JVM memory is getting saturated. Metaspace is the region where metadata details that are required to execute your application are stored. In a nutshell, it contains class definitions, method definitions, and other metadata of your application. 

The java.lang.OutOfMemoryError: Metaspace indicates that the amount of native memory allocated for Java class metadata is exausted.

In general terms, the more classes you are loading into the JVM, the more memory will be consumed by the metaspace. In Java 8 and later, the maximum amount of memory allocated for Java classes (MaxMetaspaceSize) is by default unlimited, so in most cases there is no need to change this setting.

 -XX:MaxMetaspaceSize=3200m

In order to find the Root cause, its recommended that you follow these steps:

1. Understand how the JVM manages the MetaSpace data
2. Learn how to monitor the MetaSpace in a running Java application, there are tools available to monitor that like visualGC
3. Check the Heap dump of your application to find offending Classes

For any OOM exception it is necessary to analize the Heap dump.