OutOfMemoryError

[Voon_Siong]

Hi,

We have recently found OutOfMemoryErrors in our error.log file.  When this happens, the CQ instance is still running, but no HTTP connection can be made to the instance.  In the error.log file, we

22.09.2014 23:17:00.583 *ERROR* [Shell Script Executor Thread for diskusage.sh] org.apache.sling.extensions.threaddump.internal.Activator Uncaught exception in Thread Thread[Shell Script Executor Thread for diskusage.sh,5,main] java.lang.OutOfMemoryError: unable to create new native thread

        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:691)
        at com.adobe.granite.monitoring.impl.ShellScriptExecutorImpl.execute(ShellScriptExecutorImpl.java:121)
        at com.adobe.granite.monitoring.impl.ScriptMBean.invoke(ScriptMBean.java:99)
        at com.adobe.granite.monitoring.impl.ScriptMBean.invoke(ScriptMBean.java:158)
        at com.adobe.granite.monitoring.impl.ScriptConfigImpl$ExecutionThread.run(ScriptConfigImpl.java:208)
        at java.lang.Thread.run(Thread.java:722)
22.09.2014 23:17:01.145 *ERROR* [Process Executor for cpu.sh] com.adobe.granite.monitoring.impl.ShellScriptExecutorImpl Error while executing script /opt/cq/cqauthor_7502/crx-quickstart/monitoring/cpu.sh java.io.IOException: Cannot run program "/opt/cq/cqauthor_7502/crx-quickstart/monitoring/cpu.sh" (in directory "/opt/cq/cqauthor_7502/crx-quickstart/monitoring"): error=11, Resource temporarily unavailable
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
        at java.lang.Runtime.exec(Runtime.java:615)
        at com.adobe.granite.monitoring.impl.ShellScriptExecutorImpl$ProcessExecutor.run(ShellScriptExecutorImpl.java:307)
        at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException: error=11, Resource temporarily unavailable
        at java.lang.UNIXProcess.forkAndExec(Native Method)
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:135)
        at java.lang.ProcessImpl.start(ProcessImpl.java:130)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1021)
        ... 3 more

All other errors that are logged have the same OutOfMemoryError: unable to create new native thread message.  When we tried to use the stop script in /bin to stop CQ, it would error out with the message "The resource is unavailable".  The only way for us to stop CQ and restart would be to issue a "ps -ef | grep java", find the PID and issue a "kill".

Has anyone encountered this?  How can we resolve this issue?

Thank You.

[Sham_HC · Accepted solution]

IMO The error has no relation with shutdown problem. The cpu and diskusage monitoring scripts will running multiple times. The good thing is that these two scripts do not actually provide much value so it is safe to disable them[1]. They just allow you to view the cpu usage and disk usage via JMX. In case you want use it then modify script as per your storage system & mounting point.
 

Regarding stopping issue make sure if it is upgraded system you are using right start/stop script and you need to make custom configuration at /bin/*.sh  If everything is fine, take a 10 thread dumps when shutdown is initiated and file a support request.

[1]   http://helpx.adobe.com/experience-manager/kb/CQ55MonitoringTooManyOpenFiles.html

[shivangim985778]

How to get cpu and diskusage monitoring scripts?