High CPU Usage @ around 90%, ThreadDumps shows many instances of jdk.internal.misc.Unsafe.park, what to do?

[AEMWizard]

One of our publish servers keeps getting CPU usage notifications for 90% consumption. We pulled down the thread dumps, and analyzed it on the IBM Thread dump analyzer. We realized that there are many instances of jdk.internal.misc.Unsafe.park, around 303 threads (look below for the total amount of Identical Stack trace's listed below in the table. 225 waiting threads, 54 timed_waiting.

Would this be the reason why our publish servers keep hitting 90% consumption?

What is the best way to debug this problem, and how can we solve this issue. There is nothing on the web that discuss about this.

Just the top 3 Identical Stack trace(grouped) with the jdk.internal.misc.Unsafe.park problem.

23 TIMED_WAITING threads

java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method) - parking to wait for <0x00000007d353a400> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.3/LockSupport.java:234) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.3/AbstractQueuedSynchronizer.java:2123) ... See complete stacktrace.

19 WAITING threads

java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method) - parking to wait for <0x00000007a50b9cd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/AbstractQueuedSynchronizer.java:2081)

158 WAITING threads

java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method) - parking to wait for <0x00000007b50e9a20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/AbstractQueuedSynchronizer.java:2081)

[AEMWizard]

@Jörg_Hoh , do you think you can help me?

[AEMWizard]

@Jörg_Hoh thank you
I just reviewed the threaddump once again, and we only got 29 RUNNABLE (running).

Hmm, why did our AEM publish keep crashing?

[Jörg_Hoh]

Hard to say just from this data. Threads in WAITING or TIMED_WAITING isn't a problem per se, as especially threadpools and other "support-threads" can be in that state without a negative impact on the rest of the application.

The high CPU sage comes from the threads in the state "RUNNING". You might have more threads in RUNNING than available CPU cores, but that's ok. Because RUNNING just indicates that this thread can run (and is not blocked by something else), it does not indicate that it is scheduled to be run right now.

[AEMWizard]

@Jörg_Hoh thank you
I just reviewed the threaddump once again, and we only got 29 RUNNABLE (running).

Hmm, why did our AEM publish keep crashing?

[Jörg_Hoh]

What do you mean with "crashing"? Does the JVM process terminate? 

[santhoshm687661]

@AEMWizardDo you find any workaround for this query.