Thread Dump and Heap Dump issues

[GleniB]

Good Morning,

Our publisher servers seem to be having a very hard time with the load recently. We were analyzing the thread dumps to see if we could find anything unusual and we saw that we had A LOT of waiting and blocked threads. All the waiting threads give this message:
- waiting to lock <0x3b9b994f> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) owned by "null" tid=0x-1

After some research we found that this could be caused by garbage collection taking up resources as it is trying to clear up space.
We tried analyzing the heap dumps, but full honesty we did not know exactly what to look for in them.

Has anyone seen this before or have any advice on how we can proceed to troubleshoot this issue?
Thank you!

[Pulkit_Jain_]

@GleniB 

Garbage collection happens every now and then on the server but if the blocking thread belongs to a doamon thread e.g. garbage collection thread then it could be an issue. Its worth reviewing the heap dumps[0] as well to check why is the garbage collection happening so frequently. 

For a quicker turnaround, please raise a support ticket along with the required details/dumps.

[0] - https://heaphero.io/ 

[GleniB]

Thank you, 
Is there anything in specific we should be looking for in the heap dump?

[Pulkit_Jain_ · Accepted solution]

@GleniB 

I think the first step would be to review the thread dump thoroughly for any stuck/blocked threads on any particular process/activity, since you've mentioned "having a very hard time with the load recently".

In case of OutOfMemory situations OR frequent garbage collection activities (drawing conclusion from thread dump analysis), you can review the Heap dumps to check for any memory leakage, objects with high memory usage or object graph to identify objects of not releasing memory.