Deadlocks in Thread dump analysis after AEM Author slowness

[Kashyap_P]

Hi Community,

We are facing severe performance issues in our AEM author environment. We have 4 clustered authors accessing a MongoDb instance. We generated the JVM thread dumps during system slowness and found that there is a deadlock situation which looks like below . I see that there is a deadlock situation relating to locks on DocumentNodeStore. There is a thread which is trying to read something from the Oak repo and waiting to lock the object org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore. The same object is locked by a write/update thread and that thread is in a waiting state. I think that because of this deadlock situation , reading Oak repo becomes quite cumbersome and loading the pages is extremely slow. Is my thought process correct. If anybody has faced such situation before , what was the cause of such deadlocks and what did you do to get rid of it ?

"DocumentNodeStore background update thread (5)" #71 daemon prio=5 os_prio=0 tid=0x00007f7de8921800 nid=0x2522 waiting on condition [0x00007f7da9c1d000]

   java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

- parking to wait for  <0x00000001c261a480> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)

at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)

at org.apache.jackrabbit.oak.plugins.document.UnsavedModifications.persist(UnsavedModifications.java:157)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.backgroundWrite(DocumentNodeStore.java:2086)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.internalRunBackgroundUpdateOperations(DocumentNodeStore.java:1711)

- locked <0x00000001c1ccb590> (a org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.runBackgroundUpdateOperations(DocumentNodeStore.java:1689)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$BackgroundOperation.execute(DocumentNodeStore.java:2626)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$NodeStoreTask.run(DocumentNodeStore.java:2601)

at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:

- None

"DocumentNodeStore background read thread (5)" #70 daemon prio=5 os_prio=0 tid=0x00007f7de8920000 nid=0x2521 waiting for monitor entry [0x00007f7daa70d000]

   java.lang.Thread.State: BLOCKED (on object monitor)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.internalRunBackgroundReadOperations(DocumentNodeStore.java:1742)

- waiting to lock <0x00000001c1ccb590> (a org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.runBackgroundReadOperations(DocumentNodeStore.java:1730)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$BackgroundReadOperation.execute(DocumentNodeStore.java:2642)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$NodeStoreTask.run(DocumentNodeStore.java:2601)

at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:

- None

[kautuk_sahni]

Jörg Hoh Any help here?

~kautuk

[Jörg_Hoh · Accepted solution]

Hi,

please raise a Daycare ticket. Include all of these informations:

* a number of threaddumps (at least 5-7) which have been created during this slowness

* a complete configuration dump (/system/console/config/configuration-status.zip) of the affected instances

* Mongo DB related information (config dumps, logs)

* would be also good if you could add system monitoring data of the AEM instance and the mongoDB of that time

This topic is too large (and probably too business critical) to handle it here in the forums alone.

Jörg

[stejan-rai]

Is there any info about that?
Is there a hotfix or is it solved in a cfp?

Best regards

JAn    

[dineshc12261746]

Hi , Can we have any update here ? what made the thread to go to blocked state ? , do we have any hotfix for the same ?

Thanks,

Dinesh