Expand my Community achievements bar.

SOLVED

Deadlocks in Thread dump analysis after AEM Author slowness

Avatar

Level 2

Hi Community,

We are facing severe performance issues in our AEM author environment. We have 4 clustered authors accessing a MongoDb instance. We generated the JVM thread dumps during system slowness and found that there is a deadlock situation which looks like below . I see that there is a deadlock situation relating to locks on DocumentNodeStore. There is a thread which is trying to read something from the Oak repo and waiting to lock the object org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore. The same object is locked by a write/update thread and that thread is in a waiting state. I think that because of this deadlock situation , reading Oak repo becomes quite cumbersome and loading the pages is extremely slow. Is my thought process correct. If anybody has faced such situation before , what was the cause of such deadlocks and what did you do to get rid of it ?

"DocumentNodeStore background update thread (5)" #71 daemon prio=5 os_prio=0 tid=0x00007f7de8921800 nid=0x2522 waiting on condition [0x00007f7da9c1d000]

   java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

- parking to wait for  <0x00000001c261a480> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)

at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)

at org.apache.jackrabbit.oak.plugins.document.UnsavedModifications.persist(UnsavedModifications.java:157)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.backgroundWrite(DocumentNodeStore.java:2086)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.internalRunBackgroundUpdateOperations(DocumentNodeStore.java:1711)

- locked <0x00000001c1ccb590> (a org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.runBackgroundUpdateOperations(DocumentNodeStore.java:1689)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$BackgroundOperation.execute(DocumentNodeStore.java:2626)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$NodeStoreTask.run(DocumentNodeStore.java:2601)

at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:

- None

"DocumentNodeStore background read thread (5)" #70 daemon prio=5 os_prio=0 tid=0x00007f7de8920000 nid=0x2521 waiting for monitor entry [0x00007f7daa70d000]

   java.lang.Thread.State: BLOCKED (on object monitor)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.internalRunBackgroundReadOperations(DocumentNodeStore.java:1742)

- waiting to lock <0x00000001c1ccb590> (a org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.runBackgroundReadOperations(DocumentNodeStore.java:1730)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$BackgroundReadOperation.execute(DocumentNodeStore.java:2642)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$NodeStoreTask.run(DocumentNodeStore.java:2601)

at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:

- None

1 Accepted Solution

Avatar

Correct answer by
Employee Advisor

Hi,

please raise a Daycare ticket. Include all of these informations:

* a number of threaddumps (at least 5-7) which have been created during this slowness

* a complete configuration dump (/system/console/config/configuration-status.zip) of the affected instances

* Mongo DB related information (config dumps, logs)

* would be also good if you could add system monitoring data of the AEM instance and the mongoDB of that time

This topic is too large (and probably too business critical) to handle it here in the forums alone.

Jörg

View solution in original post

4 Replies

Avatar

Administrator

Jörg Hoh Any help here?

~kautuk



Kautuk Sahni

Avatar

Correct answer by
Employee Advisor

Hi,

please raise a Daycare ticket. Include all of these informations:

* a number of threaddumps (at least 5-7) which have been created during this slowness

* a complete configuration dump (/system/console/config/configuration-status.zip) of the affected instances

* Mongo DB related information (config dumps, logs)

* would be also good if you could add system monitoring data of the AEM instance and the mongoDB of that time

This topic is too large (and probably too business critical) to handle it here in the forums alone.

Jörg

Avatar

Level 1

Is there any info about that?
Is there a hotfix or is it solved in a cfp?

Best regards

JAn    

Avatar

Level 2

Hi , Can we have any update here ? what made the thread to go to blocked state ? , do we have any hotfix for the same ?

Thanks,

Dinesh