Hi Community,
We are facing severe performance issues in our AEM author environment. We have 4 clustered authors accessing a MongoDb instance. We generated the JVM thread dumps during system slowness and found that there is a deadlock situation which looks like below . I see that there is a deadlock situation relating to locks on DocumentNodeStore. There is a thread which is trying to read something from the Oak repo and waiting to lock the object org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore. The same object is locked by a write/update thread and that thread is in a waiting state. I think that because of this deadlock situation , reading Oak repo becomes quite cumbersome and loading the pages is extremely slow. Is my thought process correct. If anybody has faced such situation before , what was the cause of such deadlocks and what did you do to get rid of it ?
"DocumentNodeStore background update thread (5)" #71 daemon prio=5 os_prio=0 tid=0x00007f7de8921800 nid=0x2522 waiting on condition [0x00007f7da9c1d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000001c261a480> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
at org.apache.jackrabbit.oak.plugins.document.UnsavedModifications.persist(UnsavedModifications.java:157)
at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.backgroundWrite(DocumentNodeStore.java:2086)
at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.internalRunBackgroundUpdateOperations(DocumentNodeStore.java:1711)
- locked <0x00000001c1ccb590> (a org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore)
at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.runBackgroundUpdateOperations(DocumentNodeStore.java:1689)
at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$BackgroundOperation.execute(DocumentNodeStore.java:2626)
at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$NodeStoreTask.run(DocumentNodeStore.java:2601)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"DocumentNodeStore background read thread (5)" #70 daemon prio=5 os_prio=0 tid=0x00007f7de8920000 nid=0x2521 waiting for monitor entry [0x00007f7daa70d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.internalRunBackgroundReadOperations(DocumentNodeStore.java:1742)
- waiting to lock <0x00000001c1ccb590> (a org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore)
at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.runBackgroundReadOperations(DocumentNodeStore.java:1730)
at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$BackgroundReadOperation.execute(DocumentNodeStore.java:2642)
at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$NodeStoreTask.run(DocumentNodeStore.java:2601)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
