Your achievements

Level 1

0% to

Level 2

Tip /
Sign in

Sign in to Community

to gain points, level up, and earn exciting badges like the new
Bedrock Mission!

Learn more

View all

Sign in to view all badges

SOLVED

Deadlocks in Thread dump analysis after AEM Author slowness

Kashyap_P
Level 2
Level 2

Hi Community,

We are facing severe performance issues in our AEM author environment. We have 4 clustered authors accessing a MongoDb instance. We generated the JVM thread dumps during system slowness and found that there is a deadlock situation which looks like below . I see that there is a deadlock situation relating to locks on DocumentNodeStore. There is a thread which is trying to read something from the Oak repo and waiting to lock the object org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore. The same object is locked by a write/update thread and that thread is in a waiting state. I think that because of this deadlock situation , reading Oak repo becomes quite cumbersome and loading the pages is extremely slow. Is my thought process correct. If anybody has faced such situation before , what was the cause of such deadlocks and what did you do to get rid of it ?

"DocumentNodeStore background update thread (5)" #71 daemon prio=5 os_prio=0 tid=0x00007f7de8921800 nid=0x2522 waiting on condition [0x00007f7da9c1d000]

   java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

- parking to wait for  <0x00000001c261a480> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)

at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)

at org.apache.jackrabbit.oak.plugins.document.UnsavedModifications.persist(UnsavedModifications.java:157)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.backgroundWrite(DocumentNodeStore.java:2086)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.internalRunBackgroundUpdateOperations(DocumentNodeStore.java:1711)

- locked <0x00000001c1ccb590> (a org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.runBackgroundUpdateOperations(DocumentNodeStore.java:1689)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$BackgroundOperation.execute(DocumentNodeStore.java:2626)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$NodeStoreTask.run(DocumentNodeStore.java:2601)

at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:

- None

"DocumentNodeStore background read thread (5)" #70 daemon prio=5 os_prio=0 tid=0x00007f7de8920000 nid=0x2521 waiting for monitor entry [0x00007f7daa70d000]

   java.lang.Thread.State: BLOCKED (on object monitor)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.internalRunBackgroundReadOperations(DocumentNodeStore.java:1742)

- waiting to lock <0x00000001c1ccb590> (a org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.runBackgroundReadOperations(DocumentNodeStore.java:1730)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$BackgroundReadOperation.execute(DocumentNodeStore.java:2642)

at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$NodeStoreTask.run(DocumentNodeStore.java:2601)

at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:

- None

1 Accepted Solution
Jörg_Hoh
Correct answer by
Employee
Employee

Hi,

please raise a Daycare ticket. Include all of these informations:

* a number of threaddumps (at least 5-7) which have been created during this slowness

* a complete configuration dump (/system/console/config/configuration-status.zip) of the affected instances

* Mongo DB related information (config dumps, logs)

* would be also good if you could add system monitoring data of the AEM instance and the mongoDB of that time

This topic is too large (and probably too business critical) to handle it here in the forums alone.

Jörg

View solution in original post

4 Replies
Jörg_Hoh
Correct answer by
Employee
Employee

Hi,

please raise a Daycare ticket. Include all of these informations:

* a number of threaddumps (at least 5-7) which have been created during this slowness

* a complete configuration dump (/system/console/config/configuration-status.zip) of the affected instances

* Mongo DB related information (config dumps, logs)

* would be also good if you could add system monitoring data of the AEM instance and the mongoDB of that time

This topic is too large (and probably too business critical) to handle it here in the forums alone.

Jörg

View solution in original post

stejan-rai
Level 1
Level 1

Is there any info about that?
Is there a hotfix or is it solved in a cfp?

Best regards

JAn    

dineshc12261746
Level 1
Level 1

Hi , Can we have any update here ? what made the thread to go to blocked state ? , do we have any hotfix for the same ?

Thanks,

Dinesh