AEM 6.5 Upgrade to 6.5.7 CFP Causing Unresponsive Instances

[sdouglasmc]

We've recently upgraded AEM from 6.5.5 to 6.5.7 (didn't install 6.5.6) - running on Ubuntu 18.04.5, Java 11.

We have been pretty consistent the last 8-9 months after making some performance adjustments in migrating from 6.3 windows to 6.5 Unbuntu.  Rarely having to restart.  However, since we've upgraded from SP 6.5.5 to 6.5.7 (we bipassed 6.5.6 - which SHOULDN'T matter), we've noticed really poor performance, author instance needing restarts nearly every 2-3 days and publish instances 5-6 days because of becoming unresponsive.  No heap dumps or OutOfMemory exceptions.

I'm just wondering if anybody else might know of any obvious reasons this could be happening before I bog myself down in a sea of heapdumps and threaddumps.

[Jakub_G]

After installation of this fix, I haven't observed the deadlock anymore.

[sanjeevkumart45]

Thanks for confirming.

[narayana_chirra]

@sdouglasmc is there any resolution or solution on this issue? we are also facing the same issue in our Prod environment.

[sdouglasmc]

Get the Hotfix mentioned in the comments of this thread.

[sdouglasmc]

Hotfix-CQ-4312194-FELIX-6252-1.0.zip

[narayana_chirra · Accepted solution]

@sdouglasmc we also got the same package from Adobe, it resolved our issue.

Notes from adobe:

Checking the thread dumps and further researching internally, It seems you are running into a known issue(CQ-4312194) with SP7 where numerous threads get blocked due to a Timer with the Component Registry (org.apache.felix.scr.impl.ComponentRegistry). This causes the instance to become unresponsive.

• Follow the steps below to resolve the issue:
  - Install the attached hotfix package
  - This will trigger a restart of a couple of bundles. So, need to wait 3-5 mins
  - Go to <host>:<port>/system/console and make sure the "org.apache.felix.scr" version is updated to 2.1.20

[kautuk_sahni]

Thank you for sharing this with community.

[rahuldeou]

@narayana_chirra can you provide the hotfix package as we are also facing the same issue.

[narayana_chirra]

@rahuldeou

I would recommend, It would be better if you check with Adobe by raising a ticket rather than getting a package from me and trying as this package is not open-source yet.

That being said, someone mentioned in the comments 

"For those who haven't updated to 6.5.7 yet, Adobe has a new SP7 installation package with Hotfix-CQ-4312194-FELIX-6252 included in it. Makes upgrade faster and more stable. This is not publicly available (yet) but you can probably ask for it in a ticket."

[Jakub_G]

For those who haven't updated to 6.5.7 yet, Adobe has a new SP7 installation package with Hotfix-CQ-4312194-FELIX-6252 included in it. Makes upgrade faster and more stable. This is not publicly available (yet) but you can probably ask for it in a ticket.

[kautuk_sahni]

@Jakub_G Thank you for sharing this with community. Keep assisting others here.

[bartek__w]

Great article, thanks !

Is this package available publicly now ?