AEM 6.5 Upgrade to 6.5.7 CFP Causing Unresponsive Instances | Community
Skip to main content
S
sdouglasmcSonova
Level 4
December 18, 2020
Solved

AEM 6.5 Upgrade to 6.5.7 CFP Causing Unresponsive Instances

  • December 18, 2020
  • 6 replies
  • 11922 views

We've recently upgraded AEM from 6.5.5 to 6.5.7 (didn't install 6.5.6) - running on Ubuntu 18.04.5, Java 11.

We have been pretty consistent the last 8-9 months after making some performance adjustments in migrating from 6.3 windows to 6.5 Unbuntu.  Rarely having to restart.  However, since we've upgraded from SP 6.5.5 to 6.5.7 (we bipassed 6.5.6 - which SHOULDN'T matter), we've noticed really poor performance, author instance needing restarts nearly every 2-3 days and publish instances 5-6 days because of becoming unresponsive.  No heap dumps or OutOfMemory exceptions.

I'm just wondering if anybody else might know of any obvious reasons this could be happening before I bog myself down in a sea of heapdumps and threaddumps.

    This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
    Best answer by NarayanaCh

    @sdouglasmcsonova we also got the same package from Adobe, it resolved our issue.

    Notes from adobe:

    Checking the thread dumps and further researching internally, It seems you are running into a known issue(CQ-4312194) with SP7 where numerous threads get blocked due to a Timer with the Component Registry (org.apache.felix.scr.impl.ComponentRegistry). This causes the instance to become unresponsive.

    • Follow the steps below to resolve the issue:
      - Install the attached hotfix package
      - This will trigger a restart of a couple of bundles. So, need to wait 3-5 mins
      - Go to <host>:<port>/system/console and make sure the "org.apache.felix.scr" version is updated to 2.1.20

    6 replies

    Kunal_Gaba_
    Kunal_Gaba_
    December 18, 2020

    Can you check in the logs if there are any session leaks or if there are any slow unresponsive queries ? Do you see any errors in the logs ?  What do you see in health check dashboards ? Any patterns of high memory or CPU consumptions or disk utilizations ?

      S
      sdouglasmcSonovaAuthor
      Level 4
      December 18, 2020

      ...

      J
      JakubLG
      Level 2
      December 27, 2020

      Can you check threaddumps. I've noticed AEM 6.5.7 hanging on following deadlock on my environments:

      java.lang.Thread.State: WAITING (on object monitor) at java.base@11.0.9/jdk.internal.misc.Unsafe.park(Native Method) - waiting to lock <0x1d24408f> (a java.util.concurrent.CountDownLatch$Sync) owned by "null" tid=0x-1 at java.base@11.0.9/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.9/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885) at java.base@11.0.9/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039) at java.base@11.0.9/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345) at java.base@11.0.9/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232) at org.apache.felix.framework.ServiceRegistry.getService(ServiceRegistry.java:365)
        S
        sdouglasmcSonovaAuthor
        Level 4
        December 27, 2020
        They've sent me a Hotfix-CQ-4312194-FELIX-6252-1.0.zip. I've installed it but I'm not sure if it is working or not. Lower amount of traffic but we were getting killed in publish after replications as well when the dispatcher would flush: now.
        N
        NarayanaCh
        Level 3
        January 18, 2021

        @sdouglasmcsonova is there any resolution or solution on this issue? we are also facing the same issue in our Prod environment.

          S
          sdouglasmcSonovaAuthor
          Level 4
          January 18, 2021
          Get the Hotfix mentioned in the comments of this thread.
            N
            NarayanaChAccepted solution
            Level 3
            January 18, 2021

            @sdouglasmcsonova we also got the same package from Adobe, it resolved our issue.

            Notes from adobe:

            Checking the thread dumps and further researching internally, It seems you are running into a known issue(CQ-4312194) with SP7 where numerous threads get blocked due to a Timer with the Component Registry (org.apache.felix.scr.impl.ComponentRegistry). This causes the instance to become unresponsive.

            • Follow the steps below to resolve the issue:
              - Install the attached hotfix package
              - This will trigger a restart of a couple of bundles. So, need to wait 3-5 mins
              - Go to <host>:<port>/system/console and make sure the "org.apache.felix.scr" version is updated to 2.1.20
              kautuk_sahni
              Community Manager
              kautuk_sahniCommunity Manager
              Community Manager
              January 20, 2021
              Thank you for sharing this with community.
              Kautuk Sahni
              J
              JakubLG
              Level 2
              January 20, 2021

              For those who haven't updated to 6.5.7 yet, Adobe has a new SP7 installation package with Hotfix-CQ-4312194-FELIX-6252 included in it. Makes upgrade faster and more stable. This is not publicly available (yet) but you can probably ask for it in a ticket.

                kautuk_sahni
                Community Manager
                kautuk_sahniCommunity Manager
                Community Manager
                January 21, 2021
                @jakublg Thank you for sharing this with community. Keep assisting others here.
                Kautuk Sahni
                B
                bartek__w
                Level 2
                July 21, 2021

                Great article, thanks !

                 

                Is this package available publicly now ? 

                 

                 

                 
                  Terms & ConditionsAccessibility statement

                  Loading footer...