Highlighted

LiveCycle unable to access Cache Controller. Exception message - Error accessing the cache container - Error on GET action for cache Replicated:UM_CLUSTER_INVALIDATION_CACHE

Asadullah

26-08-2014

The incident starts after all the members are removed from the DCS Stack DefaultCoreGroup view and added back again by WebSphere Application Server. Following exceptions are seen after LiveCycle was added back in again to the view.

We have a clustered environment with two nodes 2A and 2B. Server 2A crashed and therefore all members on 2B node were removed from the DCS view. Later the new core group view was installed but LiveCycle did resume operations as expected.

Errors below:

Exception caught while dealing with cache : Action - Get, ObjectType - UM_CLUSTER_INVALIDATION_CACHE, Exception message - Error accessing the cache container - Error on GET action for cache Replicated:UM_CLUSTER_INVALIDATION_CACHE

ServiceRegist W   Cache get failed for service EncryptionService Reason: Error accessing the cache container - Error on GET action for cache

Error accessing the cache container - Error on GET action for cache Local:SERVICE_FACTORY_CACHE

The following message appeared for several different services:

Cache put failed for service [CredentialService, ReaderExtensionsService,EncryptionService, etc]

SSOSessionCle W   Error in cleaning stale sessions from the database. These sessions would be deleted in next trigger

                                 java.lang.RuntimeException: com.adobe.livecycle.cache.CacheActionException: Error accessing the cache container - Error on ENTRYSET action for cache Local:UM_ASSERTION_CACHE

  at com.adobe.idp.um.businesslogic.authentication.AssertionCacheManager.removeExpiredResults(AssertionCacheManager.java:125)

  at com.adobe.idp.um.scheduler.SSOSessionCleanupJob.execute(SSOSessionCleanupJob.java:69)

  at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

  at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529)

Caused by: com.adobe.livecycle.cache.CacheActionException: Error accessing the cache container - Error on ENTRYSET action for cache Local:UM_ASSERTION_CACHE

  at com.adobe.livecycle.cache.adapter.GemfireLocalCacheAdapter.entrySet(GemfireLocalCacheAdapter.java:219)

  at com.adobe.idp.um.businesslogic.authentication.AssertionCacheManager.removeExpiredResults(AssertionCacheManager.java:99)

  ... 3 more

Caused by: com.gemstone.gemfire.distributed.DistributedSystemDisconnectedException: GemFire on 2B<v1>:9057/51073 started at Sun Aug 17 08:57:23 EDT 2014: Message distribution has terminated, caused by com.gemstone.gemfire.ForcedDisconnectException: This member has been forced out of the distributed system.  Reason='this member is no longer in the view but is initiating connections'

  at com.gemstone.gemfire.distributed.internal.DistributionManager$Stopper.generateCancelledException(DistributionManager.java:746)

  at com.gemstone.gemfire.distributed.internal.InternalDistributedSystem$Stopper.generateCancelledException(InternalDistributedSystem.java:846)

  at com.gemstone.gemfire.internal.cache.GemFireCacheImpl$Stopper.generateCancelledException(GemFireCacheImpl.java:1090)

  at com.gemstone.gemfire.CancelCriterion.checkCancelInProgress(CancelCriterion.java:59)

  at com.gemstone.gemfire.internal.cache.LocalRegion.checkRegionDestroyed(LocalRegion.java:6694)

  at com.gemstone.gemfire.internal.cache.LocalRegion.checkReadiness(LocalRegion.java:2587)

  at com.gemstone.gemfire.internal.cache.LocalRegion.entries(LocalRegion.java:1815)

  at com.gemstone.gemfire.internal.cache.LocalRegion.entrySet(LocalRegion.java:7941)

  at com.adobe.livecycle.cache.adapter.GemfireLocalCacheAdapter.entrySet(GemfireLocalCacheAdapter.java:209)

  ... 4 more

Caused by: com.gemstone.gemfire.ForcedDisconnectException: This member has been forced out of the distributed system.  Reason='this member is no longer in the view but is initiating connections'

  at com.gemstone.org.jgroups.protocols.pbcast.ParticipantGmsImpl.handleLeaveResponse(ParticipantGmsImpl.java:106)

  at com.gemstone.org.jgroups.protocols.pbcast.GMS.up(GMS.java:1289)

  at com.gemstone.org.jgroups.stack.Protocol.passUp(Protocol.java:767)

  at com.gemstone.org.jgroups.protocols.VIEW_SYNC.up(VIEW_SYNC.java:202)

  at com.gemstone.org.jgroups.stack.Protocol.passUp(Protocol.java:767)

  at com.gemstone.org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:276)

  at com.gemstone.org.jgroups.stack.Protocol.passUp(Protocol.java:767)

  at com.gemstone.org.jgroups.protocols.UNICAST.up(UNICAST.java:294)

  at com.gemstone.org.jgroups.stack.Protocol.passUp(Protocol.java:767)

  at com.gemstone.org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:625)

  at com.gemstone.org.jgroups.stack.Protocol.passUp(Protocol.java:767)

  at com.gemstone.org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:187)

  at com.gemstone.org.jgroups.stack.Protocol.passUp(Protocol.java:767)

  at com.gemstone.org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:504)

  at com.gemstone.org.jgroups.stack.Protocol.passUp(Protocol.java:767)

  at com.gemstone.org.jgroups.protocols.FD.up(FD.java:438)

  at com.gemstone.org.jgroups.stack.Protocol.passUp(Protocol.java:767)

  at com.gemstone.org.jgroups.protocols.Discovery.up(Discovery.java:258)

  at com.gemstone.org.jgroups.stack.Protocol.passUp(Protocol.java:767)

  at com.gemstone.org.jgroups.protocols.TP.handleIncomingMessage(TP.java:1110)

  at com.gemstone.org.jgroups.protocols.TP.handleIncomingPacket(TP.java:1016)

  at com.gemstone.org.jgroups.protocols.TP.receive(TP.java:923)

  at com.gemstone.org.jgroups.protocols.UDP$UcastReceiver.run(UDP.java:1320)

  at java.lang.Thread.run(Thread.java:773)

[22/08/14 0:28:10:237 EDT] 00000b4f WorkPolicyEva W   Exception thrown in background thread. Cause: java.lang.NullPointerException

[22/08/14 0:29:10:252 EDT] 00000b4f WorkPolicyEva W   Exception thrown in background thread. Cause: java.lang.NullPointerException

[22/08/14 0:30:10:268 EDT] 00000b4f WorkPolicyEva W   Exception thrown in background thread. Cause: java.lang.NullPointerException

[22/08/14 0:31:10:283 EDT] 00000b4f WorkPolicyEva W   Exception thrown in background thread. Cause: java.lang.NullPointerException

[22/08/14 0:32:10:298 EDT] 00000b4f WorkPolicyEva W   Exception thrown in background thread. Cause: java.lang.NullPointerException

[22/08/14 0:33:10:313 EDT] 00000b4f WorkPolicyEva W   Exception thrown in background thread. Cause: java.lang.NullPointerException

[22/08/14 0:34:10:328 EDT] 00000b4f WorkPolicyEva W   Exception thrown in background thread. Cause: java.lang.NullPointerException

[22/08/14 0:35:10:343 EDT] 00000b4f WorkPolicyEva W   Exception thrown in background thread. Cause: java.lang.NullPointerException

[22/08/14 0:36:10:358 EDT] 00000b4f WorkPolicyEva W   Exception thrown in background thread. Cause: java.lang.NullPointerException

[22/08/14 0:37:10:373 EDT] 00000b4f WorkPolicyEva W   Exception thrown in background thread. Cause: java.lang.NullPointerException

[22/08/14 0:38:10:389 EDT] 00000b4f WorkPolicyEva W   Exception thrown in background thread. Cause: java.lang.NullPointerExceptionor

I have tried looking for the root cause about why LiveCycle was not able to resume normally, didn't find anything related.

Replies

Highlighted

WASIL

Employee

28-08-2014

LiveCycle uses Gemfire as distributed cache for cluster members. If you are using TCP Locators (caching based on TCP instead of UDP), below are the possible situations which might lead to “ForcedDisconnectException” :

- There is time difference between two nodes.

- These is network connectivity issues.

- The high CPU usage by the member crashed.


-Wasil