IDP_JMS_Bus Messaging Engine stops by itself

[ibm_lifesci]

Hi,

In one of our LiveCycle ES installs, we have on WebSphere App Server with an Oracle database, we see that the IDP_JMS_Bus stops every now and then. This of course interrupts the availability of the LCES web services to process client invocations.

When i look at the Messaging Engine(s) list in WAS Admin console for the app server, I see that its status has changed from the green arrow (STARTED) to the "gray circle with diagonal line across" icon indicating 'Unavailable' status.

Below is the recurring message that appears in the SystemOut.log from the Livecycle ES server.

Why does this happen? How to remedy this nuisance? is this a known JMS issue when Oracle is the databse?

[7/23/09 9:58:27:478 EDT] 00000013 XARecoveryDat W   WTRN0005W: The XAResource for a transaction participant could not be recreated and transaction recovery may not be able to complete properly. The resource was [com.ibm.ws.sib.ra.recovery.impl.SibRaXaResourceInfo@1689543860 <busName=IDP_JMS_Bus> <meName=NCSUSRADMFP08Node02.server1-IDP_JMS_Bus> <meUuid=72F0B32B6B13CBB3> <userName=null> <password=null> <useServerSubject=false> <providerEndpoints=null>]. The exception stack trace follows: com.ibm.ws.Transaction.XAResourceNotAvailableException: com.ibm.websphere.sib.exception.SIResourceException: CWSIT0088E: There are currently no messaging engines in bus IDP_JMS_Bus running. Additional failure information: CWSIT0103E: No messaging engine was found that matched the following parameters: bus=IDP_JMS_Bus, targetGroup=72F0B32B6B13CBB3, targetType=MEUuid, targetSignificance=Required, transportChain=InboundBasicMessaging, proximity=Bus.
at com.ibm.ws.sib.ra.recovery.impl.SibRaXaResourceFactory.getXAResource(SibRaXaResourceFactory.java:99)
at com.ibm.ws.Transaction.JTA.XARecoveryData.getXARminst(XARecoveryData.java:529)
at com.ibm.ws.Transaction.JTA.JTAXAResourceImpl.reconnectRM(JTAXAResourceImpl.java:514)
at com.ibm.ws.Transaction.JTA.JTAXAResourceImpl.commit(JTAXAResourceImpl.java:272)
at com.ibm.ws.Transaction.JTA.RegisteredResources.deliverOutcome(RegisteredResources.java:1973)
at com.ibm.ws.Transaction.JTA.RegisteredResources.distributeOutcome(RegisteredResources.java:2470)
at com.ibm.ws.Transaction.JTA.RegisteredResources.distributeCommit(RegisteredResources.java:2791)
at com.ibm.ws.Transaction.JTA.TransactionImpl.recoverCommit(TransactionImpl.java:2767)
at com.ibm.ws.Transaction.JTA.TransactionImpl.recover(TransactionImpl.java:1333)
at com.ibm.ws.Transaction.JTA.RecoveryManager.resync(RecoveryManager.java:1846)
at com.ibm.ws.Transaction.JTA.RecoveryManager.run(RecoveryManager.java:2509)
at java.lang.Thread.run(Thread.java:797)
Caused by: com.ibm.websphere.sib.exception.SIResourceException: CWSIT0088E: There are currently no messaging engines in bus IDP_JMS_Bus running. Additional failure information: CWSIT0103E: No messaging engine was found that matched the following parameters: bus=IDP_JMS_Bus, targetGroup=72F0B32B6B13CBB3, targetType=MEUuid, targetSignificance=Required, transportChain=InboundBasicMessaging, proximity=Bus.
at com.ibm.ws.sib.trm.client.TrmSICoreConnectionFactoryImpl.localBootstrap(TrmSICoreConnectionFactoryImpl.java:351)
at com.ibm.ws.sib.trm.client.TrmSICoreConnectionFactoryImpl.createConnection(TrmSICoreConnectionFactoryImpl.java:292)
at com.ibm.ws.sib.trm.client.TrmSICoreConnectionFactoryImpl.createConnection(TrmSICoreConnectionFactoryImpl.java:210)
at com.ibm.ws.sib.ra.recovery.impl.SibRaXaResourceInfo.createConnection(SibRaXaResourceInfo.java:513)
at com.ibm.ws.sib.ra.recovery.impl.SibRaXaResourceInfo.createXaResource(SibRaXaResourceInfo.java:314)
at com.ibm.ws.sib.ra.recovery.impl.SibRaXaResourceFactory.getXAResource(SibRaXaResourceFactory.java:88)
... 11 more

[JayanKandathil]

It looks like the JMS configuration of the WebSphere instance is not correct.  Is the messaging engine running in the same JVM process as LiveCycle?

[ibm_lifesci]

Hi Jayan,

Yes, the Messaging Engine and LiveCycle ES (LCES) are on the same JVM process on the same node of WebSphere.

Which JMS configuration of the WebSphere instance do you suspect is incorrect?

I see these messages in the System Out when the LCES invocation don't work:

[7/19/09 5:11:56:454 EDT] 00000018 SibMessage    I   [IDP_JMS_Bus:NCSUSRADMFP08Node02.server1-IDP_JMS_Bus] CWSIS1546I: The messaging engine, ME_UUID=72F0B32B6B13CBB3, INC_UUID=5F4A5F4A92460BB9, has lost an existing lock or failed to gain an initial lock on the data store.
[7/19/09 5:11:56:454 EDT] 00000018 SibMessage    I   [IDP_JMS_Bus:NCSUSRADMFP08Node02.server1-IDP_JMS_Bus] CWSIS1538I: The messaging engine, ME_UUID=72F0B32B6B13CBB3, INC_UUID=5F4A5F4A92460BB9, is attempting to obtain an exclusive lock on the data store.
[7/19/09 5:12:01:460 EDT] 00000018 SibMessage    I   [IDP_JMS_Bus:NCSUSRADMFP08Node02.server1-IDP_JMS_Bus] CWSIS1546I: The messaging engine, ME_UUID=72F0B32B6B13CBB3, INC_UUID=5F4A5F4A92460BB9, has lost an existing lock or failed to gain an initial lock on the data store.
[7/19/09 5:12:01:460 EDT] 00000018 SibMessage    I   [IDP_JMS_Bus:NCSUSRADMFP08Node02.server1-IDP_JMS_Bus] CWSIS1538I: The messaging engine, ME_UUID=72F0B32B6B13CBB3, INC_UUID=5F4A5F4A92460BB9, is attempting to obtain an exclusive lock on the data store.

And then some times after a restart of server1, I see these messages and then of course the LCES invocations work:

[7/22/09 10:05:35:781 EDT] 00000014 SibMessage    I   [IDP_JMS_Bus:NCSUSRADMFP08Node02.server1-IDP_JMS_Bus] CWSIP0212I: Messaging engine NCSUSRADMFP08Node02.server1-IDP_JMS_Bus on bus IDP_JMS_Bus is starting to reconcile the WCCM destination and link configuration.
[7/22/09 10:05:36:047 EDT] 00000014 SibMessage    I   [IDP_JMS_Bus:NCSUSRADMFP08Node02.server1-IDP_JMS_Bus] CWSIP0213I: Messaging engine NCSUSRADMFP08Node02.server1-IDP_JMS_Bus on bus IDP_JMS_Bus has finished reconciling the WCCM destination and link configuration.
[7/22/09 10:05:36:484 EDT] 00000014 SibMessage    I   [IDP_JMS_Bus:NCSUSRADMFP08Node02.server1-IDP_JMS_Bus] CWSID0016I: Messaging engine NCSUSRADMFP08Node02.server1-IDP_JMS_Bus is in state Started.

attached you will find the complete log

[JayanKandathil]

From the log you copy-pasted, this much is clear:

Your messaging engine with the ID "72F0B32B6B13CBB3" is attempting to obtain an exclusive lock on the JMS data store (SIB tables in the database).  See here:

http://blogs.adobe.com/livecycle/2007/07/what_are_those_sibtables_in_my.html

It fails ("failed to gain an initial lock on the data store").  This is probably because the ME_UUID in the table SIBOWNER is different from 72F0B32B6B13CBB3, or the DBMS is refusing to let it get a lock on the table.

Try the following:

1) Shut down the WebSphere instance hosting LiveCycle and the messaging engine.

2) Get the DBA involved and drop the SIB tables (no other tables should be dropped) and get confirmation that this has been done.

3) Start the WebSphere instance

4) Watch the logs, and make sure those tables have ben re-created