JBoss for AdobeLiveCycle ES v8.0 Needs constant restart

[pagan_dev]

Hi,

Every monday we start getting errors from AdobeLiveCycle "java.sql.SQLException: No operations allowed after connection closed." The only thing that gets it up and running is restarting the JBoss windows service. I found this document on the Adobe site acknowledging the error but saying its a non-issue:

http://help.adobe.com/en_US/livecycle/es/errorCodeReference.pdf  Page 64

The only thing I want is to proactively restart it every monday before start of business day to see if that avoids the errors that come soon after people start using it. I've noticed that net stop and net start do not work on this service. Is there a way to restart this service via a batch file?

Sorry if I dumped this in wrong thread but it was closest related thread I could find.

Thanks,

[chetanm_oct]

Can you provide some more details about the log.

[pagan_dev]

Here's the entire error that I received including stack trace. It seems to be directly related to the 2 day period of down time. Sunday night I restarted the JBoss service and no errors were encountered Monday morning.

Sorry for the late response I've been juggling alot. Thank you:

java.sql.SQLException: No operations allowed after connection closed.Connection was implicitly closed due to underlying exception/error:

** BEGIN NESTED EXCEPTION **

com.mysql.jdbc.CommunicationsException

MESSAGE: Communications link failure due to underlying exception:

** BEGIN NESTED EXCEPTION **

java.net.SocketException

MESSAGE: Connection reset

STACKTRACE:

java.net.SocketException: Connection reset

       at java.net.SocketInputStream.read(SocketInputStream.java:168)

       at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:113)

       at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:160)

       at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:188)

       at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1902)

       at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2351)

       at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)

       at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1571)

       at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1666)

       at com.mysql.jdbc.Connection.execSQL(Connection.java:2988)

       at com.mysql.jdbc.Connection.commit(Connection.java:2161)

       at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.jdbcCommit(BaseWrapperManagedConnection.java:436)

       at org.jboss.resource.adapter.jdbc.WrappedConnection.commit(WrappedConnection.java:331)

       at sun.reflect.GeneratedMethodAccessor253.invoke(Unknown Source)

       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

       at java.lang.reflect.Method.invoke(Method.java:585)

       at org.quartz.impl.jdbcjobstore.AttributeRestoringConnectionInvocationHandler.invoke(AttributeRestoringConnectionInvocationHandler.java:71)

       at $Proxy241.commit(Unknown Source)

       at org.quartz.impl.jdbcjobstore.JobStoreSupport.commitConnection(JobStoreSupport.java:3523)

       at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3663)

       at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2654)

       at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:235)

** END NESTED EXCEPTION **

Last packet sent to the server was 750 ms ago.

STACKTRACE:

com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception:

** BEGIN NESTED EXCEPTION **

java.net.SocketException

MESSAGE: Connection reset

STACKTRACE:

java.net.SocketException: Connection reset

       at java.net.SocketInputStream.read(SocketInputStream.java:168)

       at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:113)

       at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:160)

       at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:188)

       at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1902)

       at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2351)

       at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)

       at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1571)

       at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1666)

       at com.mysql.jdbc.Connection.execSQL(Connection.java:2988)

       at com.mysql.jdbc.Connection.commit(Connection.java:2161)

       at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.jdbcCommit(BaseWrapperManagedConnection.java:436)

       at org.jboss.resource.adapter.jdbc.WrappedConnection.commit(WrappedConnection.java:331)

       at sun.reflect.GeneratedMethodAccessor253.invoke(Unknown Source)

       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

       at java.lang.reflect.Method.invoke(Method.java:585)

       at org.quartz.impl.jdbcjobstore.AttributeRestoringConnectionInvocationHandler.invoke(AttributeRestoringConnectionInvocationHandler.java:71)

       at $Proxy241.commit(Unknown Source)

       at org.quartz.impl.jdbcjobstore.JobStoreSupport.commitConnection(JobStoreSupport.java:3523)

       at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3663)

       at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2654)

       at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:235)

** END NESTED EXCEPTION **

Last packet sent to the server was 750 ms ago.

       at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)

       at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)

       at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1571)

       at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1666)

       at com.mysql.jdbc.Connection.execSQL(Connection.java:2988)

       at com.mysql.jdbc.Connection.commit(Connection.java:2161)

       at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.jdbcCommit(BaseWrapperManagedConnection.java:436)

       at org.jboss.resource.adapter.jdbc.WrappedConnection.commit(WrappedConnection.java:331)

       at sun.reflect.GeneratedMethodAccessor253.invoke(Unknown Source)

       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

       at java.lang.reflect.Method.invoke(Method.java:585)

       at org.quartz.impl.jdbcjobstore.AttributeRestoringConnectionInvocationHandler.invoke(AttributeRestoringConnectionInvocationHandler.java:71)

       at $Proxy241.commit(Unknown Source)

       at org.quartz.impl.jdbcjobstore.JobStoreSupport.commitConnection(JobStoreSupport.java:3523)

       at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3663)

       at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2654)

       at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:235)

** END NESTED EXCEPTION **

[pavrda]

It seems, your database or network connection link shuts down. Do you have it installed on the same machine with jboss? Do you know about some night periodic services? Like server backup, database cleaning, logs rotating ...

[pagan_dev]

I thought it might be related to the Symantec anti virus running on the server so we turned it off and we're stil receiving the error. This error is now occuring every night. I started looking through all the files in the directory where MYSQL is installed and located a file called "[serverName].err" In this file there was an error logged at 2:37 AM:

090820  2:37:59  InnoDB: Operating system error number 33 in a file operation.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/mysql/en/Operating_System_error_codes.html
InnoDB: File name .\ibdata1
InnoDB: File operation call: 'aio write'.

I looked up this error code in the URL provided in the error message and the description is:

The process cannto access the file because another process has locked a portion of the file.

I then proceeded to speak with the support guy that handles server administration and he told me the backup on the server was running between 1:30 and 3:37 AM and it backs up all files on that server. I believe the previous poster was pointing in the right direction. I believe what is happenning is the server backup is accessing the file but the MYSQL service is still running so it tries to write to the file but cannot it then fails and is unable to recover from that error. This would explain why simply restarting the service brings it back up without a hitch. I will be arranging with support to ignore that given MYSQL directory when backing up to see if that eliminates the file lock. I will post back my results.

[JayanKandathil]

However, ibdata1 is a key InnoDB file that you actually want to get backed up for future recovery.

[pagan_dev]

Do you have a general idea of what is stored in that file? I'm guessing if its that significant that we would have to allow the back up to run and just schedule a batch file to run every morning and restart the Service using net stop and net start. I would like to avoid this type of hack solution I find it weird that MySql just throws it hands in the air and says "I quit" simply because it was unable to reach 1 file in 1 attempt.

[pavrda]

I sugest you dump the database, reinstall it and restore again. It is not complicated and I think it will help you to be sure the database itself is not the problem.

--- Jaroslav

[JayanKandathil]

This thread explains it better than I can:

http://forum.percona.com/index.php/m/1437/#msg_1437

[pagan_dev]

The issue was related to the version of JBoss running on the machine failing while trying to connect to file that had a partial lock because of a server backup. JBoss would fail and not come back up. The solution was to allow the back up to continue and schedule a windows task that executes batch script to restart the livecycle service which in turn restarts jboss.