Our LiveCycle installation has been plagued by an unstable network. The database LiveCycle runs off of has been randomly dropping from the network, severing the connection with our application server (JBoss). When this happens LiveCycle enters a dangerous state, where if a user takes action on a task, the process in which that task is involved in goes in a stuck state that cannot be recovered. The process does not stall it just gets stuck, and there is no way to recover from it, except to Terminate and start over. Because the network connectivity will continue to be an issue for us (no one to date has found a reason why servers randomly drop from the network), we have had to come up with some ideas for how to proactively watch for this DB drop happening. We have decided to tackle this issue with a three piece solution:
First we are automatically restarting the application server very early in the morning. This will reestablish the connection with the DB should it have gone down over night.
Second, we have set up a short lived process on the server to catch the exception when it is thrown, and email the sys admin, so that they can restart JBoss (I'm not entirely sure if this will work since the process is on the server that would be having problems connecting to its database to begin with)
Third, we would like to scan the current JBoss server log on the LiveCycle server, at the time a user tries logging into the workspace tool. If we see the exception in the log, then the idea is to display an error message, and keep the user from being able to log in.
The idea here is to cover all our bases until our infastructure team can figure out the network stability problem.
My question is in regards to the third step I have laid out. I realize that this involves building a custom workspace, of which we have already done, but is it possible to check the log file from the workspace tool like I have described? All of the documentation I've been able to dig up to date makes it look like I can only look at files local to the user attempting to log in. If this is true, then this approach may not work for us. Ideally resolving the network stability issue would be the best way to fix this problem, but that issue is outside of my control. Does anyone have any ideas on this, or maybe you have run into a similar problem, and have implimented a different solution. Thank you ahead of time for any assistance you can offer.
I just wonder, is the problem caused by physical problems on the network making the database go down? The reason I ask is that we had a problem with MySQL and JBOSS. Our problem was that after livecycle being inactive for a while, the connection to MySQL was lost, and could not be restored before the server was restarted. I wonder if this could be related to your problem.
Thank you for the response Daniel. I'm not really sure what the cause of the problem is, I just know the way in which the problem affects our server. The problem appears to be something in the overall network down here. We have seen the DB drop on its own from the network, as well as seeing all of our databases drop (like in our DEV, TEST, and PROD environments). Our infastructure team has not been able to track down why this is happening to date. Since they can't find the root cause, and we continue to be troubled by the network issues, I have taken it upon myself to at least minimalize the possibility of having processes hang, and become unrecoverable. Ideally solving the overall network issue would be the best way to address the issue, but not being part of the infastructure team, I do not have access to the network, and cannot trouble-shoot such problems. Our DBs are all on separate servers from LiveCycle, and in the case of our production environment, the SQL database is set up on a cluster. I know that the DBs are dropping off the network from time to time, because I've watched the JBoss log when such a drop has occurred, and am not able to access the database through an SQL client tool when the drop happens. The infastructure team should be able to figure out what is causing their servers to drop, but in the mean time I need a way to stop our data from becoming corrupted.
I understand your concern. It is a bit tricky because it seems to rely on the network/database. I don't think the patch I received from Adobe would help, as it was no problem with the database access itself. If I remember correctly, it was a problem with LC. What I would suggest is that you set up a copy of the database on the LiveCycle server, and use that one untill the problems on the network has been solved. Is that a possible workaround for now? Are you running SQL Server, Oracle or MySQL?
I faced similar issues with the DB dropping off the network as well. My LC runs on a VM. I originally thought the VM may have caused the DB to drop from time to time, but It looks like it's happenning for everybody. Hope someone from Adobe will give us an explanation and how to prevent this.
Thank you for the suggestion. We actually came to the same conclusion on Friday as well. The problem looks to be with the JMS and the Quartz timer that LiveCycle uses to run everything. When the DB drops it is the JMS provider that cannot reconnect itself with the DB. We currently connect to an SQL DB, but are looking into setting up a mySQL DB to run the Quartz timer from locally. Our client requires that we use their SQL DB to connect to and store data. I'll post an update on this once we have everything in place to let everyone know if this configuration will work.