<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<body link="#355491" alink="#4262a1" vlink="#355491" style="background: #e2e2e2; margin: 0; padding: 20px;">
<div>
<table cellpadding="0" bgcolor="#FFFFFF" border="0" cellspacing="0" style="border: 1px solid #dadada; margin-bottom: 30px; width: 100%; -moz-border-radius: 6px; -webkit-border-radius: 6px;">
<tbody>
<tr>
<td>
<table border="0" cellpadding="0" cellspacing="0" bgcolor="#FFFFFF" style="border: solid 2px #ccc; background: #dadada; width: 100%; -moz-border-radius: 6px; -webkit-border-radius: 6px;">
<tbody>
<tr>
<td bgcolor="#000000" valign="middle" height="58px" style="border-bottom: 1px solid #ccc; padding: 20px; -moz-border-radius-topleft: 3px; -moz-border-radius-topright: 3px; -webkit-border-top-right-radius: 5px; -webkit-border-top-left-radius: 5px;">
<h1 style="color: #333333; font: bold 22px Arial, Helvetica, sans-serif; margin: 0; display: block !important;">
<!-- To have a header image/logo replace the name below with your img tag -->
<!-- Email clients will render the images when the message is read so any image -->
<!-- must be made available on a public server, so that all recipients can load the image. -->
<a href="http://community.jboss.org/index.jspa" style="text-decoration: none; color: #E1E1E1">JBoss Community</a></h1>
</td>
</tr>
<tr>
<td bgcolor="#FFFFFF" style="font: normal 12px Arial, Helvetica, sans-serif; color:#333333; padding: 20px; -moz-border-radius-bottomleft: 4px; -moz-border-radius-bottomright: 4px; -webkit-border-bottom-right-radius: 5px; -webkit-border-bottom-left-radius: 5px;"><h3 style="margin: 10px 0 5px; font-size: 17px; font-weight: normal;">
JobExecutor becomes broken and cannot recover (JBPM3.2.2)
</h3>
<span style="margin-bottom: 10px;">
created by <a href="http://community.jboss.org/people/soft">Petr Sochurek</a> in <i>jBPM</i> - <a href="http://community.jboss.org/message/577160#577160">View the full discussion</a>
</span>
<hr style="margin: 20px 0; border: none; background-color: #dadada; height: 1px;">
<div class="jive-rendered-content"><p>Problem: JobExecutor thread becomes broken due to unknown transactional problem. As a reason no tasks are executed and only JBoss restart helps.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>Configuration and environment<br/>- components stack: JBOSS 4.2.3, JBOSS TS 4.2.3, JBPM 3.2.2, Hibernate 3.2.4.sp1.cp04<br/>- database server: MySQL 5.1.47<br/>- distributed transaction only within the one database server <br/>- operated on several Linux environments</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>The problem occurs irregurarly (once per several days and only on some environments). When it happens following errors are logged by JBPMExecutor thread:</p><p>2010-12-15 15:53:20,566 WARN  <a class="jive-link-external-small" href="com.arjuna.ats.arjuna.logging.arjLoggerI18N">com.arjuna.ats.arjuna.logging.arjLoggerI18N</a> [com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator_2</p><p>TwoPhaseCoordinator.beforeCompletion - failed for nulljava.lang.NullPointerExceptionat org.hibernate.connection.DatasourceConnectionProvider.closeConnection(DatasourceConnectionProvider.java:74)at org.hibernate.jdbc.ConnectionManager.closeConnection(ConnectionManager.java:451)at org.hibernate.jdbc.ConnectionManager.aggressiveRelease(ConnectionManager.java:406)at org.hibernate.jdbc.ConnectionManager.afterStatement(ConnectionManager.java:281)at org.hibernate.jdbc.ConnectionManager.flushEnding(ConnectionManager.java:480)at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:305)at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27)at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000)at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:338)at org.hibernate.transaction.CacheSynchronization.beforeCompletion(CacheSynchronization.java:59)at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:114)at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:247)at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:86)at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:177)at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1389)at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:135)at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:87)at org.jboss.tm.usertx.client.ServerVMClientUserTransaction.commit(ServerVMClientUserTransaction.java:140)at org.jbpm.persistence.jta.JtaDbPersistenceService.endJtaTransaction(JtaDbPersistenceService.java:80)at org.jbpm.persistence.jta.JtaDbPersistenceService.close(JtaDbPersistenceService.java:41)at org.jbpm.svc.Services.close(Services.java:225)at org.jbpm.JbpmContext.close(JbpmContext.java:139)at org.jbpm.job.executor.JobExecutorThread.acquireJobs(JobExecutorThread.java:144)at org.jbpm.job.executor.JobExecutorThread.run(JobExecutorThread.java:58)2010-12-15 15:53:20,566 WARN  <a class="jive-link-external-small" href="com.arjuna.ats.arjuna.logging.arjLoggerI18N">com.arjuna.ats.arjuna.logging.arjLoggerI18N</a> <a class="jive-link-external-small" href="com.arjuna.ats.arjuna.coordinator.CheckedAction_2">com.arjuna.ats.arjuna.coordinator.CheckedAction_2</a> - CheckedAction::check - atomic action</p><p>a471f1f:c46e:4d08ce0f:c5a aborting with 1 threads active!2010-12-15 15:53:20,582 ERROR <a class="jive-link-external-small" href="org.jbpm.svc.Services">org.jbpm.svc.Services</a> problem closing service 'persistence'org.jbpm.JbpmException: couldn't commit JTA transactionat org.jbpm.persistence.jta.JtaDbPersistenceService.endJtaTransaction(JtaDbPersistenceService.java:82)at org.jbpm.persistence.jta.JtaDbPersistenceService.close(JtaDbPersistenceService.java:41)at org.jbpm.svc.Services.close(Services.java:225)at org.jbpm.JbpmContext.close(JbpmContext.java:139)at org.jbpm.job.executor.JobExecutorThread.acquireJobs(JobExecutorThread.java:144)at org.jbpm.job.executor.JobExecutorThread.run(JobExecutorThread.java:58)Caused by: java.lang.NullPointerExceptionat com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:235)at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:86)at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:177)at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1389)at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:135)at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:87)at org.jboss.tm.usertx.client.ServerVMClientUserTransaction.commit(ServerVMClientUserTransaction.java:140)at org.jbpm.persistence.jta.JtaDbPersistenceService.endJtaTransaction(JtaDbPersistenceService.java:80)... 5 more2010-12-15 15:53:20,589 ERROR <a class="jive-link-external-small" href="org.jbpm.job.executor.JobExecutorThread">org.jbpm.job.executor.JobExecutorThread</a> exception in job executor thread. waiting 5000 millisecondsorg.jbpm.JbpmException: problem closing services {persistence=org.jbpm.JbpmException: couldn't commit JTA transaction}at org.jbpm.svc.Services.close(Services.java:245)at org.jbpm.JbpmContext.close(JbpmContext.java:139)at org.jbpm.job.executor.JobExecutorThread.acquireJobs(JobExecutorThread.java:144)at org.jbpm.job.executor.JobExecutorThread.run(JobExecutorThread.java:58)Caused by: org.jbpm.JbpmException: couldn't commit JTA transactionat org.jbpm.persistence.jta.JtaDbPersistenceService.endJtaTransaction(JtaDbPersistenceService.java:82)at org.jbpm.persistence.jta.JtaDbPersistenceService.close(JtaDbPersistenceService.java:41)at org.jbpm.svc.Services.close(Services.java:225)... 3 moreCaused by: java.lang.NullPointerExceptionat com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:235)at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:86)at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:177)at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1389)at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:135)at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:87)at org.jboss.tm.usertx.client.ServerVMClientUserTransaction.commit(ServerVMClientUserTransaction.java:140)at org.jbpm.persistence.jta.JtaDbPersistenceService.endJtaTransaction(JtaDbPersistenceService.java:80)</p><p>... 5 more</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>From now all the successive iterations of JobExecutor always end up with following error (and the waiting time doubles after error):</p><p>2010-12-15 15:53:23,524 ERROR <a class="jive-link-external-small" href="org.jbpm.job.executor.JobExecutorThread">org.jbpm.job.executor.JobExecutorThread</a> exception in job executor thread. waiting 10000 milliseconds</p><p>org.jbpm.JbpmException: couldn't start JTA transaction</p><p>at org.jbpm.persistence.jta.JtaDbPersistenceService.beginJtaTransaction(JtaDbPersistenceService.java:55)</p><p>at org.jbpm.persistence.jta.JtaDbPersistenceService.(JtaDbPersistenceService.java:28)</p><p>at org.jbpm.persistence.jta.JtaDbPersistenceServiceFactory.openService(JtaDbPersistenceServiceFactory.java:17)</p><p>at org.jbpm.svc.Services.getService(Services.java:144)</p><p>at org.jbpm.svc.Services.getPersistenceService(Services.java:183)</p><p>at org.jbpm.JbpmContext.getPersistenceService(JbpmContext.java:628)</p><p>at org.jbpm.JbpmContext.getJobSession(JbpmContext.java:561)</p><p>at org.jbpm.job.executor.JobExecutorThread.acquireJobs(JobExecutorThread.java:112)</p><p>at org.jbpm.job.executor.JobExecutorThread.run(JobExecutorThread.java:58)</p><p>Caused by: javax.transaction.NotSupportedException</p><p>at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.begin(BaseTransaction.java:79)</p><p>at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.begin(BaseTransactionManagerDelegate.java:77)</p><p>at org.jboss.tm.usertx.client.ServerVMClientUserTransaction.begin(ServerVMClientUserTransaction.java:124)</p><p>at org.jbpm.persistence.jta.JtaDbPersistenceService.beginJtaTransaction(JtaDbPersistenceService.java:53)</p><p>... 8 more</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>I suppose that the root cause is some database connection problem or network problem (maybe firewall). The problem occurs irregurarly only on production environments and I wasn't able to find out what it coudl be.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>Anyway, I would expect that  regardless of such low-level errors, the JBPM (or Hibernate, Arjuna or anything else inside) should be able to fully recover.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>Locally I can reproduce problem by locking Mysql database tables using scenario:</p><p>1) deployed testing jbpm process whose transitions are changed automatically every minute <br/>2) locked mysql database tables using "FLUSH TABLES WITH READ LOCK \G;" command<br/>3) wait 1 hour<br/>4) unlock database tables using "UNLOCK TABLES;" command<br/>5) now errors (see above) appear in the log</p><p><br/>Can you please give me any advice how to ensure that JBPM recovers correctly?</p><p>Maybe I omited some important configuration in Hibernate/Arjuna/JBPM, Jboss (the JBPM datasource is configured correctly now).</p><p>I cannot belive that the JBOSS restart is sufficent solution.</p><p><br/>Thanks a lot.<br/>Petr</p></div>
<div style="background-color: #f4f4f4; padding: 10px; margin-top: 20px;">
<p style="margin: 0;">Reply to this message by <a href="http://community.jboss.org/message/577160#577160">going to Community</a></p>
<p style="margin: 0;">Start a new discussion in jBPM at <a href="http://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2034">Community</a></p>
</div></td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</div>
</body>
</html>