[rules-users] Transaction problems in high concurrency JBPM+Drools+Hibernate instalation
Alberto R. Galdo
argaldo at gmail.com
Mon Nov 12 13:30:56 EST 2012
Let me expand a bit on this.
What we have is web application that interacts with long running persisted
processes using JBPM and Drools ( JPA and Hibernate ). We cannot limit the
request from our users, they're concurrent by nature and as such there's
almost always more than one thread trying to interact with jBPM and the
knowledge session.
When running in this kind of enviroment using a StatefulKnowledgeSession,
what we are observing is an increase of rollbacks and transaction problems
in hibernate as the concurrency increases ( "proxy handle is no longer
valid", "Flush during cascade is dangerous", "Could not commit transaction"
and so on ). None of them is related to our application. And all can be
traced up until org.drools.persistence.jta.JtaTransactionManager
Whenever we see that kind of exceptions, the problems are of different
nature and generally bad ... task doesn't get created, process execution
die without chance of later restoring it's state ... task state change
doesn't happen ...
One obvious solution would be to synchronize all our methods that interact
with JBPM just to queue the requests but that, by experience, leads to
deadlocks in JBPM when two threads try to get a lock to execute a certain
operation ( by the nature of our application, most of the time something
related with human task life cycle things ).
As JBPM and Drools manage themselves their persistence contexts and manage
the transactions on their own pace, we are in a situation where little can
be done from our part ( this statement can be very wrong, please shout if
it is ).
We're using JBPM for human task related work, our shortest process lasts 1
day and several of our processes can last up to 1 month ... ( that's why we
need high avalailabily in the form of persistence both for Drools and JBPM
), all of them need human tasks to be completed to resume the execution of
the process.
Our Drools session is created like this ( snippet, only relevant code ):
// Transaction environment
Context ctx = new InitialContext();
TransactionManager txm =
(TransactionManager)ctx.lookup("java:/TransactionManager");
UserTransaction ut = (UserTransaction)ctx.lookup("java:comp/UserTransaction");
// JPA persistence Context
emf = Persistence.createEntityManagerFactory( "org.jbpm.persistence.jpa" );
// KnowledgeBase
env = KnowledgeBaseFactory.newEnvironment();
env.set( EnvironmentName.ENTITY_MANAGER_FACTORY, emf );
env.set(EnvironmentName.TRANSACTION_MANAGER,txm);
env.set(EnvironmentName.TRANSACTION,ut);
// Human task service
TaskService taskService = new org.jbpm.task.service.TaskService(emf,
SystemEventListenerFactory.getSystemEventListener());
Map<String, User> users = new HashMap<String, User>();
KnowledgeBuilder kbuilder = KnowledgeBuilderFactory.newKnowledgeBuilder();
// process definition files follow
kbuilder.add(ResourceFactory.newInputStreamResource(bpmn_process),ResourceType.BPMN2);
....
KnowledgeBase kbase = kbuilder.newKnowledgeBase();
...
if (thereisasession){
// loads the persisted session
sesion = JPAKnowledgeService.loadStatefulKnowledgeSession(sessionId,
kbase, null, env );
} else {
sesion = JPAKnowledgeService.newStatefulKnowledgeSession( kbase, null, env );
}
...
// Human task handler
LocalHTWorkItemHandler localHTWorkItemHandler = new
LocalHTWorkItemHandler(humanTaskClient, sesion,
OnErrorAction.RETHROW);
localHTWorkItemHandler.connect();
sesion.getWorkItemManager().registerWorkItemHandler("Human Task",
localHTWorkItemHandler);
Our JPA environment is configured like this ( only relevant information
included ):
<persistence
version="1.0"
xsi:schemaLocation=
"http://java.sun.com/xml/ns/persistence
http://java.sun.com/xml/ns/persistence/persistence_1_0.xsd
http://java.sun.com/xml/ns/persistence/orm
http://java.sun.com/xml/ns/persistence/orm_1_0.xsd"
xmlns:orm="http://java.sun.com/xml/ns/persistence/orm"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://java.sun.com/xml/ns/persistence">
<persistence-unit name="org.jbpm.persistence.jpa" transaction-type="JTA">
. ...
<property name="hibernate.dialect"
value="org.hibernate.dialect.PostgreSQLDialect"/>
<property name="hibernate.connection.autocommit" value="false" />
<property name="hibernate.max_fetch_depth" value="3"/>
<property name="hibernate.hbm2ddl.auto" value="update" />
<property name="hibernate.show_sql" value="false" />
<property name="hibernate.transaction.jta.platform"
value="org.hibernate.service.jta.platform.internal.JBossAppServerJtaPlatform"
/>
<property name="hibernate.order_updates" value="true"/>
<property name="hibernate.connection.isolation" value="2"/>
...
</persistence>
Our pooled XA datasource for JBPM is defined like this ( JBoss 7, IronJCAmar) :
<xa-datasource jta="true" jndi-name="java:/JBPM_PG_DS"
pool-name="JBPM_PG_DS_Pool" enabled="true" use-java-context="true"
use-ccm="false">
<xa-datasource-property name="ServerName">localhost</xa-datasource-property>
<xa-datasource-property name="PortNumber">5432</xa-datasource-property>
<xa-datasource-property name="DatabaseName">JBPM</xa-datasource-property>
<xa-datasource-property name="User">jbpm</xa-datasource-property>
<xa-datasource-property name="Password">jbpm</xa-datasource-property>
<driver>postgresXA</driver>
<transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation>
<xa-pool>
<min-pool-size>25</min-pool-size>
<max-pool-size>100</max-pool-size>
<prefill>true</prefill>
<use-strict-min>false</use-strict-min>
</xa-pool>
<security>
<user-name>jbpm</user-name>
<password>jbpm</password>
</security>
<validation>
<check-valid-connection-sql>SELECT 1</check-valid-connection-sql>
<validate-on-match>false</validate-on-match>
<background-validation>false</background-validation>
</validation>
</xa-datasource>
I'm now wondering what to do to improve the concurrency of the system
to avoid those transaction problems ....
- Is there additional parameter/configuration we should be doing?
- Any magic hibernate, JPA or JBoss JCA parameer we should be using?
- Will dividing the persistence context into two, one for Drools
related info and one for JBPM info will help?
- What about creating a Drools session with
org.drools.conf.MultithreadEvaluationOption.YES ??
We're really getting stuck with this ... :(
Alberto R. Galdo
argaldo at gmail.com
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCS/IT/S/M s a C+++ L++ P E--- W++ w
M++ PS tv t++ b++ G h-- r+++ y+
------END GEEK CODE BLOCK------
On Mon, Nov 12, 2012 at 9:38 AM, Alberto R. Galdo <argaldo at gmail.com> wrote:
> Hi guys,
>
> We have an application built around long running JBPM processes with a
> large user base and high concurrency ... early on we began to have issues
> with transactions in JBPM and nowadays are watching lots of different
> realizations of transaction managing problems that lead to dead human tasks
> ( state: ERROR ), errors running processes and so on ( different
> exceptions follows ):
>
> 13:22:48,261 WARN [org.drools.persistence.jta.JtaTransactionManager]
> (pool-11-thread-1) Unable to commit transaction:
> javax.transaction.RollbackException: ARJUNA016053: Could not commit
> transaction
> Caused by: javax.persistence.PersistenceException: error during managed
> flush
>
>
>
>
> Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not
> commit transaction.
> at
> com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1177)
> at
> com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:117)
> at
> com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
> at
> org.jboss.tm.usertx.client.ServerVMClientUserTransaction.commit(ServerVMClientUserTransaction.java:167)
> at
> org.drools.persistence.jta.JtaTransactionManager.commit(JtaTransactionManager.java:179)
> [drools-persistence-jpa-6.0.0-SNAPSHOT.jar:6.0.0-SNAPSHOT]
>
>
>
>
> Caused by: javax.persistence.PersistenceException:
> org.hibernate.HibernateException: Flush during cascade is dangerous
> at
> org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1361)
> [hibernate-entitymanager-4.0.1.Final.jar:4.0.1.Final]
> at
> org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1289)
> [hibernate-entitymanager-4.0.1.Final.jar:4.0.1.Final]
>
>
>
>
> 11:46:23,754 ERROR [stderr] (schedulerVT_Worker-8)
> javax.persistence.PersistenceException: org.hibernate.HibernateException:
> proxy handle is no longer valid
> 11:46:23,754 ERROR [stderr] (schedulerVT_Worker-8) at
> org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1361)
> 11:46:23,755 ERROR [stderr] (schedulerVT_Worker-8) at
> org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1289)
> 11:46:23,756 ERROR [stderr] (schedulerVT_Worker-8) at
> org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:261)
> 11:46:23,756 ERROR [stderr] (schedulerVT_Worker-8) at
>
>
>
> So, we're running this installation by the manual. Are using JBoss 7 as
> application server, it's transaction manager, Hibernate 4.x as entity
> manager, our connection pool delivers READ_COMMITED connections ( so does
> hibernate ) ...
>
> Is there anything else we should be doing to avoid this transactional
> problems? Has anyone ever tested JBPM in such concurrent scenario? Any
> experiences?
>
>
> Alberto R. Galdo
> argaldo at gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/rules-users/attachments/20121112/b69c46ed/attachment.html
More information about the rules-users
mailing list