Re: [rules-users] Transaction problems in high concurrency JBPM+Drools+Hibernate instalation

Let me expand a bit on this. What we have is web application that interacts with long running persisted processes using JBPM and Drools ( JPA and Hibernate ). We cannot limit the request from our users, they're concurrent by nature and as such there's almost always more than one thread trying to interact with jBPM and the knowledge session. When running in this kind of enviroment using a StatefulKnowledgeSession, what we are observing is an increase of rollbacks and transaction problems in hibernate as the concurrency increases ( "proxy handle is no longer valid", "Flush during cascade is dangerous", "Could not commit transaction" and so on ). None of them is related to our application. And all can be traced up until org.drools.persistence.jta.JtaTransactionManager Whenever we see that kind of exceptions, the problems are of different nature and generally bad ... task doesn't get created, process execution die without chance of later restoring it's state ... task state change doesn't happen ... One obvious solution would be to synchronize all our methods that interact with JBPM just to queue the requests but that, by experience, leads to deadlocks in JBPM when two threads try to get a lock to execute a certain operation ( by the nature of our application, most of the time something related with human task life cycle things ). As JBPM and Drools manage themselves their persistence contexts and manage the transactions on their own pace, we are in a situation where little can be done from our part ( this statement can be very wrong, please shout if it is ). We're using JBPM for human task related work, our shortest process lasts 1 day and several of our processes can last up to 1 month ... ( that's why we need high avalailabily in the form of persistence both for Drools and JBPM ), all of them need human tasks to be completed to resume the execution of the process. Our Drools session is created like this ( snippet, only relevant code ): // Transaction environment Context ctx = new InitialContext(); TransactionManager txm = (TransactionManager)ctx.lookup("java:/TransactionManager"); UserTransaction ut = (UserTransaction)ctx.lookup("java:comp/UserTransaction"); // JPA persistence Context emf = Persistence.createEntityManagerFactory( "org.jbpm.persistence.jpa" ); // KnowledgeBase env = KnowledgeBaseFactory.newEnvironment(); env.set( EnvironmentName.ENTITY_MANAGER_FACTORY, emf ); env.set(EnvironmentName.TRANSACTION_MANAGER,txm); env.set(EnvironmentName.TRANSACTION,ut); // Human task service TaskService taskService = new org.jbpm.task.service.TaskService(emf, SystemEventListenerFactory.getSystemEventListener()); Map<String, User> users = new HashMap<String, User>(); KnowledgeBuilder kbuilder = KnowledgeBuilderFactory.newKnowledgeBuilder(); // process definition files follow kbuilder.add(ResourceFactory.newInputStreamResource(bpmn_process),ResourceType.BPMN2); .... KnowledgeBase kbase = kbuilder.newKnowledgeBase(); ... if (thereisasession){ // loads the persisted session sesion = JPAKnowledgeService.loadStatefulKnowledgeSession(sessionId, kbase, null, env ); } else { sesion = JPAKnowledgeService.newStatefulKnowledgeSession( kbase, null, env ); } ... // Human task handler LocalHTWorkItemHandler localHTWorkItemHandler = new LocalHTWorkItemHandler(humanTaskClient, sesion, OnErrorAction.RETHROW); localHTWorkItemHandler.connect(); sesion.getWorkItemManager().registerWorkItemHandler("Human Task", localHTWorkItemHandler); Our JPA environment is configured like this ( only relevant information included ): <persistence version="1.0" xsi:schemaLocation= "http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_1_0.xsd http://java.sun.com/xml/ns/persistence/orm http://java.sun.com/xml/ns/persistence/orm_1_0.xsd" xmlns:orm="http://java.sun.com/xml/ns/persistence/orm" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/persistence"> <persistence-unit name="org.jbpm.persistence.jpa" transaction-type="JTA"> . ... <property name="hibernate.dialect" value="org.hibernate.dialect.PostgreSQLDialect"/> <property name="hibernate.connection.autocommit" value="false" /> <property name="hibernate.max_fetch_depth" value="3"/> <property name="hibernate.hbm2ddl.auto" value="update" /> <property name="hibernate.show_sql" value="false" /> <property name="hibernate.transaction.jta.platform" value="org.hibernate.service.jta.platform.internal.JBossAppServerJtaPlatform" /> <property name="hibernate.order_updates" value="true"/> <property name="hibernate.connection.isolation" value="2"/> ... </persistence> Our pooled XA datasource for JBPM is defined like this ( JBoss 7, IronJCAmar) : <xa-datasource jta="true" jndi-name="java:/JBPM_PG_DS" pool-name="JBPM_PG_DS_Pool" enabled="true" use-java-context="true" use-ccm="false"> <xa-datasource-property name="ServerName">localhost</xa-datasource-property> <xa-datasource-property name="PortNumber">5432</xa-datasource-property> <xa-datasource-property name="DatabaseName">JBPM</xa-datasource-property> <xa-datasource-property name="User">jbpm</xa-datasource-property> <xa-datasource-property name="Password">jbpm</xa-datasource-property> <driver>postgresXA</driver> <transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation> <xa-pool> <min-pool-size>25</min-pool-size> <max-pool-size>100</max-pool-size> <prefill>true</prefill> <use-strict-min>false</use-strict-min> </xa-pool> <security> <user-name>jbpm</user-name> <password>jbpm</password> </security> <validation> <check-valid-connection-sql>SELECT 1</check-valid-connection-sql> <validate-on-match>false</validate-on-match> <background-validation>false</background-validation> </validation> </xa-datasource> I'm now wondering what to do to improve the concurrency of the system to avoid those transaction problems .... - Is there additional parameter/configuration we should be doing? - Any magic hibernate, JPA or JBoss JCA parameer we should be using? - Will dividing the persistence context into two, one for Drools related info and one for JBPM info will help? - What about creating a Drools session with org.drools.conf.MultithreadEvaluationOption.YES ?? We're really getting stuck with this ... :( Alberto R. Galdo argaldo(a)gmail.com -----BEGIN GEEK CODE BLOCK----- Version: 3.12 GCS/IT/S/M s a C+++ L++ P E--- W++ w M++ PS tv t++ b++ G h-- r+++ y+ ------END GEEK CODE BLOCK------ On Mon, Nov 12, 2012 at 9:38 AM, Alberto R. Galdo <argaldo(a)gmail.com&gt; wrote: