[rules-users] Possibly race condition in a persisted Drools + JBPM during a StatefulKnowledgeSession

Mauricio Salatino salaboy at gmail.com
Tue May 29 12:19:07 EDT 2012 

:)
Keep your mind open, because there is no single solution for all the
problems.

On Tue, May 29, 2012 at 1:15 PM, Alberto R. Galdo <argaldo at gmail.com> wrote:

> Thank you very much Mauricio. You've been of great help. Now is time for
> us to cook all this valuable information and go for a solution. xD
>
> Alberto R. Galdo
> argaldo at gmail.com
>
>
>
> On Tue, May 29, 2012 at 5:59 PM, Mauricio Salatino <salaboy at gmail.com>wrote:
>
>> Comments inline,
>>
>> On Tue, May 29, 2012 at 12:24 PM, Alberto R. Galdo <argaldo at gmail.com>wrote:
>>
>>>    Sure!. I didn't mention that we're running in a OSGi environment and
>>> we're getting the session from JPAKnowledgeService like that:
>>>
>>>
>>>
>>> EntityManagerFactory emf =
>>> Persistence.createEntityManagerFactory("com.package");
>>>
>>> Environment env = KnowledgeBaseFactory.newEnvironment();
>>>
>>> env.set(EnvironmentName.ENTITY_MANAGER_FACTORY, emf);
>>>
>>> env.set(EnvironmentName.TRANSACTION_MANAGER,(TransactionManager)
>>> bundleContext.getService(this.tmServiceRef));
>>>
>>> env.set(EnvironmentName.TRANSACTION, (UserTransaction)
>>> bundleContext.getService(this.userTransactionServiceRef));
>>>
>>> env.set(EnvironmentName.GLOBALS, new MapGlobalResolver());
>>>
>>> env.set(EnvironmentName.OBJECT_MARSHALLING_STRATEGIES, new
>>> ObjectMarshallingStrategy[] {
>>> MarshallerFactory.newSerializeMarshallingStrategy() });
>>>
>>> final StatefulKnowledgeSession ksession =
>>> JPAKnowledgeService.newStatefulKnowledgeSession(kbase, sessionConfig, env);
>>>
>>>
>>>    We are just following the manual ( and some acquired knowledge
>>> through the days ... ) .... I understand that inserting an event and
>>> starting a process at the same time would lead to the current situation,
>>> but Isn't this the reason why we are using drools with drools fusion + jbpm
>>> to build a CEP?
>>>
>> I'm not sure about your reasons :)
>>
>>>
>>>    We chose Drools and a statefulsession in a fireUntilHalt loop just
>>> because we would be able to have our rules and processes reacting to events
>>> that enter the system at their own pace ... and found ourselves in this
>>> nightmare when trying to make the system fault-tolerant through
>>> persistence... :(
>>>
>>> FireUntilHalt and Persistence will cause "conflicts", because
>> fireUntilHalts by default creates a different thread == race conditions.
>>
>>
>>>    Do you think we should change our fault-tolerance strategy by
>>> reconstructing the knowledge session, process and tasks state by our own
>>> means leaving Drools and JBPM JPA persistence appart?
>>>
>>>  A common problem is to think that a session will do everything for you.
>> I'm not saying that you need to separate your processes from your rules,
>> I'm just saying that probably decoupling responsibilities will help you to
>> tackle down some of the problems that you are having. If you have in memory
>> processes and fusion entry points, you can keep those together and in a
>> separate persistence session handle all the human task interactions and
>> long running processes. If you are getting events in real time and you want
>> to process them, you cannot expect to persist all the session state and
>> have no performance impacts.
>>
>>>
>>> Alberto R. Galdo
>>> argaldo at gmail.com
>>>
>>>
>>> Cheers
>>
>>>
>>> On Tue, May 29, 2012 at 4:43 PM, Mauricio Salatino <salaboy at gmail.com>wrote:
>>>
>>>> Ok, so probably I'm not understanding your scenario, but are you using
>>>> the JPAKnowledgeService to create persistent sessions?
>>>> If you use that everything is aware of everything :) Drools and jBPM5
>>>> will run using the same persistence resources. For each interaction:
>>>> startProcess, insert, update, fireAllRules a transaction will be created.
>>>> If you are starting a process and at the same time inserting a Fusion event
>>>> you will be generating two transaction that will fight for the resources
>>>> (the same resources because you have only one session).
>>>> Obviously if you take the persistence mechanisms outside of the
>>>> discussion everything will work perfectly fine, because you don't have any
>>>> transactional resource to fight for :)
>>>>
>>>> Hope it helps.
>>>> Cheers
>>>>
>>>>
>>>> On Tue, May 29, 2012 at 11:36 AM, Alberto R. Galdo <argaldo at gmail.com>wrote:
>>>>
>>>>>    Your answer is confusing me, I think I'm not getting it right.
>>>>>
>>>>>    It seems that we you're saying that will have to modify JBPM or
>>>>> Drools behaviour to get our architecture done ... but I don't think we are
>>>>> doing nothing special than what Drools & JBPM are able to do in a
>>>>> non-persisting enviroment ( at least without JBPM persisted ).
>>>>>
>>>>>    Maybe deeping a bit more the architecture will lead to a better
>>>>> understanding of the problem:
>>>>>
>>>>>    We have a Drools StatefulKnowledgeSession wich is populated with
>>>>> packages of rules and different BPMN 2.0 process definitions. That session
>>>>> in Drools is persisted using JPA and our JTA provider. Our solution is a
>>>>> CEP that uses events that get into the session using Drools Fusion. There,
>>>>> several rules fire and then several consequences start processes wich
>>>>> contain both Tasks and HumanTasks. Those rule activations are already in a
>>>>> queue, the Agenda. Given that JBPM acts as the processmanager provider for
>>>>> Drools, our processes get started in JBPM, and it is persisted using it's
>>>>> own JPA manager.  Just rules, that start processes in response to events in
>>>>> a fireUntilHalt loop in a persistent enviroment, both for Drools & JBPM.
>>>>> Then some processes interact with the knowledge session using BussinessTask
>>>>> nodes( which make other rules fire and maybe launch a different set of
>>>>> processes ) and other simply get to the Stop node. I see nothing wrong
>>>>> here, in fact, taking the persistence appart from the equation, everything
>>>>> runs as beautifuly as expected. We've tested that.
>>>>>
>>>>>    I'm just trying to undersand the problem, I'm pretty 90% sure I'm
>>>>> wrong, but what I think is happening here is that Drools isn't aware that
>>>>> JBPM may be using the same resources and vice-versa, Isn't it?
>>>>>
>>>>>    Shouldn't the solution be to make Drools & JBPM aware of each other
>>>>> in what is related to the syncronization of the transaction management to
>>>>> avoid race conditions?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Alberto R. Galdo
>>>>> argaldo at gmail.com
>>>>>
>>>>>
>>>>> On Tue, May 29, 2012 at 3:58 PM, Mauricio Salatino <salaboy at gmail.com>wrote:
>>>>>
>>>>>> I'm in no way saying this:
>>>>>> "  Let me see if I'm getting this right, What you mean is that there
>>>>>> is no way for the JPA persistence of Drools & JBPM to coexist in the same
>>>>>> knowledge session because of race conditions between them competing for
>>>>>> their shared resources? I'm afraid so.. :(
>>>>>>
>>>>>> They can coexist. As you can see they are working together. If you
>>>>>> take a look at the underlying layers you will see that each operation that
>>>>>> you run against the session is wrapped in a transaction. That leads to your
>>>>>> architecture, If you have a single entry point for your session everything
>>>>>> will work correctly. You can achieve this by receiving all the operations
>>>>>> in a queue and then execute one after the other.
>>>>>> If you have multiple threads interacting with the session, you can
>>>>>> implement a retrying mechanism if the transaction gets rolled back and as
>>>>>> soon as the transaction win the right to commit it will work. It really
>>>>>> depends on the problem that you are trying to solve.
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> On Tue, May 29, 2012 at 10:42 AM, Alberto R. Galdo <argaldo at gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>>    Thanks Mauricio for your kind response, and kick, as usual ;-)
>>>>>>>
>>>>>>>    Sure, we see lots of rolled back transactions ... Everywhere!
>>>>>>>
>>>>>>>    Let me see if I'm getting this right, What you mean is that there
>>>>>>> is no way for the JPA persistence of Drools & JBPM to coexist in the same
>>>>>>> knowledge session because of race conditions between them competing for
>>>>>>> their shared resources? I'm afraid so.. :(
>>>>>>>
>>>>>>>    If it is so, Are there any plans for the integration of a shared
>>>>>>> JPA persistence management of both Drools & JBPM, wich may be the solution
>>>>>>> for this problem? ... Will we be better thinking to change our architecture
>>>>>>> ASAP?
>>>>>>>
>>>>>>> Greets,
>>>>>>>
>>>>>>> Alberto R. Galdo
>>>>>>> argaldo at gmail.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, May 29, 2012 at 1:54 PM, Mauricio Salatino <
>>>>>>> salaboy at gmail.com> wrote:
>>>>>>>
>>>>>>>> Ok, so what is happening is that two or more threads are fighting
>>>>>>>> for the resources, causing that two or more transactions wants to be
>>>>>>>> completed, but just one can win. All the other must be retried. Are you
>>>>>>>> seeing rolled back exceptions in your stack trace?
>>>>>>>> My question to you is if you really need to handle everything in
>>>>>>>> just one big persistent session. I've solved similar issues in the past
>>>>>>>> using more than one session, for example one session to run business
>>>>>>>> processes(persistent) and one or more for receiving and reacting to events.
>>>>>>>> If you have an async way to communicate both sessions there should be no
>>>>>>>> problem.
>>>>>>>> Hope it helps!
>>>>>>>> Cheers
>>>>>>>>
>>>>>>>> On Tue, May 29, 2012 at 7:30 AM, Alberto R. Galdo <
>>>>>>>> argaldo at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>>    We've been for long time now developing a complex event
>>>>>>>>> processing system that ( simplified version ahead !! ) involves a set of
>>>>>>>>> rules that in turn activates a set processes that fulfill human tasks and
>>>>>>>>> other kind of tasks.. This all is running in a StatefulKnowledgeSession
>>>>>>>>> with JPA persistence configured both for Drools and JBPM. We are running an
>>>>>>>>> stack of Drools, JBPM, Drools Integration, Drools fussion, etc..
>>>>>>>>>
>>>>>>>>>    We've been able to persist Drools sessions & JBPM processes in
>>>>>>>>> the same Persistence Context  ( not without pain :( ) using different JTA
>>>>>>>>> implementations including ( but not limited to ) Bitronix & Atomikos.
>>>>>>>>>
>>>>>>>>>     What we are observing here is what seems to be some kind of
>>>>>>>>> race condition between Drools and JBPM when running the knowledge session
>>>>>>>>> when JPA&JTA persistence is configured. Very often, as soon as after 2-3
>>>>>>>>> processes get created as rule's consequences are fired in response of
>>>>>>>>> events inside the session we see how JBPM finds its instance nullified by
>>>>>>>>> Drools when it tries to end a process and persist it.
>>>>>>>>>
>>>>>>>>>    We've been able to find where Drools decides to delete an
>>>>>>>>> instance of the process ... at a given time Drools executes
>>>>>>>>> JPAProcessInstanceManager.clearProcessInstances() [1] when it finalizes a
>>>>>>>>> SingleSessionCommand wich in turn calls disconnect() for *all* the
>>>>>>>>> local-stored processinstances ( wich gets populated with instances of
>>>>>>>>> processes every time a process is started in the knowledge session ):
>>>>>>>>>
>>>>>>>>>     public void clearProcessInstances() {
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         for (ProcessInstance processInstance: new ArrayList<ProcessInstance>(processInstances.values())) {
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>             ((ProcessInstanceImpl) processInstance).disconnect();
>>>>>>>>>
>>>>>>>>>         }
>>>>>>>>>     }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>     So, Drools decides to disconnect all process instances in it's
>>>>>>>>> JPA context without taking in account the state the process is in, and when
>>>>>>>>> an processinstance that is not stopped gets removed then JBPM finds it's
>>>>>>>>> NullPointerException...
>>>>>>>>>
>>>>>>>>>     We've modified the code to make Drools aware of the state of
>>>>>>>>> the process before wiping it from the context   ( no problem here, there
>>>>>>>>> will be no leak as a running processinstance will be removed in future
>>>>>>>>> calls of "clearProcessInstances" given the process is closed ). But
>>>>>>>>> unfortunatelly this seems to resolve this problem, but lots of other
>>>>>>>>> problems ( wich seems also race conditions arise :  for instance: Drools
>>>>>>>>> closes connections to the database and JBPM finds the connection closed,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>     So, we are really worried about using Drools & JBPM in a
>>>>>>>>> persisted environment. Maybe our asumptions are wrong...  Is it possible to
>>>>>>>>> have an scenario like ours given the current Drools & JBPM integration
>>>>>>>>> status for a persistent statefulKnowledge Session?  Did anyone build a
>>>>>>>>> complex event processing system like ours in a unaltered persistence
>>>>>>>>> environment such as provided in Drools and JBPM by default?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Greets,
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://github.com/droolsjbpm/jbpm/blob/master/jbpm-persistence-jpa/src/main/java/org/jbpm/persistence/processinstance/JPAProcessInstanceManager.java
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Alberto R. Galdo
>>>>>>>>> argaldo at gmail.co <argaldo at gmail.com>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> rules-users mailing list
>>>>>>>>> rules-users at lists.jboss.org
>>>>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>  - MyJourney @ http://salaboy.wordpress.com
>>>>>>>>  - Co-Founder @ http://www.jugargentina.org
>>>>>>>>  - Co-Founder @ http://www.jbug.com.ar
>>>>>>>>
>>>>>>>>  - Salatino "Salaboy" Mauricio -
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> rules-users mailing list
>>>>>>>> rules-users at lists.jboss.org
>>>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> rules-users mailing list
>>>>>>> rules-users at lists.jboss.org
>>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>  - MyJourney @ http://salaboy.wordpress.com
>>>>>>  - Co-Founder @ http://www.jugargentina.org
>>>>>>  - Co-Founder @ http://www.jbug.com.ar
>>>>>>
>>>>>>  - Salatino "Salaboy" Mauricio -
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> rules-users mailing list
>>>>>> rules-users at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> rules-users mailing list
>>>>> rules-users at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>  - MyJourney @ http://salaboy.wordpress.com
>>>>  - Co-Founder @ http://www.jugargentina.org
>>>>  - Co-Founder @ http://www.jbug.com.ar
>>>>
>>>>  - Salatino "Salaboy" Mauricio -
>>>>
>>>>
>>>> _______________________________________________
>>>> rules-users mailing list
>>>> rules-users at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>
>>>>
>>>
>>> _______________________________________________
>>> rules-users mailing list
>>> rules-users at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>
>>>
>>
>>
>> --
>>  - MyJourney @ http://salaboy.wordpress.com
>>  - Co-Founder @ http://www.jugargentina.org
>>  - Co-Founder @ http://www.jbug.com.ar
>>
>>  - Salatino "Salaboy" Mauricio -
>>
>>
>> _______________________________________________
>> rules-users mailing list
>> rules-users at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/rules-users
>>
>>
>
> _______________________________________________
> rules-users mailing list
> rules-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
>
>


-- 
 - MyJourney @ http://salaboy.wordpress.com
 - Co-Founder @ http://www.jugargentina.org
 - Co-Founder @ http://www.jbug.com.ar

 - Salatino "Salaboy" Mauricio -
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/rules-users/attachments/20120529/fa3637fc/attachment-0001.html

More information about the rules-users mailing list