[rules-users] Possibly race condition in a persisted Drools + JBPM during a StatefulKnowledgeSession

Alberto R. Galdo argaldo at gmail.com
Tue May 29 12:15:59 EDT 2012 

Thank you very much Mauricio. You've been of great help. Now is time for us
to cook all this valuable information and go for a solution. xD

Alberto R. Galdo
argaldo at gmail.com


On Tue, May 29, 2012 at 5:59 PM, Mauricio Salatino <salaboy at gmail.com>wrote:

> Comments inline,
>
> On Tue, May 29, 2012 at 12:24 PM, Alberto R. Galdo <argaldo at gmail.com>wrote:
>
>>    Sure!. I didn't mention that we're running in a OSGi environment and
>> we're getting the session from JPAKnowledgeService like that:
>>
>>
>>
>> EntityManagerFactory emf =
>> Persistence.createEntityManagerFactory("com.package");
>>
>> Environment env = KnowledgeBaseFactory.newEnvironment();
>>
>> env.set(EnvironmentName.ENTITY_MANAGER_FACTORY, emf);
>>
>> env.set(EnvironmentName.TRANSACTION_MANAGER,(TransactionManager)
>> bundleContext.getService(this.tmServiceRef));
>>
>> env.set(EnvironmentName.TRANSACTION, (UserTransaction)
>> bundleContext.getService(this.userTransactionServiceRef));
>>
>> env.set(EnvironmentName.GLOBALS, new MapGlobalResolver());
>>
>> env.set(EnvironmentName.OBJECT_MARSHALLING_STRATEGIES, new
>> ObjectMarshallingStrategy[] {
>> MarshallerFactory.newSerializeMarshallingStrategy() });
>>
>> final StatefulKnowledgeSession ksession =
>> JPAKnowledgeService.newStatefulKnowledgeSession(kbase, sessionConfig, env);
>>
>>
>>    We are just following the manual ( and some acquired knowledge through
>> the days ... ) .... I understand that inserting an event and starting a
>> process at the same time would lead to the current situation, but Isn't
>> this the reason why we are using drools with drools fusion + jbpm to build
>> a CEP?
>>
> I'm not sure about your reasons :)
>
>>
>>    We chose Drools and a statefulsession in a fireUntilHalt loop just
>> because we would be able to have our rules and processes reacting to events
>> that enter the system at their own pace ... and found ourselves in this
>> nightmare when trying to make the system fault-tolerant through
>> persistence... :(
>>
>> FireUntilHalt and Persistence will cause "conflicts", because
> fireUntilHalts by default creates a different thread == race conditions.
>
>
>>    Do you think we should change our fault-tolerance strategy by
>> reconstructing the knowledge session, process and tasks state by our own
>> means leaving Drools and JBPM JPA persistence appart?
>>
>>  A common problem is to think that a session will do everything for you.
> I'm not saying that you need to separate your processes from your rules,
> I'm just saying that probably decoupling responsibilities will help you to
> tackle down some of the problems that you are having. If you have in memory
> processes and fusion entry points, you can keep those together and in a
> separate persistence session handle all the human task interactions and
> long running processes. If you are getting events in real time and you want
> to process them, you cannot expect to persist all the session state and
> have no performance impacts.
>
>>
>> Alberto R. Galdo
>> argaldo at gmail.com
>>
>>
>> Cheers
>
>>
>> On Tue, May 29, 2012 at 4:43 PM, Mauricio Salatino <salaboy at gmail.com>wrote:
>>
>>> Ok, so probably I'm not understanding your scenario, but are you using
>>> the JPAKnowledgeService to create persistent sessions?
>>> If you use that everything is aware of everything :) Drools and jBPM5
>>> will run using the same persistence resources. For each interaction:
>>> startProcess, insert, update, fireAllRules a transaction will be created.
>>> If you are starting a process and at the same time inserting a Fusion event
>>> you will be generating two transaction that will fight for the resources
>>> (the same resources because you have only one session).
>>> Obviously if you take the persistence mechanisms outside of the
>>> discussion everything will work perfectly fine, because you don't have any
>>> transactional resource to fight for :)
>>>
>>> Hope it helps.
>>> Cheers
>>>
>>>
>>> On Tue, May 29, 2012 at 11:36 AM, Alberto R. Galdo <argaldo at gmail.com>wrote:
>>>
>>>>    Your answer is confusing me, I think I'm not getting it right.
>>>>
>>>>    It seems that we you're saying that will have to modify JBPM or
>>>> Drools behaviour to get our architecture done ... but I don't think we are
>>>> doing nothing special than what Drools & JBPM are able to do in a
>>>> non-persisting enviroment ( at least without JBPM persisted ).
>>>>
>>>>    Maybe deeping a bit more the architecture will lead to a better
>>>> understanding of the problem:
>>>>
>>>>    We have a Drools StatefulKnowledgeSession wich is populated with
>>>> packages of rules and different BPMN 2.0 process definitions. That session
>>>> in Drools is persisted using JPA and our JTA provider. Our solution is a
>>>> CEP that uses events that get into the session using Drools Fusion. There,
>>>> several rules fire and then several consequences start processes wich
>>>> contain both Tasks and HumanTasks. Those rule activations are already in a
>>>> queue, the Agenda. Given that JBPM acts as the processmanager provider for
>>>> Drools, our processes get started in JBPM, and it is persisted using it's
>>>> own JPA manager.  Just rules, that start processes in response to events in
>>>> a fireUntilHalt loop in a persistent enviroment, both for Drools & JBPM.
>>>> Then some processes interact with the knowledge session using BussinessTask
>>>> nodes( which make other rules fire and maybe launch a different set of
>>>> processes ) and other simply get to the Stop node. I see nothing wrong
>>>> here, in fact, taking the persistence appart from the equation, everything
>>>> runs as beautifuly as expected. We've tested that.
>>>>
>>>>    I'm just trying to undersand the problem, I'm pretty 90% sure I'm
>>>> wrong, but what I think is happening here is that Drools isn't aware that
>>>> JBPM may be using the same resources and vice-versa, Isn't it?
>>>>
>>>>    Shouldn't the solution be to make Drools & JBPM aware of each other
>>>> in what is related to the syncronization of the transaction management to
>>>> avoid race conditions?
>>>>
>>>>
>>>>
>>>>
>>>> Alberto R. Galdo
>>>> argaldo at gmail.com
>>>>
>>>>
>>>> On Tue, May 29, 2012 at 3:58 PM, Mauricio Salatino <salaboy at gmail.com>wrote:
>>>>
>>>>> I'm in no way saying this:
>>>>> "  Let me see if I'm getting this right, What you mean is that there
>>>>> is no way for the JPA persistence of Drools & JBPM to coexist in the same
>>>>> knowledge session because of race conditions between them competing for
>>>>> their shared resources? I'm afraid so.. :(
>>>>>
>>>>> They can coexist. As you can see they are working together. If you
>>>>> take a look at the underlying layers you will see that each operation that
>>>>> you run against the session is wrapped in a transaction. That leads to your
>>>>> architecture, If you have a single entry point for your session everything
>>>>> will work correctly. You can achieve this by receiving all the operations
>>>>> in a queue and then execute one after the other.
>>>>> If you have multiple threads interacting with the session, you can
>>>>> implement a retrying mechanism if the transaction gets rolled back and as
>>>>> soon as the transaction win the right to commit it will work. It really
>>>>> depends on the problem that you are trying to solve.
>>>>>
>>>>> Cheers
>>>>>
>>>>> On Tue, May 29, 2012 at 10:42 AM, Alberto R. Galdo <argaldo at gmail.com>wrote:
>>>>>
>>>>>>    Thanks Mauricio for your kind response, and kick, as usual ;-)
>>>>>>
>>>>>>    Sure, we see lots of rolled back transactions ... Everywhere!
>>>>>>
>>>>>>    Let me see if I'm getting this right, What you mean is that there
>>>>>> is no way for the JPA persistence of Drools & JBPM to coexist in the same
>>>>>> knowledge session because of race conditions between them competing for
>>>>>> their shared resources? I'm afraid so.. :(
>>>>>>
>>>>>>    If it is so, Are there any plans for the integration of a shared
>>>>>> JPA persistence management of both Drools & JBPM, wich may be the solution
>>>>>> for this problem? ... Will we be better thinking to change our architecture
>>>>>> ASAP?
>>>>>>
>>>>>> Greets,
>>>>>>
>>>>>> Alberto R. Galdo
>>>>>> argaldo at gmail.com
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, May 29, 2012 at 1:54 PM, Mauricio Salatino <salaboy at gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Ok, so what is happening is that two or more threads are fighting
>>>>>>> for the resources, causing that two or more transactions wants to be
>>>>>>> completed, but just one can win. All the other must be retried. Are you
>>>>>>> seeing rolled back exceptions in your stack trace?
>>>>>>> My question to you is if you really need to handle everything in
>>>>>>> just one big persistent session. I've solved similar issues in the past
>>>>>>> using more than one session, for example one session to run business
>>>>>>> processes(persistent) and one or more for receiving and reacting to events.
>>>>>>> If you have an async way to communicate both sessions there should be no
>>>>>>> problem.
>>>>>>> Hope it helps!
>>>>>>> Cheers
>>>>>>>
>>>>>>> On Tue, May 29, 2012 at 7:30 AM, Alberto R. Galdo <argaldo at gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>>    We've been for long time now developing a complex event
>>>>>>>> processing system that ( simplified version ahead !! ) involves a set of
>>>>>>>> rules that in turn activates a set processes that fulfill human tasks and
>>>>>>>> other kind of tasks.. This all is running in a StatefulKnowledgeSession
>>>>>>>> with JPA persistence configured both for Drools and JBPM. We are running an
>>>>>>>> stack of Drools, JBPM, Drools Integration, Drools fussion, etc..
>>>>>>>>
>>>>>>>>    We've been able to persist Drools sessions & JBPM processes in
>>>>>>>> the same Persistence Context  ( not without pain :( ) using different JTA
>>>>>>>> implementations including ( but not limited to ) Bitronix & Atomikos.
>>>>>>>>
>>>>>>>>     What we are observing here is what seems to be some kind of
>>>>>>>> race condition between Drools and JBPM when running the knowledge session
>>>>>>>> when JPA&JTA persistence is configured. Very often, as soon as after 2-3
>>>>>>>> processes get created as rule's consequences are fired in response of
>>>>>>>> events inside the session we see how JBPM finds its instance nullified by
>>>>>>>> Drools when it tries to end a process and persist it.
>>>>>>>>
>>>>>>>>    We've been able to find where Drools decides to delete an
>>>>>>>> instance of the process ... at a given time Drools executes
>>>>>>>> JPAProcessInstanceManager.clearProcessInstances() [1] when it finalizes a
>>>>>>>> SingleSessionCommand wich in turn calls disconnect() for *all* the
>>>>>>>> local-stored processinstances ( wich gets populated with instances of
>>>>>>>> processes every time a process is started in the knowledge session ):
>>>>>>>>
>>>>>>>>     public void clearProcessInstances() {
>>>>>>>>
>>>>>>>>
>>>>>>>>         for (ProcessInstance processInstance: new ArrayList<ProcessInstance>(processInstances.values())) {
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>             ((ProcessInstanceImpl) processInstance).disconnect();
>>>>>>>>
>>>>>>>>         }
>>>>>>>>     }
>>>>>>>>
>>>>>>>>
>>>>>>>>     So, Drools decides to disconnect all process instances in it's
>>>>>>>> JPA context without taking in account the state the process is in, and when
>>>>>>>> an processinstance that is not stopped gets removed then JBPM finds it's
>>>>>>>> NullPointerException...
>>>>>>>>
>>>>>>>>     We've modified the code to make Drools aware of the state of
>>>>>>>> the process before wiping it from the context   ( no problem here, there
>>>>>>>> will be no leak as a running processinstance will be removed in future
>>>>>>>> calls of "clearProcessInstances" given the process is closed ). But
>>>>>>>> unfortunatelly this seems to resolve this problem, but lots of other
>>>>>>>> problems ( wich seems also race conditions arise :  for instance: Drools
>>>>>>>> closes connections to the database and JBPM finds the connection closed,
>>>>>>>>
>>>>>>>>
>>>>>>>>     So, we are really worried about using Drools & JBPM in a
>>>>>>>> persisted environment. Maybe our asumptions are wrong...  Is it possible to
>>>>>>>> have an scenario like ours given the current Drools & JBPM integration
>>>>>>>> status for a persistent statefulKnowledge Session?  Did anyone build a
>>>>>>>> complex event processing system like ours in a unaltered persistence
>>>>>>>> environment such as provided in Drools and JBPM by default?
>>>>>>>>
>>>>>>>>
>>>>>>>> Greets,
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://github.com/droolsjbpm/jbpm/blob/master/jbpm-persistence-jpa/src/main/java/org/jbpm/persistence/processinstance/JPAProcessInstanceManager.java
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Alberto R. Galdo
>>>>>>>> argaldo at gmail.co <argaldo at gmail.com>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> rules-users mailing list
>>>>>>>> rules-users at lists.jboss.org
>>>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>  - MyJourney @ http://salaboy.wordpress.com
>>>>>>>  - Co-Founder @ http://www.jugargentina.org
>>>>>>>  - Co-Founder @ http://www.jbug.com.ar
>>>>>>>
>>>>>>>  - Salatino "Salaboy" Mauricio -
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> rules-users mailing list
>>>>>>> rules-users at lists.jboss.org
>>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> rules-users mailing list
>>>>>> rules-users at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>  - MyJourney @ http://salaboy.wordpress.com
>>>>>  - Co-Founder @ http://www.jugargentina.org
>>>>>  - Co-Founder @ http://www.jbug.com.ar
>>>>>
>>>>>  - Salatino "Salaboy" Mauricio -
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> rules-users mailing list
>>>>> rules-users at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> rules-users mailing list
>>>> rules-users at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>
>>>>
>>>
>>>
>>> --
>>>  - MyJourney @ http://salaboy.wordpress.com
>>>  - Co-Founder @ http://www.jugargentina.org
>>>  - Co-Founder @ http://www.jbug.com.ar
>>>
>>>  - Salatino "Salaboy" Mauricio -
>>>
>>>
>>> _______________________________________________
>>> rules-users mailing list
>>> rules-users at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>
>>>
>>
>> _______________________________________________
>> rules-users mailing list
>> rules-users at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/rules-users
>>
>>
>
>
> --
>  - MyJourney @ http://salaboy.wordpress.com
>  - Co-Founder @ http://www.jugargentina.org
>  - Co-Founder @ http://www.jbug.com.ar
>
>  - Salatino "Salaboy" Mauricio -
>
>
> _______________________________________________
> rules-users mailing list
> rules-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/rules-users/attachments/20120529/5f58e70d/attachment-0001.html

More information about the rules-users mailing list