[jboss-jira] [JBoss JIRA] Commented: (JBPM-983) concurrent process execution fails

[Edward Staub (JIRA)]

    [ http://jira.jboss.com/jira/browse/JBPM-983?page=comments#action_12367072 ] 
            
Edward Staub commented on JBPM-983:
-----------------------------------

I've found several problems, but still can't keep make this test case run for more than a couple of minutes without getting a hibernate stale-object exception.  So there's more to do.

I have identified these concurrency-related failure scenarios, all discovered with the test case provided.
Reproduction of these kinds of problems is often dependent on broadening time windows a little where concurrency problems may occur.  I have been running with debug logging turned on for a few relevant classes, with the log output going to the (Intellij) console.  The I/O delays from log output may be essential to easy reproduction of these problems.

Problems observed, using logs and breakpoints : 

 - two tokens can collide in the parent token at a join, resulting in a Hibernate StaleObjectStateException.  I'm not sure how this would happen if the two tokens are "exclusive", so it may only occur because of the following two problems...

 - if a job is saved while other jobs are in process for the same Process Instance, the saved job may run immediately, ignoring the "exclusive" flag

 - when a ProcessState is completed on a job that started in the subprocess, a collision can occur with another thread concurrently running in the parent process, resulting in a StaleObjectStateException

- when a ProcessState launches a subprocess, and the subprocess immediately reaches a node with "async=exclusive", the node job may be picked up and started before the ProcessInstance object has been committed to the database.  This has been noted before on the forum, but I can't find a JIRA item regarding it.

I'm not including the fixes I've developed, although if someone really wants them, just ask!
I'm not especially happy with them, for many reasons:
 - I haven't regression-tested them, and won't have time for a while.
 - I suspect they don't behave well if there's no messaging service.
 - They are too conservative, in some cases.  I believe that a separate concurrency support object should be factored out, and implemented differently depending on whether clustered, J2EE, single-threaded, etc.
 - They are tactical, where a significant design change toward "less optimism" might be more clearly correct and more testable
 - They are not clusterable
 - They lock at the process level, when sometimes only a token-level lock is required.  I'm not referring to any current token locking mechanism that I know of - I mean one similar to the JobExecutorThread ProcessInstance mechanism that handles exclusivity.
 - I don't think the current "async=exclusive" feature is a good fit with what people need or how they think about synchronization.  I don't think it should be necessary to obtain correct JBPM behavior, either.  So a general review of design in this area is desirable, I think.
 - As I see it, the current JMS messaging implementation has even less concurrency support than that in the JobExecutorThread class used in this test.  I suspect that use of jobs will be necessary to obtain good JMS behavior, also.  This test case should be retested using JMS/J2EE.  I suspect that this will lead to some additional concurrency-related design changes.


> concurrent process execution fails
> ----------------------------------
>
>                 Key: JBPM-983
>                 URL: http://jira.jboss.com/jira/browse/JBPM-983
>             Project: JBoss jBPM
>          Issue Type: Bug
>          Components: Core Engine
>    Affects Versions: jBPM jPDL 3.2
>         Environment: Hypersonic in-memory database, JobExecutor configured with 5 threads
>            Reporter: Alexander Schlett
>         Assigned To: Tom Baeyens
>         Attachments: SimpleTest.java, SimpleTest.java, SimpleTest.java, SimpleTest.java, SimpleTest.java
>
>
> concurrent execution of async nodes with multiple JobExecutor threads fails. the effect is:
> 1) job sync within JobExecutor fails due to org.hibernate.StaleObjectStateException
> 2) process gets stuck in join node and never ends
> junit test for this is attached, it's a simple process with just a fork and a join and some scripts inbetween.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira