[ http://jira.jboss.com/jira/browse/JBPM-983?page=all ] Alexander Schlett updated JBPM-983: ----------------------------------- Attachment: SimpleTest.java junit test repeating a simple process 10 times -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JBPM-983?page=all ] Alexander Schlett updated JBPM-983: ----------------------------------- Attachment: SimpleTest.java Results are as follows: jBPM 3.2.0 with 1 JobExecutor thread: works jBPM 3.2.0 with 5 JobExecutor threads: fails jBPM CVS 3.2.1 as of today 2007/06/12 with 1 JobExecutor thread: FAILS! jBPM CVS 3.2.1 as of today 2007/06/12 with 5 JobExecutor threads: fails -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JBPM-983?page=all ] Alexander Schlett updated JBPM-983: ----------------------------------- Attachment: SimpleTest.java here's another version, it's using an ActionHandler and no scripts or events or anything else, but sub proccesses though. behavior is exactly the same. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JBPM-983?page=comments#action_12366449 ] Ronald van Kuijk commented on JBPM-983: --------------------------------------- What I did: - use CVS version as of today 2007-06-22 - 2 jobexecurtor threads - Take the non-subproces version - removed the async=true op de join en fork - changed async=true to async=exclusive in node 1 and node 2 Then it works... -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JBPM-983?page=comments#action_12366597 ] philipp commented on JBPM-983: ------------------------------ Hi all, I have the same problem in my project. I made some tests with this unittest and found out that the job object is manipulated from different threads and therefore hibernate throws an exception. Then I changed the implementation, so that only one thread is selecting the jobs and different threads (worker thread) are executing the tokens. Then the next problem occurred. The single thread loads the jobs that are still executed by the worker thread. See hibernate.queries.hbm.xml: select job from org.jbpm.job.Job as job where ( (job.lockOwner is null) or (job.lockOwner = :lockOwner) ) The problem is that the worker thread also loads the job object. In my test I removed job.lockOwner = :lockOwner. After that I got the third problem. In the unittest to worker threads reach the join node in the same time. In that case the last worker thread gets a hibernate error. This problem I solved by setting the join to async="true" and synchronising the execution of the tokens by process and node: syncKey=("synckey"+job.getProcessInstance().getId()+":"+job.getToken().getNode().getId()).intern(); synchronized (syncKey) { executeJob(job); } After all that the test seems to work parallel with many worker threads. In the documentation I can read: 1.7. The JBoss jBPM Job Executor In an enterprise environment, JMS and the EJB TimerSerive can be used for that purpose. Does this mean that there is a JMS implementation? -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JBPM-983?page=comments#action_12366641 ] Edward Staub commented on JBPM-983: ----------------------------------- I'm still working on it, but I believe the StaleObjectStateException happens because the job started on the child process, and the Join on the parent process is happening on the same job without respecting exclusivity. More later... -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JBPM-983?page=comments#action_12367164 ] Edward Staub commented on JBPM-983: ----------------------------------- I suppose I should describe the fixes I'm not publishing. I made a static map reflecting all processes that are currently locked for exclusive execution. The map contains process id as key, and Thread, lock count, and process id in a class as the value. The lock count is needed to allow a thread to lock/unlock a process more than once in a nested fashion. All use of the map is synchronized on the map. Whenever a process instance is in use by a job, it is locked by adding it to the map. After all jobs for a process are completed and committed in JobExecutorThread, all process locks held by that thread are released (deleted from the map), by scanning the map for the thread. The map is synchronize-locked during querying for a job. When job(s) have been selected, their process-instance is locked and then the synchronization-lock is released. This prevents the problem where, if a new job is spawned while old ones are running on a process instance, the new job may be picked up by another thread immediately. When a subprocess is created in ProcessState, it is locked until the job is completed. This prevents execution of jobs on the new process before it has been committed. When a subprocess completes in ProcessState, there is an attempt to lock the parent process. If that succeeds, execution proceeds normally. If the lock is already held by someone else, a job is spawned to wake up the parent process. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JBPM-983?page=comments#action_12367428 ] Jan-Philipp Bolle commented on JBPM-983: ---------------------------------------- Thank you, Edward, for your support. In our project we solved the problem for our needs. It's not a good solution but it seems to work. -First we created a single thread how to select the jobs and to distribute the work to free threads. -Then we removed the restart functionality. -We always synchronise the jobexecution with nodeID and prozessID. This only works if you use async in a join node and doesn't use subprocesses. -To solve the problems with subprocesses we add a dummynode before finishing and set this to exclusive. In the Jobexecution we synchronise all threads which are marked as exclusive. -Then the last problem was by starting a new thread directly after the next with the same process we got an exception. We solved this problem by waiting a second after the execution. It's really dirty but it works. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JBPM-983?page=all ] Ronald van Kuijk updated JBPM-983: ---------------------------------- Priority: Critical (was: Major) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JBPM-983?page=comments#action_12367534 ] Edward Staub commented on JBPM-983: ----------------------------------- Tom, I tested with default database and isolation, in order to avoid any confusion about db-related issues. As I explained in JBPM-995, Hibernate logs this Exception itself at Error level. AFAIK, there is no way in JBPM to keep it out of the log. To my mind, emitting spurious error messages into the log is a bug - it means that anyone who doesn't know which errors to ignore is going to waste a lot of time chasing phantoms. In any event, that particular problem has an easy fix, as described in the report. But that's all irrelevant. The problems in this JIRA report relevant to StaleObjectStateException are not at the job level and are not handled in any way. All of the stale objects I've seen have been tokens, due to the problems described below (and earlier in this thread). JBPM defaults: HQSLDB, isolation level 0. FWIW, we'll be using Oracle in production. The three problems I've nailed down are: -- if a job is saved while other jobs are in process for the same Process Instance, the saved job may run immediately, ignoring the "exclusive" flag -- when a ProcessState is completed on a job that started in the subprocess, a collision can occur with another thread concurrently running in the parent process, resulting in a StaleObjectStateException -- when a ProcessState launches a subprocess, and the subprocess immediately reaches a node with "async=exclusive", the node job may be picked up and started before the ProcessInstance object has been committed to the database. This has been noted before on the forum, but I can't find a JIRA item regarding it. The following may also be a problem, but I found it before the others, and I think it may be a second-order effect: -- two tokens can collide in the parent token at a join, resulting in a Hibernate StaleObjectStateException. I'm not sure how this would happen if the two tokens are "exclusive", so it may only occur because of the following two problems... AFAIK, none of these are prevented in any way by the current JobExecutorThread design, either as you've described or as I've read. With one exception, the concurrency issues are due to collisions at the ProcessInstance level or Token level. The one exception is when a job is both queued and started while another job for the same ProcessInstance is already running. All of these were reproduced using Alex's test case. As I noted above, in order to ensure reproducibility, it's important to widen the critical sections, in my case by turning up logging and logging to a console. I found that sometimes I'd be trying to chase one of these bugs and suddenly one of the other ones would start happening first, preventing me from following the first one. I learned to be flexible; I just tackled whichever one showed its ugly head first. There are still more bugs in here, but I don't have any more time right now to find them. Let me know if more info is useful. See my first 28-June comment above for some design thoughts, and the 27-June note for issues I can see with the designs. To fix these problems, we need a way to synchronize across threads and servers on ProcessInstances and Tokens . You may want to factor out a "Concurrency Service" or similar to deal with each of these three configurations in the most efficient manner: - single-threaded - multi-threaded (jobs/JMS/messaging; in-memory synchronization) - clustered (jobs/JMS/messaging; in-database synchronization) In the clustered case, in most cases it would be desirable to implement server affinity for ProcessInstances - that way all token synchronization could be in-memory, because all tokens are known to be in the same server. Of course, if the server goes away, it should give up all its ProcessInstances automatically. Database row-locking might be a way to implement this. -Ed Staub -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JBPM-983?page=comments#action_12367830 ] Jan-Philipp Bolle commented on JBPM-983: ---------------------------------------- In our project we run the unittest with the databases hsql and mysql and the operation systems windows XP and Suse linux. In all combinations we got an error. By the way we used solation level 4. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JBPM-983?page=comments#action_12372722 ] Ronald van Kuijk commented on JBPM-983: --------------------------------------- the issue in JBPM-1042 is one of the errors I encountered during testing the above cases. So look there for some more info -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JBPM-983?page=comments#action_12375308 ] Tom Baeyens commented on JBPM-983: ---------------------------------- Here are my initial comments on reading Ed's comments. I'll now try to run the attached test case and see what that learns me. " The three problems I've nailed down are: -- if a job is saved while other jobs are in process for the same Process Instance, the saved job may run immediately, ignoring the "exclusive" flag " [tom]: that should be added in the documentation as a limitation. see my comment on your proposed fix later in this post " -- when a ProcessState is completed on a job that started in the subprocess, a collision can occur with another thread concurrently running in the parent process, resulting in a StaleObjectStateException " [tom]: i don't really get this problem. and then i don't see why the staleobjectexception is a problem. that is to be expected when 2 independent threads operation on the same process instance runtime information item, no ? " -- when a ProcessState launches a subprocess, and the subprocess immediately reaches a node with "async=exclusive", the node job may be picked up and started before the ProcessInstance object has been committed to the database. This has been noted before on the forum, but I can't find a JIRA item regarding it. " [tom]: isn't this due to your isolation level 0 ? " To fix these problems, we need a way to synchronize across threads and servers on ProcessInstances and Tokens . You may want to factor out a "Concurrency Service" ... " [tom]: that is exactly what we don't want to do. we should only offer what we can do by synchronizing on the DB. that should give us enough features to work with. the downside is that some features may depend on isolation levels. that should be documented better imo. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JBPM-983?page=comments#action_12375360 ] Edward Staub commented on JBPM-983: ----------------------------------- [ed]-- if a job is saved while other jobs are in process for the same Process Instance, the saved job may run immediately, ignoring the "exclusive" flag [tom]: that should be added in the documentation as a limitation. see my comment on your proposed fix later in this post [ed]--I believe this would mean that in general, it's invalid to use async/exclusive nodes anywhere in a child (forked) token if an async/exclusive node has previously been started somewhere. That's a pretty broad limitation - is it what you meant? Note that the current behavior results in a timing-dependent, hard-to-reproduce bug. --------- [ed] when a ProcessState is completed on a job that started in the subprocess, a collision can occur with another thread concurrently running in the parent process, resulting in a StaleObjectStateException [tom]: i don't really get this problem. and then i don't see why the staleobjectexception is a problem. that is to be expected when 2 independent threads operation on the same process instance runtime information item, no ? [ed] The limitation in this case is that it is invalid to use a subprocess on a fork that may complete while active execution is happening on another branch of the fork. Is that what you mean to say? How, in practice, are you thinking that people would be able to use subprocesses on forks with this limitation? ---------- [ed] when a ProcessState launches a subprocess, and the subprocess immediately reaches a node with "async=exclusive", the node job may be picked up and started before the ProcessInstance object has been committed to the database. This has been noted before on the forum, but I can't find a JIRA item regarding it. [tom]: isn't this due to your isolation level 0 ? [ed] Yes. Are you saying that this should be a documented limitation also? --------- [ed] To fix these problems, we need a way to synchronize across threads and servers on ProcessInstances and Tokens . You may want to factor out a "Concurrency Service" ... [tom]: that is exactly what we don't want to do. we should only offer what we can do by synchronizing on the DB. that should give us enough features to work with. the downside is that some features may depend on isolation levels. that should be documented better imo. [ed] I may have been unclear. I suspect that you thought I meant some scheme based on network communications (beyond that used by the DB and transaction manager). I didn't. I do, however, think that you should consider use of Java synchronization mechanisms when a single JBPM instance is in control of a database. I was trying to suggest that there might be a concurrency-service interface that is populated by a different class based on whether multiple JBPM instances (multiple servers or EARs) are being used, and on what isolation level is in use. When a single JBPM is in use, simple Java synchronization can be used to reliably support concurrent operations on even the wimpiest of databases. When multiple servers are used, database locks and transaction support can be used. With this kind of factoring, It would then be much easier to throw an exception or otherwise flag scenarios that are not supported in the given configuration, such as those discussed above. . I'm not at all suggesting that optimistic locking should be abandoned. But I do think that there seem to be a few spots that need a bit more to avoid difficult-to-resolve concurrency bugs. . On a tangent... it might be possible to provide some useful (if imperfect) static execution analysis if nodes that are implicitly asynchronous (task-like nodes that "wait" on an external event for completion) were marked as such. This might be implemented as a "isAsync()" method (returns yes/no/sometimes) that extends ActionHandler, or through some more general mechanism for ActionHandlers to contain metadata. I mention this mostly as something to think about for Tempranillo's "Action" interface. I mention this because some of the limitations we've talked about require analysis to detect, and are difficult to express precisely in human language. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JBPM-983?page=comments#action_12375372 ] Tom Baeyens commented on JBPM-983: ---------------------------------- [ed] when a ProcessState is completed on a job that started in the subprocess, a collision can occur with another thread concurrently running in the parent process, resulting in a StaleObjectStateException [tom]: i don't really get this problem. and then i don't see why the staleobjectexception is a problem. that is to be expected when 2 independent threads operation on the same process instance runtime information item, no ? [ed] The limitation in this case is that it is invalid to use a subprocess on a fork that may complete while active execution is happening on another branch of the fork. Is that what you mean to say? How, in practice, are you thinking that people would be able to use subprocesses on forks with this limitation? [tom] i still didn't get the problem. i'm looking for the scenario where things go wrong. maybe it is easiest if you can help me get the test going in the code base: jpdl/jar/src/main/java/org/jbpm/job/executor/ConcurrencyTest.java then it might be easier to help me reproduce and see the problem -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JBPM-983?page=comments#action_12375409 ] Edward Staub commented on JBPM-983: ----------------------------------- Re "when a ProcessState is completed on a job that started in the subprocess, a collision can occur with another thread concurrently running in the parent process, resulting in a StaleObjectStateException". Here's more info, let me know if it's not enough. In ProcessInstance.end(), there's a bit of code that signals the ProcessInstance: // check if this process was started as a subprocess of a super process if (superProcessToken!=null) { addCascadeProcessInstance(superProcessToken.getProcessInstance()); ExecutionContext superExecutionContext = new ExecutionContext(superProcessToken); superExecutionContext.setSubProcessInstance(this); superProcessToken.signal(superExecutionContext); } Remember, in the failure scenario the subprocess has passed an async node, so it's running on a "new" thread. If it completes very quickly, the calling token may not yet be saved, resulting in a SOSException on the token. I don't know of a way, using just test code, to guarantee reproduction of this problem. A delay must be guaranteed between posting the job (in GraphElement.executeActions() and saving the Hibernate session. You might want to temporarily insert a delay into GraphElement.executeActions(), after the messageService.send(job) call. For me, just turning on enough logging to the console is sufficient. This is another case that a higher transaction isolation MAY take care of. But I'm pretty doubtful; it seems like code in ProcessInstance.end() must block somehow until the parent token is done executing. If I understand it correctly, job-posting is not performed as part of the overall JBPM transaction, because it's on a separate Hibernate session. Correct? In my workaround, it submits a job to do the signal if the conditions warrant it, and job token-exclusion mechanisms (made more robust to eliminate SOSExceptions) take care of the rest. I don't like this much. Maybe a LOCK_UPGRADE on the Token could be used instead, before the messageService.send() in GraphElement.executeActions()? I don't know what Hypersonic supports. We know the thread is about to save the session in this context, so the lock would only be held for a short time. This may seem like a bizarre scenario - who would call a subprocess that immediately returns? Likely cases include those where the subprocess immediately bails out because of a business-logic assertion failure of some kind, or where the subprocess was factored out of the main process just to make it easy to independently modify. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JBPM-983?page=comments#action_12375766 ] Edward Staub commented on JBPM-983: ----------------------------------- Good! I was hoping I might be wrong about that. For us - we'll be using Oracle, so I'm fine - I couldn't use level 0 in production even if I wanted to! I'm not nearly so optimistic about SOSExceptions with level 0, but real stakeholders should speak about that - not me. So it seems like the bottom line on all these issues is that as a general rule of thumb, if anyone's using two or three of these three features: - forks, - subprocesses, - asynchronous nodes (using async attribute, or tasks, or anything else that completes asynchronously) ... then they probably need to use READ_COMMITTED or better isolation to avoid race conditions bugs. Does that sound about right? [I'm just trying to help with the documentation wordsmithing.] It's not totally correct, but it is easy to read, understand, and "test" a process definition against. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira