[jboss-jira] [JBoss JIRA] (JBRULES-2794) Running More Than 100 Threads Results in ArrayIndexOutOfBoundsException from Drools AbstractHashTable
Tim Griffith (JIRA)
jira-events at lists.jboss.org
Thu Sep 27 16:58:03 EDT 2012
[ https://issues.jboss.org/browse/JBRULES-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722422#comment-12722422 ]
Tim Griffith edited comment on JBRULES-2794 at 9/27/12 4:56 PM:
----------------------------------------------------------------
Hi Mark,
we are seeing the same as the above with jBPM 5.2 [JRules R5.3.1] execution logic for us is as follows:
1. build a KnowledgeBase (done at application start)
2. within a threadpool (Spring JMS container with 5 threads):
a. on request; build a statefulSession from the KnowledgeBase
b. register the workItemHandlers and processEventListeners
c. execute ksession.startProcess
we are getting *very* intermittent failures with the ArrayIndexOutOfBounds in the AbstractHashTable as commented in this bug. We are working to try and build a deterministic, localized testcase which we can upload... however this is not likely to be within the week.
We were surprised that we were hitting this issue because we were under the impression that, given the session being localized to the thread, it would not be possible for cross-thread contamination.
>From static analysis of the code - it appears that the issue, as per your comment with the straight JRules case, is caused 'at a higher level' - following through the stack the issue appears to be:
a) StatefulSessions by default 'keepReference' with the AbstractRuleBase [ see ReteooRuleBase.newStatefulSession(...):350 and :407 (in reference to drools-core-5.3.1.Final)
b) PatternBuilder.attachObjectTypeNode(...) gets the full set of sessions back (:244) and iterates over them under ObjectTypeNode.attach(...):365
c) the EntryPointNode.updateSink(...): method appears to take no further locking - thus it is possible for all Threads to interferre with each other doing this process.
We are attempting to avoid the issue by providing the SessionConfiguration to the KnowledgeBase.newStatefulKnowledgeSession(...) explicitly, having set the keepReference to *false*
Our exception trace is as follows:
java.lang.RuntimeException: A Generic Exception Message
at anon.JBpmProcessExecutor.executeProcess(JBpmProcessExecutor.java:298)
at anon.XmlMessageProcessor.processMessage(XmlMessageProcessor.java:35)
at anon.JmsTextMessageListener.onMessage(JmsTextMessageListener.java:39)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:562)
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:500)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:468)
at org.springframework.jms.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:440)
at org.springframework.jms.listener.SimpleMessageListenerContainer.processMessage(SimpleMessageListenerContainer.java:340)
at org.springframework.jms.listener.SimpleMessageListenerContainer$1$1.run(SimpleMessageListenerContainer.java:307)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.drools.RuntimeDroolsException: Unexpected exception executing action org.drools.reteoo.ReteooWorkingMemory$WorkingMemoryReteAssertAction at 1e125710
at org.drools.common.AbstractWorkingMemory.executeQueuedActions(AbstractWorkingMemory.java:978)
at org.drools.impl.StatefulKnowledgeSessionImpl.executeQueuedActions(StatefulKnowledgeSessionImpl.java:854)
at org.jbpm.process.instance.ProcessRuntimeImpl.createProcessInstance(ProcessRuntimeImpl.java:148)
at org.jbpm.process.instance.ProcessRuntimeImpl.startProcess(ProcessRuntimeImpl.java:135)
at org.drools.common.AbstractWorkingMemory.startProcess(AbstractWorkingMemory.java:1079)
at org.drools.impl.StatefulKnowledgeSessionImpl.startProcess(StatefulKnowledgeSessionImpl.java:306)
at anon.JBpmProcessExecutor.executeProcess(JBpmProcessExecutor.java:252)
... 11 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 16
at org.drools.core.util.AbstractHashTable$HashTableIterator.next(AbstractHashTable.java:316)
at org.drools.reteoo.EntryPointNode.updateSink(EntryPointNode.java:422)
at org.drools.reteoo.ObjectTypeNode.attach(ObjectTypeNode.java:373)
at org.drools.reteoo.builder.PatternBuilder.attachObjectTypeNode(PatternBuilder.java:268)
at org.drools.reteoo.ClassObjectTypeConf.<init>(ClassObjectTypeConf.java:99)
at org.drools.common.ObjectTypeConfigurationRegistry.getObjectTypeConf(ObjectTypeConfigurationRegistry.java:71)
at org.drools.reteoo.Rete.assertObject(Rete.java:105)
at org.drools.reteoo.ReteooRuleBase.assertObject(ReteooRuleBase.java:284)
at org.drools.reteoo.ReteooWorkingMemory$WorkingMemoryReteAssertAction.execute(ReteooWorkingMemory.java:404)
at org.drools.common.AbstractWorkingMemory.executeQueuedActions(AbstractWorkingMemory.java:976)
... 17 more
was (Author: tim.griffith):
Hi Mark,
we are seeing the same as the above with jBPM 5.2 [JRules R5.3.1] execution logic for us is as follows:
1. build a KnowledgeBase (done at application start)
2. within a threadpool (Spring JMS container with 5 threads):
a. on request; build a statefulSession from the KnowledgeBase
b. register the workItemHandlers and processEventListeners
c. execute ksession.startProcess
we are getting *very* intermittent failures with the ArrayIndexOutOfBounds in the AbstractHashTable as commented in this bug. We are working to try and build a deterministic, localized testcase which we can upload... however this is not likely to be within the week.
We were surprised that we were hitting this issue because we were under the impression that, given the session being localized to the thread, it would not be possible for cross-thread contamination.
>From static analysis of the code - it appears that the issue, as per your comment with the straight JRules case, is caused 'at a higher level' - following through the stack the issue appears to be:
a) StatefulSessions by default 'keepReference' with the AbstractRuleBase [ see ReteooRuleBase.newStatefulSession(...):350 and :407 (in reference to drools-core-5.3.1.Final)
b) PatternBuilder.attachObjectTypeNode(...) gets the full set of sessions back (:244) and iterates over them under ObjectTypeNode.attach(...):365
c) the EntryPointNode.updateSink(...): method appears to take no further locking - thus it is possible for all Threads to interferre with each other doing this process.
We are attempting to avoid the issue by providing the SessionConfiguration to the KnowledgeBase.newStatefulKnowledgeSession(...) explicitly, having set the keepReference to *false*
Our exception trace is as follows:
java.lang.RuntimeException: A Generic Exception Message
at anon.JBpmProcessExecutor.executeProcess(JBpmProcessExecutor.java:298)
at anon.XmlMessageProcessor.processMessage(XmlMessageProcessor.java:35)
at anon.JmsTextMessageListener.onMessage(JmsTextMessageListener.java:39)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:562)
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:500)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:468)
at org.springframework.jms.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:440)
at org.springframework.jms.listener.SimpleMessageListenerContainer.processMessage(SimpleMessageListenerContainer.java:340)
at org.springframework.jms.listener.SimpleMessageListenerContainer$1$1.run(SimpleMessageListenerContainer.java:307)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.drools.RuntimeDroolsException: Unexpected exception executing action org.drools.reteoo.ReteooWorkingMemory$WorkingMemoryReteAssertAction at 1e125710
at org.drools.common.AbstractWorkingMemory.executeQueuedActions(AbstractWorkingMemory.java:978)
at org.drools.impl.StatefulKnowledgeSessionImpl.executeQueuedActions(StatefulKnowledgeSessionImpl.java:854)
at org.jbpm.process.instance.ProcessRuntimeImpl.createProcessInstance(ProcessRuntimeImpl.java:148)
at org.jbpm.process.instance.ProcessRuntimeImpl.startProcess(ProcessRuntimeImpl.java:135)
at org.drools.common.AbstractWorkingMemory.startProcess(AbstractWorkingMemory.java:1079)
at org.drools.impl.StatefulKnowledgeSessionImpl.startProcess(StatefulKnowledgeSessionImpl.java:306)
at com.barcap.esit.motif.bpm.process.JBpmProcessExecutor.executeProcess(JBpmProcessExecutor.java:252)
... 11 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 16
at org.drools.core.util.AbstractHashTable$HashTableIterator.next(AbstractHashTable.java:316)
at org.drools.reteoo.EntryPointNode.updateSink(EntryPointNode.java:422)
at org.drools.reteoo.ObjectTypeNode.attach(ObjectTypeNode.java:373)
at org.drools.reteoo.builder.PatternBuilder.attachObjectTypeNode(PatternBuilder.java:268)
at org.drools.reteoo.ClassObjectTypeConf.<init>(ClassObjectTypeConf.java:99)
at org.drools.common.ObjectTypeConfigurationRegistry.getObjectTypeConf(ObjectTypeConfigurationRegistry.java:71)
at org.drools.reteoo.Rete.assertObject(Rete.java:105)
at org.drools.reteoo.ReteooRuleBase.assertObject(ReteooRuleBase.java:284)
at org.drools.reteoo.ReteooWorkingMemory$WorkingMemoryReteAssertAction.execute(ReteooWorkingMemory.java:404)
at org.drools.common.AbstractWorkingMemory.executeQueuedActions(AbstractWorkingMemory.java:976)
... 17 more
> Running More Than 100 Threads Results in ArrayIndexOutOfBoundsException from Drools AbstractHashTable
> -----------------------------------------------------------------------------------------------------
>
> Key: JBRULES-2794
> URL: https://issues.jboss.org/browse/JBRULES-2794
> Project: Drools
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Affects Versions: 5.1.1.FINAL, FUTURE
> Environment: Ubuntu, Intellij / Windows XP, Eclipse
> Reporter: Anatoly Polinsky
> Assignee: Mark Proctor
> Labels: drools-core, drools-flow
> Attachments: AbstractHashTable-thread_localing_a_table_row_.patch
>
> Original Estimate: 1 day
> Remaining Estimate: 1 day
>
> Effected: Drools Flow and Drools Core
> --------------------------------------------------------
> Unfortunately it is not a consistent behaviour, but happens quite often.
> java.util.concurrent.ExecutionException: org.drools.RuntimeDroolsException: Unexpected exception executing action org.drools.reteoo.ReteooWorkingMemory$WorkingMemoryReteAssertAction at 1102fab
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at org.custom.workflow.stress.test.WorkflowStressTest.shouldRunFlow(WorkflowStressTest.java:50)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:74)
> at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:82)
> at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:72)
> at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:240)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:180)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
> at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:94)
> at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:192)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:64)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:115)
> Caused by: org.drools.RuntimeDroolsException: Unexpected exception executing action org.drools.reteoo.ReteooWorkingMemory$WorkingMemoryReteAssertAction at 1102fab
> at org.drools.common.AbstractWorkingMemory.executeQueuedActions(AbstractWorkingMemory.java:1473)
> at org.drools.common.AbstractWorkingMemory.startProcess(AbstractWorkingMemory.java:1631)
> at org.drools.impl.StatefulKnowledgeSessionImpl.startProcess(StatefulKnowledgeSessionImpl.java:318)
> at org.drools.command.runtime.process.StartProcessCommand.execute(StartProcessCommand.java:99)
> at org.drools.command.runtime.process.StartProcessCommand.execute(StartProcessCommand.java:38)
> at org.drools.persistence.session.SingleSessionCommandService.execute(SingleSessionCommandService.java:285)
> at org.drools.command.impl.CommandBasedStatefulKnowledgeSession.startProcess(CommandBasedStatefulKnowledgeSession.java:193)
> at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)
> at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:692)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:625)
> at org.custom.workflow.stress.test.WorkflowRunner.call(WorkflowRunner.java:18)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
> at org.drools.core.util.AbstractHashTable$HashTableIterator.next(AbstractHashTable.java:317)
> at org.drools.reteoo.EntryPointNode.updateSink(EntryPointNode.java:323)
> at org.drools.reteoo.ObjectTypeNode.attach(ObjectTypeNode.java:303)
> at org.drools.reteoo.builder.PatternBuilder.attachObjectTypeNode(PatternBuilder.java:257)
> at org.drools.reteoo.ClassObjectTypeConf.<init>(ClassObjectTypeConf.java:92)
> at org.drools.common.ObjectTypeConfigurationRegistry.getObjectTypeConf(ObjectTypeConfigurationRegistry.java:68)
> at org.drools.reteoo.Rete.assertObject(Rete.java:111)
> at org.drools.reteoo.ReteooRuleBase.assertObject(ReteooRuleBase.java:280)
> at org.drools.reteoo.ReteooWorkingMemory$WorkingMemoryReteAssertAction.execute(ReteooWorkingMemory.java:360)
> at org.drools.common.AbstractWorkingMemory.executeQueuedActions(AbstractWorkingMemory.java:1471)
> ... 24 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
More information about the jboss-jira
mailing list