[jboss-jira] [JBoss JIRA] (DROOLS-2174) Deadlock between AbstractWorkingMemory and DefaultAgenda

Umang Sharma (JIRA) issues at jboss.org
Tue Dec 12 08:02:02 EST 2017 

     [ https://issues.jboss.org/browse/DROOLS-2174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Umang Sharma updated DROOLS-2174:
---------------------------------
    Description: 
 I am getting a similar deadlock as the one in the cloned issue. We are using following dependencies:
<dependency>
<groupId>org.drools</groupId>
<artifactId>drools-templates</artifactId>
<version>$
{runtime.version}</version>
</dependency>
<dependency>
<groupId>org.kie</groupId>
<artifactId>kie-api</artifactId>
<version>${runtime.version}
</version>
</dependency>
<dependency>
<groupId>org.drools</groupId>
<artifactId>drools-core</artifactId>
<version>$
{runtime.version}</version>
</dependency>
<dependency>
<groupId>org.drools</groupId>
<artifactId>drools-compiler</artifactId>
<version>${runtime.version}
</version>
</dependency>
runtime.version = 6.5.0-Final
I have around 50 off threads stuck in a deadlock. 49 of them are having following stack trace:
"THREAD-4" #124 prio=5 os_prio=0 tid=0x00000000015fc800 nid=0x3a5d waiting on condition [0x00007fc773e7b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
parking to wait for <0x000000074d189818> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at org.drools.core.impl.KnowledgeBaseImpl.readLock(KnowledgeBaseImpl.java:703)
at org.drools.core.impl.KnowledgeBaseImpl.newStatefulSession(KnowledgeBaseImpl.java:1434)
at org.drools.core.impl.KnowledgeBaseImpl.newStatefulSession(KnowledgeBaseImpl.java:1422)
at org.drools.core.impl.KnowledgeBaseImpl.newStatefulKnowledgeSession(KnowledgeBaseImpl.java:308)
at org.drools.core.impl.KnowledgeBaseImpl.newKieSession(KnowledgeBaseImpl.java:385)
at org.drools.compiler.kie.builder.impl.KieContainerImpl.newKieSession(KieContainerImpl.java:693)
at org.drools.compiler.kie.builder.impl.KieContainerImpl.newKieSession(KieContainerImpl.java:629)
at org.drools.compiler.kie.builder.impl.KieContainerImpl.newKieSession(KieContainerImpl.java:611)
And one of them is stuck in the following stack trace:
"THREAD-4" #124 prio=5 os_prio=0 tid=0x00000000015fc800 nid=0x3a5d waiting on condition [0x00007fc773e7b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
parking to wait for <0x000000074d189818> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at org.drools.core.impl.KnowledgeBaseImpl.readLock(KnowledgeBaseImpl.java:703)
at org.drools.core.impl.KnowledgeBaseImpl.newStatefulSession(KnowledgeBaseImpl.java:1434)
at org.drools.core.impl.KnowledgeBaseImpl.newStatefulSession(KnowledgeBaseImpl.java:1422)
at org.drools.core.impl.KnowledgeBaseImpl.newStatefulKnowledgeSession(KnowledgeBaseImpl.java:308)
at org.drools.core.impl.KnowledgeBaseImpl.newKieSession(KnowledgeBaseImpl.java:385)
at org.drools.compiler.kie.builder.impl.KieContainerImpl.newKieSession(KieContainerImpl.java:693)
at org.drools.compiler.kie.builder.impl.KieContainerImpl.newKieSession(KieContainerImpl.java:629)
at org.drools.compiler.kie.builder.impl.KieContainerImpl.newKieSession(KieContainerImpl.java:611)
Please help us out in fixing this. Its a production system serving millions of users.
PS: This is was reproduced under a medium load where we create 50 stateful sessions in parallel and also some rule were also getting added in the KieContainer at the same time.



  was:
I experienced a deadlock when running in STREAM mode and using negative patterns with temporal constraints. 
When the rule gets fired by the installed timer job and at the same time another thread is firing rules, a deadlock can occur:

1. [rule firing thread] StatefulKnowledgeSessionImpl.execute(StatefulKnowledgeSessionImpl.java:820)
acquires lock in AbstractWorkingMemory

2. [timer-thread] DefaultAgenda.fireActivation()
call of synchronized method

3. [rule firing thread] DefaultAgenda.fireActivation()
synchronized method, wait for other thread to exit synchornized method

4. [timer-thread] AbstractWorkingMemory.getGlobal()
tries to access a global, tries to acquire lock in AbstractWorkingMemory which is held by the other thread

--> Deadlock


Excerpt from thread dump:

"http-thread-pool-8082(374)":
	at org.drools.common.DefaultAgenda.fireActivation(DefaultAgenda.java:1264)
	- waiting to lock <0x00007f50407e4950> (a org.drools.common.DefaultAgenda)
	at org.drools.common.DefaultAgenda.fireNextItem(DefaultAgenda.java:1221)
	at org.drools.common.DefaultAgenda.fireAllRules(DefaultAgenda.java:1456)
	at org.drools.common.AbstractWorkingMemory.fireAllRules(AbstractWorkingMemory.java:710)
	at org.drools.common.AbstractWorkingMemory.fireAllRules(AbstractWorkingMemory.java:679)
	at org.drools.impl.StatefulKnowledgeSessionImpl.fireAllRules(StatefulKnowledgeSessionImpl.java:234)
	at org.drools.command.runtime.rule.FireAllRulesCommand.execute(FireAllRulesCommand.java:102)
	at org.drools.command.runtime.rule.FireAllRulesCommand.execute(FireAllRulesCommand.java:32)
	at org.drools.command.runtime.BatchExecutionCommandImpl.execute(BatchExecutionCommandImpl.java:155)
	at org.drools.command.runtime.BatchExecutionCommandImpl.execute(BatchExecutionCommandImpl.java:76)
	at org.drools.impl.StatefulKnowledgeSessionImpl.execute(StatefulKnowledgeSessionImpl.java:821)


"pool-4251-thread-1":
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  0x00007f503fc71a20> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
	at org.drools.common.AbstractWorkingMemory.getGlobal(AbstractWorkingMemory.java:634)
	at myrules.Rule_ruleWithTimeout_ece4a3c9fc2741a4ab97a92b77611203DefaultConsequenceInvokerGenerated.evaluate(Unknown Source)
	at org.drools.common.DefaultAgenda.fireActivation(DefaultAgenda.java:1287)
	- locked <0x00007f50407e4950> (a org.drools.common.DefaultAgenda)
	at org.drools.common.DefaultAgenda.fireTimedActivation(DefaultAgenda.java:1344)
	- locked <0x00007f50407e4950> (a org.drools.common.DefaultAgenda)
	at org.drools.common.Scheduler$ActivationTimerJob.execute(Scheduler.java:83)



> Deadlock between AbstractWorkingMemory and DefaultAgenda
> --------------------------------------------------------
>
>                 Key: DROOLS-2174
>                 URL: https://issues.jboss.org/browse/DROOLS-2174
>             Project: Drools
>          Issue Type: Bug
>    Affects Versions: 5.5.0.Final
>            Reporter: Umang Sharma
>            Assignee: Mario Fusco
>             Fix For: 6.0.0.CR2
>
>
>  I am getting a similar deadlock as the one in the cloned issue. We are using following dependencies:
> <dependency>
> <groupId>org.drools</groupId>
> <artifactId>drools-templates</artifactId>
> <version>$
> {runtime.version}</version>
> </dependency>
> <dependency>
> <groupId>org.kie</groupId>
> <artifactId>kie-api</artifactId>
> <version>${runtime.version}
> </version>
> </dependency>
> <dependency>
> <groupId>org.drools</groupId>
> <artifactId>drools-core</artifactId>
> <version>$
> {runtime.version}</version>
> </dependency>
> <dependency>
> <groupId>org.drools</groupId>
> <artifactId>drools-compiler</artifactId>
> <version>${runtime.version}
> </version>
> </dependency>
> runtime.version = 6.5.0-Final
> I have around 50 off threads stuck in a deadlock. 49 of them are having following stack trace:
> "THREAD-4" #124 prio=5 os_prio=0 tid=0x00000000015fc800 nid=0x3a5d waiting on condition [0x00007fc773e7b000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> parking to wait for <0x000000074d189818> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
> at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
> at org.drools.core.impl.KnowledgeBaseImpl.readLock(KnowledgeBaseImpl.java:703)
> at org.drools.core.impl.KnowledgeBaseImpl.newStatefulSession(KnowledgeBaseImpl.java:1434)
> at org.drools.core.impl.KnowledgeBaseImpl.newStatefulSession(KnowledgeBaseImpl.java:1422)
> at org.drools.core.impl.KnowledgeBaseImpl.newStatefulKnowledgeSession(KnowledgeBaseImpl.java:308)
> at org.drools.core.impl.KnowledgeBaseImpl.newKieSession(KnowledgeBaseImpl.java:385)
> at org.drools.compiler.kie.builder.impl.KieContainerImpl.newKieSession(KieContainerImpl.java:693)
> at org.drools.compiler.kie.builder.impl.KieContainerImpl.newKieSession(KieContainerImpl.java:629)
> at org.drools.compiler.kie.builder.impl.KieContainerImpl.newKieSession(KieContainerImpl.java:611)
> And one of them is stuck in the following stack trace:
> "THREAD-4" #124 prio=5 os_prio=0 tid=0x00000000015fc800 nid=0x3a5d waiting on condition [0x00007fc773e7b000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> parking to wait for <0x000000074d189818> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
> at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
> at org.drools.core.impl.KnowledgeBaseImpl.readLock(KnowledgeBaseImpl.java:703)
> at org.drools.core.impl.KnowledgeBaseImpl.newStatefulSession(KnowledgeBaseImpl.java:1434)
> at org.drools.core.impl.KnowledgeBaseImpl.newStatefulSession(KnowledgeBaseImpl.java:1422)
> at org.drools.core.impl.KnowledgeBaseImpl.newStatefulKnowledgeSession(KnowledgeBaseImpl.java:308)
> at org.drools.core.impl.KnowledgeBaseImpl.newKieSession(KnowledgeBaseImpl.java:385)
> at org.drools.compiler.kie.builder.impl.KieContainerImpl.newKieSession(KieContainerImpl.java:693)
> at org.drools.compiler.kie.builder.impl.KieContainerImpl.newKieSession(KieContainerImpl.java:629)
> at org.drools.compiler.kie.builder.impl.KieContainerImpl.newKieSession(KieContainerImpl.java:611)
> Please help us out in fixing this. Its a production system serving millions of users.
> PS: This is was reproduced under a medium load where we create 50 stateful sessions in parallel and also some rule were also getting added in the KieContainer at the same time.



--
This message was sent by Atlassian JIRA
(v7.5.0#75005)

More information about the jboss-jira mailing list