<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Hi Edson,<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Our emails just crossed paths. Yes, I believe that I can
reproduce this. I’ll try to turn this around quickly.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Ed<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>
<p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>
rules-dev-bounces@lists.jboss.org [mailto:rules-dev-bounces@lists.jboss.org] <b>On
Behalf Of </b>Edson Tirelli<br>
<b>Sent:</b> Wednesday, November 04, 2009 5:06 PM<br>
<b>To:</b> Rules Dev List<br>
<b>Subject:</b> Re: [rules-dev] Deadlock in the Drools core - Drools 5.0 - any
suggestions for resolution?<o:p></o:p></span></p>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
Edward,<br>
<br>
Are you able to provide us with a test case? that would help us
ensure we fix this and prevent future regressions.<br>
<br>
Thanks,<br>
Edson<o:p></o:p></p>
<div>
<p class=MsoNormal>2009/11/4 Edward Archibald <<a
href="mailto:edward.archibald@continuent.com">edward.archibald@continuent.com</a>><o:p></o:p></p>
<p class=MsoNormal>Hi Greg,<br>
<br>
Thanks for the post. I'll give this a shot. Turns out that I can
reproduce the issue often enough that I'll be able to see if this simple change
resolves it.<br>
<br>
Regards,<br>
<br>
Edward<br>
<br>
________________________________________<br>
From: <a href="mailto:rules-dev-bounces@lists.jboss.org">rules-dev-bounces@lists.jboss.org</a>
[<a href="mailto:rules-dev-bounces@lists.jboss.org">rules-dev-bounces@lists.jboss.org</a>]
On Behalf Of Greg Barton [<a href="mailto:greg_barton@yahoo.com">greg_barton@yahoo.com</a>]<br>
Sent: Tuesday, November 03, 2009 9:43 PM<br>
To: Rules Dev List<br>
Subject: Re: [rules-dev] Deadlock in the Drools core - Drools 5.0 - any suggestions
for resolution?<o:p></o:p></p>
<div>
<div>
<p class=MsoNormal><br>
Well, I'm not sure how to avoid the deadlock without changing the drools
codebase. I was, however, able to change the type of
AbstractWorkingMemory.actionQueue to java.util.concurrent.ConcurrentLinkedQueue
and remove the synchronization over the queue with no apparent ill effects.
(Two tests failed for drools-core, but they failed whether the change was made
or not.) Also I don't like the fact that the current code synchronizes on
actionQueue, but then exposes it outside the class through the getActionQueue()
method, where access can be unsynchronized. Changing it to
ConcurrentLinkedQueue makes it safe to expose externally. (Not to mention that
the lock can be stolen externally with the current code.)<br>
<br>
diff attached. If you can run drools compiled from trunk, apply the diff
and see if it resolves the deadlock. If it does it's up to the drools
devs as to whether the change should be made. I'm just hacking about. :P<br>
<br>
--- On Tue, 11/3/09, Edward Archibald <<a
href="mailto:edward.archibald@continuent.com">edward.archibald@continuent.com</a>>
wrote:<br>
<br>
> From: Edward Archibald <<a href="mailto:edward.archibald@continuent.com">edward.archibald@continuent.com</a>><br>
> Subject: [rules-dev] Deadlock in the Drools core - Drools 5.0 - any suggestions
for resolution?<br>
> To: "<a href="mailto:rules-dev@lists.jboss.org">rules-dev@lists.jboss.org</a>"
<<a href="mailto:rules-dev@lists.jboss.org">rules-dev@lists.jboss.org</a>><br>
> Date: Tuesday, November 3, 2009, 9:41 PM<br>
><br>
> I found the following deadlock which is, apparently, due to<br>
> the concurrent execution<br>
> of a task for a 'delayed' rule with a concurrently<br>
> executing application thread attempting to get access to a<br>
> 'global'. Any recommendations for avoiding this type<br>
> of deadlock besides not using rules with 'duration()' etc.<br>
> which cause asynchronous execution with respect to my main<br>
> application thread?<br>
><br>
> This problem is somewhat difficult to reproduce on demand<br>
> but it does come up frequently when the 'delayed' rule<br>
> "DETECT MONITORING HAS STOPPED" is activated as a result of<br>
> the trigger conditions.<br>
><br>
>
===================================================================================<br>
><br>
> This thread, my application's EnterprisePolicyManager<br>
> thread, is attempting to get access to a global, policyMgr,<br>
> and is waiting for<br>
> the 'lock.lock' on RetooStatefulSession<br>
><br>
> It owns the 'ReteooStatefulSession.actionQueue'<br>
> and is waiting for the ReteooStatefulSession.lock.lock<br>
><br>
> owns: java.util.LinkedList<E> (id=207)<br>
> waited by: Thread [pool-3-thread-1] (Suspended)<br>
> owns:<br>
> com.continuent.tungsten.cluster.manager.rules.engine.RulesEngine<br>
> (id=208)<br>
> sun.misc.Unsafe.park(boolean, long) line: not available<br>
> [native method] [local variables unavailable]<br>
> java.util.concurrent.locks.LockSupport.park() line: 118<br>
> [local variables unavailable]<br>
>
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()<br>
> line: 681 [local variables unavailable]<br>
> java.util.concurrent.locks.ReentrantLock$NonfairSync(java.util.concurrent.locks.AbstractQueuedSynchronizer).acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node,<br>
> int) line: 711<br>
>
java.util.concurrent.locks.ReentrantLock$NonfairSync(java.util.concurrent.locks.AbstractQueuedSynchronizer).acquire(int)<br>
> line: 1041<br>
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock()<br>
> line: 184 [local variables unavailable]<br>
> java.util.concurrent.locks.ReentrantLock.lock() line: 256<br>
> [local variables unavailable]<br>
> org.drools.reteoo.ReteooStatefulSession(org.drools.common.AbstractWorkingMemory).getGlobal(java.lang.String)<br>
> line: 587<br>
>
com.continuent.tungsten.cluster.manager.policy.Rule_IF_IN_MAINTENANCE_MODE__CONSUME_ALL_NOTIFICATIONS_0Eval0Invoker.evaluate(org.drools.spi.Tuple,<br>
> org.drools.rule.Declaration[], org.drools.WorkingMemory,<br>
> java.lang.Object) line: not available<br>
> org.drools.rule.EvalCondition.isAllowed(org.drools.spi.Tuple,<br>
> org.drools.WorkingMemory, java.lang.Object) line: 117<br>
> org.drools.reteoo.EvalConditionNode.assertLeftTuple(org.drools.reteoo.LeftTuple,<br>
> org.drools.spi.PropagationContext,<br>
> org.drools.common.InternalWorkingMemory) line: 180<br>
>
org.drools.reteoo.SingleLeftTupleSinkAdapter.doPropagateAssertLeftTuple(org.drools.spi.PropagationContext,<br>
> org.drools.common.InternalWorkingMemory,<br>
> org.drools.reteoo.LeftTuple) line: 117<br>
>
org.drools.reteoo.SingleLeftTupleSinkAdapter.propagateAssertLeftTuple(org.drools.reteoo.LeftTuple,<br>
> org.drools.reteoo.RightTuple,<br>
> org.drools.spi.PropagationContext,<br>
> org.drools.common.InternalWorkingMemory, boolean) line: 28<br>
>
org.drools.reteoo.JoinNode.assertObject(org.drools.common.InternalFactHandle,<br>
> org.drools.spi.PropagationContext,<br>
> org.drools.common.InternalWorkingMemory) line: 175<br>
> org.drools.reteoo.SingleObjectSinkAdapter.propagateAssertObject(org.drools.common.InternalFactHandle,<br>
> org.drools.spi.PropagationContext,<br>
> org.drools.common.InternalWorkingMemory) line: 42<br>
>
org.drools.reteoo.PropagationQueuingNode$AssertAction.execute(org.drools.reteoo.ObjectSinkPropagator,<br>
> org.drools.common.InternalWorkingMemory) line: 326<br>
>
org.drools.reteoo.PropagationQueuingNode.propagateActions(org.drools.common.InternalWorkingMemory)<br>
> line: 221<br>
>
org.drools.reteoo.PropagationQueuingNode$PropagateAction.execute(org.drools.common.InternalWorkingMemory)<br>
> line: 394<br>
>
org.drools.reteoo.ReteooStatefulSession(org.drools.common.AbstractWorkingMemory).executeQueuedActions()<br>
> line: 1486<br>
>
org.drools.common.NamedEntryPoint.insert(org.drools.common.InternalFactHandle,<br>
> java.lang.Object, org.drools.rule.Rule,<br>
> org.drools.spi.Activation) line: 158<br>
> org.drools.common.NamedEntryPoint.insert(java.lang.Object,<br>
> boolean, boolean, org.drools.rule.Rule,<br>
> org.drools.spi.Activation) line: 122<br>
> org.drools.common.NamedEntryPoint.insert(java.lang.Object)<br>
> line: 80<br>
> com.continuent.tungsten.cluster.manager.rules.engine.RulesEngine.insertFact(com.continuent.tungsten.commons.cluster.resource.notification.NotificationStreamID,<br>
> java.lang.Object, boolean) line: 162<br>
> com.continuent.tungsten.cluster.manager.policy.EnterprisePolicyManager.run()<br>
> line: 249<br>
> java.lang.Thread.run() line: 595<br>
><br>
> The rule implicated in the above thread is:<br>
><br>
> rule "IF IN MAINTENANCE MODE, CONSUME ALL NOTIFICATIONS"<br>
> salience 999<br>
> when<br>
> notification : ClusterResourceNotification()<br>
> from entry-point "MONITORING"<br>
> eval(policyMgr.getMode() ==<br>
> ClusterPolicyManagerMode.MAINTENANCE)<br>
> then<br>
> statistics.increment("IF IN<br>
> MAINTENANCE MODE, CONSUME ALL NOTIFICATIONS");<br>
> retract(notification);<br>
> end<br>
><br>
><br>
><br>
> This other thread, apparently a scheduled thread for a rule<br>
> with a 10 second duration,<br>
> is attempting to insert a fact and owns the 'lock.lock' on<br>
> ReteooStatefulSession and<br>
> is waiting for the 'ReteooStatefulSession.actionQueue'.<br>
><br>
> owns: org.drools.common.DefaultAgenda (id=4046)<br>
> waiting for: java.util.LinkedList<E> (id=207)<br>
>
org.drools.reteoo.ReteooStatefulSession(org.drools.common.AbstractWorkingMemory).executeQueuedActions()<br>
> line: 1480<br>
>
org.drools.reteoo.ReteooStatefulSession(org.drools.common.AbstractWorkingMemory).insert(org.drools.common.InternalFactHandle,<br>
> java.lang.Object, org.drools.rule.Rule,<br>
> org.drools.spi.Activation, org.drools.reteoo.ObjectTypeConf)<br>
> line: 1051<br>
>
org.drools.reteoo.ReteooStatefulSession(org.drools.common.AbstractWorkingMemory).insert(java.lang.Object,<br>
> boolean, boolean, org.drools.rule.Rule,<br>
> org.drools.spi.Activation) line: 1001<br>
> org.drools.base.DefaultKnowledgeHelper.insert(java.lang.Object,<br>
> boolean) line: 114<br>
> org.drools.base.DefaultKnowledgeHelper.insert(java.lang.Object)<br>
> line: 108<br>
>
com.continuent.tungsten.cluster.manager.policy.Rule_DETECT_MONITORING_HAS_STOPPED_0.consequence(org.drools.spi.KnowledgeHelper,<br>
>
com.continuent.tungsten.commons.cluster.resource.notification.DataServerNotification,<br>
> org.drools.FactHandle, java.lang.String,<br>
> org.drools.FactHandle,<br>
> com.continuent.tungsten.cluster.manager.policy.EnterprisePolicyManager,<br>
> org.apache.log4j.Logger) line: not available<br>
>
com.continuent.tungsten.cluster.manager.policy.Rule_DETECT_MONITORING_HAS_STOPPED_0ConsequenceInvoker.evaluate(org.drools.spi.KnowledgeHelper,<br>
> org.drools.WorkingMemory) line: not available<br>
> org.drools.common.DefaultAgenda.fireActivation(org.drools.spi.Activation)<br>
> line: 934<br>
>
org.drools.common.Scheduler$DuractionJob.execute(org.drools.time.JobContext)<br>
> line: 70<br>
> org.drools.time.impl.JDKTimerService$JDKCallableJob.call()<br>
> line: 132<br>
> org.drools.time.impl.JDKTimerService$JDKCallableJob.call()<br>
> line: 110<br>
> java.util.concurrent.FutureTask$Sync.innerRun() line: 269<br>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask<V>(java.util.concurrent.FutureTask<V>).run()<br>
> line: 123<br>
>
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask<V>.access$301(java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask)<br>
> line: 65<br>
>
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask<V>.run()<br>
> line: 168<br>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable)<br>
> line: 650<br>
> java.util.concurrent.ThreadPoolExecutor$Worker.run() line:<br>
> 675<br>
> java.lang.Thread.run() line: 595<br>
><br>
> The rule for this task looks like:<br>
> rule "DETECT MONITORING HAS STOPPED"<br>
> duration(10s)<br>
> salience 1000<br>
> when<br>
> lastNotification :<br>
> DataServerNotification($resourceName : resourceName)<br>
><br>
> from entry-point "MONITORING"<br>
><br>
> not (DataServerNotification(resourceName ==<br>
> $resourceName,<br>
><br>
><br>
> this after [10s] lastNotification)<br>
><br>
> from entry-point "MONITORING")<br>
><br>
> not (ManagerFailedAlarm(expired == false,<br>
><br>
> resourceName == $resourceName))<br>
><br>
> not (DataSource(name == $resourceName,<br>
><br>
> state == ResourceState.SHUNNED ||<br>
><br>
> state == ResourceState.FAILED))<br>
><br>
> then<br>
> Object[] params = {$resourceName};<br>
> if (policyMgr.getMode() !=<br>
> ClusterPolicyManagerMode.MAINTENANCE)<br>
> {<br>
><br>
> lastNotification.setResourceState(ResourceState.UNKNOWN);<br>
> ManagerFailedAlarm alarm =<br>
><br>
> new<br>
> ManagerFailedAlarm(lastNotification, "rule detected monitor<br>
> stop",<br>
><br>
><br>
> 6, AlarmSeverity.FAULT);<br>
> <a href="http://logger.info" target="_blank">logger.info</a>(alarm.toString());<br>
> insert(alarm);<br>
> update(lastNotification);<br>
> }<br>
> end<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> _______________________________________________<br>
> rules-dev mailing list<br>
> <a href="mailto:rules-dev@lists.jboss.org">rules-dev@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/rules-dev"
target="_blank">https://lists.jboss.org/mailman/listinfo/rules-dev</a><br>
><br>
<br>
<br>
<br>
_______________________________________________<br>
rules-dev mailing list<br>
<a href="mailto:rules-dev@lists.jboss.org">rules-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/rules-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/rules-dev</a><o:p></o:p></p>
</div>
</div>
</div>
<p class=MsoNormal><br>
<br clear=all>
<br>
-- <br>
Edson Tirelli<br>
JBoss Drools Core Development<br>
JBoss by Red Hat @ <a href="http://www.jboss.com">www.jboss.com</a><o:p></o:p></p>
</div>
</body>
</html>