<div dir="ltr">We are also experiencing the same problems using 5.x and it is causing us big problems.<div><br></div><div>I haven't had time to try it yet but do any developers know whether we would be safe to change the HashMap the <span style="background-color:rgb(255,255,204);font-family:arial,sans-serif;font-size:13px">CompositiveClassLoader is using</span> to a ConcurrentHashMap?</div>
<div><br></div><div>thanks</div><div>Steve</div></div><div class="gmail_extra"><br clear="all"><div><div dir="ltr">-------------------------------------------------------------------<br>Steven Williams<div><br></div></div>
</div>
<br><br><div class="gmail_quote">On Wed, Mar 19, 2014 at 12:31 AM, mikerod <span dir="ltr"><<a href="mailto:mjr4184@gmail.com" target="_blank">mjr4184@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I cannot easily reproduce an issue that I'm seeing. This is an intermittent<br>
issue that happens probably 3% of the time or less.<br>
<br>
This is observed behavior in<br>
* Drools v5.6.0.Final, using<br>
* Janino compiler v2.5.16 && transitively Drools brings<br>
* mvel2 v2.1.8.Final<br>
<br>
We have an environment that loads around 10 different KnowledgeBases into a<br>
list.<br>
Then one-by-one a single StatefulKnowledgeSession is created for a single<br>
KnowledgeBase, facts are inserted, and then fireAllRules is called.<br>
<br>
Every so often, we noticed that we were getting what seemed to be ininite<br>
looping behavior for during either insertion time or fireAllRules time.<br>
When we subsequently re-run the *same* session with the *same* KnowledgeBase<br>
and the *same* facts inserted, it was successful and finished in<br>
milliseconds (the average for our successful runs). We have repeatedly seen<br>
this behavior of 1 failure, followed by a re-run that is successful.<br>
<br>
We were able to attach a profiler to several of these hung sessions to<br>
determine what was happening. It turns out that in both the scenario where<br>
we saw a hang up on fact insertion and on fireAllRules call, the thread dump<br>
was the same.<br>
<br>
The stack looks like:<br>
```<br>
"main" - Thread t@1<br>
java.lang.Thread.State: RUNNABLE<br>
at java.util.HashMap.getEntry(HashMap.java:347)<br>
at java.util.HashMap.containsKey(HashMap.java:335)<br>
at<br>
org.drools.util.CompositeClassLoader$CachingLoader.load(CompositeClassLoader.java:244)<br>
at<br>
org.drools.util.CompositeClassLoader$CachingLoader.load(CompositeClassLoader.java:237)<br>
at<br>
org.drools.util.CompositeClassLoader.loadClass(CompositeClassLoader.java:88)<br>
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)<br>
at java.lang.Class.forName0(Native Method)<br>
at java.lang.Class.forName(Class.java:247)<br>
at<br>
org.mvel2.ParserConfiguration.checkForDynamicImport(ParserConfiguration.java:163)<br>
at<br>
org.mvel2.ParserConfiguration.hasImport(ParserConfiguration.java:191)<br>
at org.mvel2.ParserContext.hasImport(ParserContext.java:360)<br>
at org.mvel2.ParserContext.isVariableVisible(ParserContext.java:715)<br>
at<br>
org.mvel2.compiler.ExpressionCompiler.verify(ExpressionCompiler.java:394)<br>
at<br>
org.mvel2.compiler.ExpressionCompiler._compile(ExpressionCompiler.java:250)<br>
at<br>
org.mvel2.compiler.ExpressionCompiler.compile(ExpressionCompiler.java:62)<br>
at org.mvel2.MVEL.compileExpression(MVEL.java:810)<br>
at<br>
org.drools.base.mvel.MVELCompilationUnit.compile(MVELCompilationUnit.java:417)<br>
at<br>
org.drools.base.mvel.MVELCompilationUnit.getCompiledExpression(MVELCompilationUnit.java:238)<br>
at<br>
org.drools.rule.constraint.MvelConstraint.createMvelConditionEvaluator(MvelConstraint.java:224)<br>
at<br>
org.drools.rule.constraint.MvelConstraint.evaluate(MvelConstraint.java:208)<br>
at<br>
org.drools.rule.constraint.MvelConstraint.isAllowed(MvelConstraint.java:175)<br>
at org.drools.reteoo.AlphaNode.assertObject(AlphaNode.java:133)<br>
at<br>
org.drools.reteoo.CompositeObjectSinkAdapter.doPropagateAssertObject(CompositeObjectSinkAdapter.java:497)<br>
at<br>
org.drools.reteoo.CompositeObjectSinkAdapter.propagateAssertObject(CompositeObjectSinkAdapter.java:382)<br>
at<br>
org.drools.reteoo.ObjectTypeNode.assertObject(ObjectTypeNode.java:302)<br>
at<br>
org.drools.reteoo.EntryPointNode.assertObject(EntryPointNode.java:254)<br>
at org.drools.common.NamedEntryPoint.insert(NamedEntryPoint.java:366)<br>
at<br>
org.drools.common.SimpleBeliefSystem.insert(SimpleBeliefSystem.java:38)<br>
at<br>
org.drools.common.TruthMaintenanceSystem.addLogicalDependency(TruthMaintenanceSystem.java:207)<br>
at<br>
org.drools.common.TruthMaintenanceSystem.addLogicalDependency(TruthMaintenanceSystem.java:179)<br>
at org.drools.common.NamedEntryPoint.insert(NamedEntryPoint.java:247)<br>
at<br>
org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:950)<br>
at<br>
org.drools.base.DefaultKnowledgeHelper.insertLogical(DefaultKnowledgeHelper.java:263)<br>
at<br>
org.drools.base.DefaultKnowledgeHelper.insertLogical(DefaultKnowledgeHelper.java:228)<br>
at<br>
org.drools.base.DefaultKnowledgeHelper.insertLogical(DefaultKnowledgeHelper.java:223)<br>
<application-stack><br>
at some.drools.generated.rule.package.Rule_<drools-generated2><br>
at some.drools.generated.rule.package.Rule_<drools-generated1><br>
at<br>
org.drools.common.DefaultAgenda.fireActivation(DefaultAgenda.java:1282)<br>
- locked <656dd9a0> (a org.drools.common.DefaultAgenda)<br>
at<br>
org.drools.common.DefaultAgenda.fireNextItem(DefaultAgenda.java:1216)<br>
- locked <656dd9a0> (a org.drools.common.DefaultAgenda)<br>
at<br>
org.drools.common.DefaultAgenda.fireAllRules(DefaultAgenda.java:1451)<br>
at<br>
org.drools.common.AbstractWorkingMemory.fireAllRules(AbstractWorkingMemory.java:756)<br>
at<br>
org.drools.common.AbstractWorkingMemory.fireAllRules(AbstractWorkingMemory.java:718)<br>
at<br>
org.drools.impl.StatefulKnowledgeSessionImpl.fireAllRules(StatefulKnowledgeSessionImpl.java:230)<br>
<facts-inserted-previously><br>
```<br>
<br>
We were able to get the same hung thread dump prior to fireAllRules. I<br>
think the related entry point is during one of the #assertObject calls<br>
midway through the stack.<br>
The CompositiveClassLoader$CachingLoader definitely seems to be the issue.<br>
<br>
When we found this behavior consistently occurring (although rare), I found<br>
the option of turning off the use of the caching loader by setting the<br>
"drools.classLoaderCacheEnabled"="false" option referenced in the<br>
ClassLoaderCacheOption class.<br>
With this setting, we have done extensive retries to find this behavior and<br>
it has vanished.<br>
<br>
This CompositiveClassLoader$CachingLoader uses an internal j.u.HashMap that<br>
is not safe for concurrent access. I have listed below several related<br>
posts on the topic. I'm fairly sure if this used a j.u.c.ConcurrentHashMap,<br>
this hanging thread scenario would not happen.<br>
<br>
However, most of the posts I've seen on this subject are from use-cases<br>
where the application is explicitly doing some sort of multithreaded access<br>
to the Drools<br>
KnowledgeBase and/or StatefulKnowledgeSession.<br>
<br>
In my case, I do not know of *any* multithreaded actions taking place within<br>
my application around the Drools KnowledgeBase or StatefulKnowledgeSession.<br>
I am not seeing ConcurrentModificationException though. Instead I'm seeing<br>
a, seemingly infinite, loop in the j.u.HashMap#getEntry.<br>
<br>
Hanging in a "get" method of the j.u.HashMap would suggest to me that there<br>
is a race condition where sometimes the j.u.HashMap#put on<br>
the `classLoaderResultMap` field in the<br>
CompositiveClassLoader$CachingLoader#load method is being executed at the<br>
same time as another thread is doing the<br>
j.u.HashMap#getEntry. If the j.u.HashMap were to resize at this point, it<br>
could cause an infinite looping behavior.<br>
<br>
Does Drools internally use multithreading? Is there somewhere in the MVEL<br>
lib or the Janino compiler where it may be concurrently accessing the<br>
CompositiveClassLoader$CachingLoader$load method?<br>
I have been digging around a lot and I cannot find what could be the root<br>
cause of this behavior.<br>
<br>
<br>
I think this relates directly to these:<br>
<br>
* <a href="http://lists.jboss.org/pipermail/rules-users/2013-July/032446.html" target="_blank">http://lists.jboss.org/pipermail/rules-users/2013-July/032446.html</a><br>
* <a href="http://lists.jboss.org/pipermail/rules-users/2013-July/032446.html" target="_blank">http://lists.jboss.org/pipermail/rules-users/2013-July/032446.html</a><br>
<br>
However, I do not see a real resolution to the problem.<br>
<br>
I also see several related issues:<br>
<br>
* <a href="http://lists.jboss.org/pipermail/rules-users/2013-July/032446.html" target="_blank">http://lists.jboss.org/pipermail/rules-users/2013-July/032446.html</a><br>
* <a href="https://issues.jboss.org/browse/JBRULES-3552" target="_blank">https://issues.jboss.org/browse/JBRULES-3552</a><br>
<br>
<br>
<br>
<br>
--<br>
View this message in context: <a href="http://drools.46999.n3.nabble.com/CompositiveClassLoader-CachingLoader-load-method-intermittently-hangs-tp4028780.html" target="_blank">http://drools.46999.n3.nabble.com/CompositiveClassLoader-CachingLoader-load-method-intermittently-hangs-tp4028780.html</a><br>
Sent from the Drools: User forum mailing list archive at Nabble.com.<br>
_______________________________________________<br>
rules-users mailing list<br>
<a href="mailto:rules-users@lists.jboss.org">rules-users@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/rules-users" target="_blank">https://lists.jboss.org/mailman/listinfo/rules-users</a><br>
</blockquote></div><br></div>