Deadlock in ProjectClassLoader ------------------------------ Key: DROOLS-5615 URL: https://issues.redhat.com/browse/DROOLS-5615 Project: Drools Issue Type: Bug Components: core engine Affects Versions: 6.5.0.Final Reporter: Michael Osganian Assignee: Mario Fusco Priority: Major Hi I am using 6.5.0.Final of drools and I have a deadlock situation that is hard to reproduce.  We are running about 4 threads each running its own rule engine with the same set of rules but different data.  Below is the deadlock on 3 of the threads and looks like the ProjectClassLoader is at the root of the problem:   Found one Java-level deadlock:Found one Java-level deadlock:============================="InferenceThread-4":  waiting to lock monitor 0x00007fd5cc044a38 (object 0x0000000081b0ee18, a org.drools.core.rule.JavaDialectRuntimeData$PackageClassLoader),  which is held by "InferenceThread-2""InferenceThread-2":  waiting to lock monitor 0x00007fd5cc044ae8 (object 0x0000000081a2a4a8, a org.drools.core.common.ProjectClassLoader$DefaultInternalTypesClassLoader),  which is held by "InferenceThread-1""InferenceThread-1":  waiting to lock monitor 0x00007fd5bc01fd28 (object 0x0000000081acc610, a org.drools.core.common.ProjectClassLoader),  which is held by "InferenceThread-2" Java stack information for the threads listed above:==================================================="InferenceThread-4": at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.mvel2.util.ParseTools.forNameWithInner(ParseTools.java:2187) at org.mvel2.ParserConfiguration.checkForDynamicImport(ParserConfiguration.java:169) at org.mvel2.ParserConfiguration.hasImport(ParserConfiguration.java:197) at org.mvel2.ParserContext.hasImport(ParserContext.java:373) at org.mvel2.compiler.AbstractParser.createPropertyToken(AbstractParser.java:1399) at org.mvel2.compiler.AbstractParser.nextToken(AbstractParser.java:903) at org.mvel2.compiler.ExpressionCompiler._compile(ExpressionCompiler.java:111) at org.mvel2.compiler.ExpressionCompiler.compile(ExpressionCompiler.java:52) at org.mvel2.MVEL.compileExpression(MVEL.java:810) at org.drools.core.base.mvel.MVELCompilationUnit.compile(MVELCompilationUnit.java:451) at org.drools.core.base.mvel.MVELCompilationUnit.getCompiledExpression(MVELCompilationUnit.java:273) at org.drools.core.rule.constraint.MvelConstraint.createMvelConditionEvaluator(MvelConstraint.java:269) at org.drools.core.rule.constraint.MvelConstraint.evaluate(MvelConstraint.java:249) at org.drools.core.rule.constraint.MvelConstraint.isAllowedCachedLeft(MvelConstraint.java:227) at org.drools.core.common.TripleBetaConstraints.isAllowedCachedLeft(TripleBetaConstraints.java:118) at org.drools.core.phreak.PhreakJoinNode.doLeftInserts(PhreakJoinNode.java:113) at org.drools.core.phreak.PhreakJoinNode.doNode(PhreakJoinNode.java:76) at org.drools.core.phreak.RuleNetworkEvaluator.switchOnDoBetaNode(RuleNetworkEvaluator.java:519) at org.drools.core.phreak.RuleNetworkEvaluator.evalBetaNode(RuleNetworkEvaluator.java:505) at org.drools.core.phreak.RuleNetworkEvaluator.evalNode(RuleNetworkEvaluator.java:341) at org.drools.core.phreak.RuleNetworkEvaluator.innerEval(RuleNetworkEvaluator.java:301) at org.drools.core.phreak.RuleNetworkEvaluator.outerEval(RuleNetworkEvaluator.java:136) at org.drools.core.phreak.RuleNetworkEvaluator.evaluateNetwork(RuleNetworkEvaluator.java:94) at org.drools.core.phreak.RuleExecutor.reEvaluateNetwork(RuleExecutor.java:194) at org.drools.core.phreak.RuleExecutor.evaluateNetworkAndFire(RuleExecutor.java:73) at org.drools.core.common.DefaultAgenda.fireNextItem(DefaultAgenda.java:970) at org.drools.core.common.DefaultAgenda.fireLoop(DefaultAgenda.java:1312) at org.drools.core.common.DefaultAgenda.fireAllRules(DefaultAgenda.java:1251) at org.drools.core.impl.StatefulKnowledgeSessionImpl.internalFireAllRules(StatefulKnowledgeSessionImpl.java:1359) at org.drools.core.impl.StatefulKnowledgeSessionImpl.fireAllRules(StatefulKnowledgeSessionImpl.java:1350) at org.drools.core.impl.StatefulKnowledgeSessionImpl.fireAllRules(StatefulKnowledgeSessionImpl.java:1331) at ActivityInferenceSession.fireRules(ActivityInferenceSession.java:261) at InferenceRunnable.doTryInference(InferenceRunnable.java:226) at InferenceRunnable.doInference(InferenceRunnable.java:129) at InferenceRunnable.run(InferenceRunnable.java:100) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)"InferenceThread-2": at java.lang.ClassLoader.loadClass(ClassLoader.java:404) - waiting to lock <0x0000000081a2a4a8> (a org.drools.core.common.ProjectClassLoader$DefaultInternalTypesClassLoader) at org.drools.core.common.ProjectClassLoader$DefaultInternalTypesClassLoader.loadType(ProjectClassLoader.java:394) at org.drools.core.common.ProjectClassLoader.loadType(ProjectClassLoader.java:172) at org.drools.core.common.ProjectClassLoader.loadClass(ProjectClassLoader.java:143) - locked <0x0000000081acc610> (a org.drools.core.common.ProjectClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.drools.core.rule.JavaDialectRuntimeData$PackageClassLoader.loadClass(JavaDialectRuntimeData.java:646) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.mvel2.util.ParseTools.forNameWithInner(ParseTools.java:2187) at org.mvel2.ParserConfiguration.checkForDynamicImport(ParserConfiguration.java:169) at org.mvel2.ParserConfiguration.hasImport(ParserConfiguration.java:197) at org.mvel2.ParserContext.hasImport(ParserContext.java:373) at org.mvel2.compiler.AbstractParser.createPropertyToken(AbstractParser.java:1399) at org.mvel2.compiler.AbstractParser.nextToken(AbstractParser.java:903) at org.mvel2.compiler.ExpressionCompiler._compile(ExpressionCompiler.java:111) at org.mvel2.compiler.ExpressionCompiler.compile(ExpressionCompiler.java:52) at org.mvel2.MVEL.compileExpression(MVEL.java:810) at org.drools.core.base.mvel.MVELCompilationUnit.compile(MVELCompilationUnit.java:451) at org.drools.core.base.mvel.MVELCompilationUnit.getCompiledExpression(MVELCompilationUnit.java:273) at org.drools.core.rule.constraint.MvelConstraint.createMvelConditionEvaluator(MvelConstraint.java:269) at org.drools.core.rule.constraint.MvelConstraint.evaluate(MvelConstraint.java:249) at org.drools.core.rule.constraint.MvelConstraint.isAllowedCachedLeft(MvelConstraint.java:227) at org.drools.core.common.TripleBetaConstraints.isAllowedCachedLeft(TripleBetaConstraints.java:118) at org.drools.core.phreak.PhreakJoinNode.doLeftInserts(PhreakJoinNode.java:113) at org.drools.core.phreak.PhreakJoinNode.doNode(PhreakJoinNode.java:76) at org.drools.core.phreak.RuleNetworkEvaluator.switchOnDoBetaNode(RuleNetworkEvaluator.java:519) at org.drools.core.phreak.RuleNetworkEvaluator.evalBetaNode(RuleNetworkEvaluator.java:505) at org.drools.core.phreak.RuleNetworkEvaluator.evalNode(RuleNetworkEvaluator.java:341) at org.drools.core.phreak.RuleNetworkEvaluator.innerEval(RuleNetworkEvaluator.java:301) at org.drools.core.phreak.RuleNetworkEvaluator.outerEval(RuleNetworkEvaluator.java:136) at org.drools.core.phreak.RuleNetworkEvaluator.evaluateNetwork(RuleNetworkEvaluator.java:94) at org.drools.core.phreak.RuleExecutor.reEvaluateNetwork(RuleExecutor.java:194) at org.drools.core.phreak.RuleExecutor.evaluateNetworkAndFire(RuleExecutor.java:73) at org.drools.core.common.DefaultAgenda.fireNextItem(DefaultAgenda.java:970) at org.drools.core.common.DefaultAgenda.fireLoop(DefaultAgenda.java:1312) at org.drools.core.common.DefaultAgenda.fireAllRules(DefaultAgenda.java:1251) at org.drools.core.impl.StatefulKnowledgeSessionImpl.internalFireAllRules(StatefulKnowledgeSessionImpl.java:1359) at org.drools.core.impl.StatefulKnowledgeSessionImpl.fireAllRules(StatefulKnowledgeSessionImpl.java:1350) at org.drools.core.impl.StatefulKnowledgeSessionImpl.fireAllRules(StatefulKnowledgeSessionImpl.java:1331) at ActivityInferenceSession.fireRules(ActivityInferenceSession.java:261) at InferenceRunnable.doTryInference(InferenceRunnable.java:226) at InferenceRunnable.doInference(InferenceRunnable.java:129) at InferenceRunnable.run(InferenceRunnable.java:100) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)"InferenceThread-1": at org.drools.core.common.ProjectClassLoader$DefaultInternalTypesClassLoader.loadClass(ProjectClassLoader.java:385) - waiting to lock <0x0000000081acc610> (a org.drools.core.common.ProjectClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at com.sun.beans.finder.ClassFinder.findClass(ClassFinder.java:103) at java.beans.Introspector.findCustomizerClass(Introspector.java:1301) at java.beans.Introspector.getTargetBeanDescriptor(Introspector.java:1295) at java.beans.Introspector.getBeanInfo(Introspector.java:425) at java.beans.Introspector.getBeanInfo(Introspector.java:173) at org.apache.commons.beanutils.PropertyUtilsBean.getPropertyDescriptors(PropertyUtilsBean.java:980) at org.apache.commons.beanutils.PropertyUtilsBean.getPropertyDescriptors(PropertyUtilsBean.java:1084) at org.apache.commons.beanutils.PropertyUtilsBean.getPropertyDescriptor(PropertyUtilsBean.java:912) at org.apache.commons.beanutils.PropertyUtilsBean.getSimpleProperty(PropertyUtilsBean.java:1319) at org.apache.commons.beanutils.PropertyUtilsBean.getNestedProperty(PropertyUtilsBean.java:770) at org.apache.commons.beanutils.PropertyUtilsBean.getProperty(PropertyUtilsBean.java:846) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mvel2.optimizers.impl.refl.ReflectiveAccessorOptimizer.getMethod(ReflectiveAccessorOptimizer.java:1120) at org.mvel2.optimizers.impl.refl.ReflectiveAccessorOptimizer.getMethod(ReflectiveAccessorOptimizer.java:1003) at org.mvel2.optimizers.impl.refl.ReflectiveAccessorOptimizer.compileGetChain(ReflectiveAccessorOptimizer.java:396) at org.mvel2.optimizers.impl.refl.ReflectiveAccessorOptimizer.optimizeAccessor(ReflectiveAccessorOptimizer.java:163) at org.mvel2.ast.ASTNode.optimize(ASTNode.java:159) at org.mvel2.ast.ASTNode.getReducedValueAccelerated(ASTNode.java:115) at org.mvel2.ast.BinaryOperation.getReducedValueAccelerated(BinaryOperation.java:117) at org.mvel2.compiler.ExecutableAccessor.getValue(ExecutableAccessor.java:38) at org.mvel2.ast.Substatement.getReducedValueAccelerated(Substatement.java:44) at org.mvel2.ast.Or.getReducedValueAccelerated(Or.java:34) at org.mvel2.compiler.ExecutableAccessor.getValue(ExecutableAccessor.java:38) at org.mvel2.ast.Substatement.getReducedValueAccelerated(Substatement.java:44) at org.mvel2.ast.And.getReducedValueAccelerated(And.java:34) at org.mvel2.MVELRuntime.execute(MVELRuntime.java:85) at org.mvel2.compiler.CompiledExpression.getDirectValue(CompiledExpression.java:123) at org.mvel2.compiler.CompiledExpression.getValue(CompiledExpression.java:119) at org.mvel2.compiler.CompiledExpression.getValue(CompiledExpression.java:113) at org.mvel2.MVEL.executeExpression(MVEL.java:929) at org.drools.core.base.mvel.MVELEvalExpression.evaluate(MVELEvalExpression.java:104) at org.drools.core.rule.EvalCondition.isAllowed(EvalCondition.java:118) at org.drools.core.phreak.PhreakEvalNode.doLeftInserts(PhreakEvalNode.java:72) at org.drools.core.phreak.PhreakEvalNode.doNode(PhreakEvalNode.java:56) at org.drools.core.phreak.RuleNetworkEvaluator.evalNode(RuleNetworkEvaluator.java:349) at org.drools.core.phreak.RuleNetworkEvaluator.innerEval(RuleNetworkEvaluator.java:301) at org.drools.core.phreak.RuleNetworkEvaluator.outerEval(RuleNetworkEvaluator.java:136) at org.drools.core.phreak.RuleNetworkEvaluator.evaluateNetwork(RuleNetworkEvaluator.java:94) at org.drools.core.phreak.RuleExecutor.reEvaluateNetwork(RuleExecutor.java:194) at org.drools.core.phreak.RuleExecutor.evaluateNetworkAndFire(RuleExecutor.java:73) at org.drools.core.common.DefaultAgenda.fireNextItem(DefaultAgenda.java:970) at org.drools.core.common.DefaultAgenda.fireLoop(DefaultAgenda.java:1312) at org.drools.core.common.DefaultAgenda.fireAllRules(DefaultAgenda.java:1251) at org.drools.core.impl.StatefulKnowledgeSessionImpl.internalFireAllRules(StatefulKnowledgeSessionImpl.java:1359) at org.drools.core.impl.StatefulKnowledgeSessionImpl.fireAllRules(StatefulKnowledgeSessionImpl.java:1350) at org.drools.core.impl.StatefulKnowledgeSessionImpl.fireAllRules(StatefulKnowledgeSessionImpl.java:1331) at ActivityInferenceSession.fireRules(ActivityInferenceSession.java:261) at InferenceRunnable.doTryInference(InferenceRunnable.java:226) at InferenceRunnable.doInference(InferenceRunnable.java:129) at InferenceRunnable.run(InferenceRunnable.java:100) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Found 1 deadlock.   I know there has been other issues entered on deadlock with ProjectClassLoader and they have either been fixed in an earlier version or closed as not reproducible.  Its hard to reproduce this as it has only happened in production a few times.  In some of the rules we are running we are using PropertyUtilsBean as you can see in one of the deadlock threads so not sure if that is a cause for concern here or not but in our situation each thread has its own PropertyUtilsBean instance.