Stuck thread due to deadlock in ProjectClassLoader -------------------------------------------------- Key: DROOLS-807 URL: https://issues.jboss.org/browse/DROOLS-807 Project: Drools Issue Type: Bug Components: core engine Affects Versions: 6.1.0.Final Reporter: Munish Kumar Assignee: Mario Fusco We see below deadlock information in our Production tomcat Found one Java-level deadlock: ============================= "tomcat-http--74": waiting to lock monitor 0x00007fae28022218 (object 0x00000000902d8d20, a org.drools.core.common.ProjectClassLoader), which is held by "tomcat-http--36" "tomcat-http--36": waiting to lock monitor 0x00007fae28042838 (object 0x000000009044e5b8, a org.drools.core.common.ProjectClassLoader$InternalTypesClassLoader), which is held by "Metrics Collection" "Metrics Collection": waiting to lock monitor 0x00007fae28022218 (object 0x00000000902d8d20, a org.drools.core.common.ProjectClassLoader), which is held by "tomcat-http--36" Java stack information for the threads listed above: =================================================== "tomcat-http--74": at org.drools.core.common.ProjectClassLoader.loadClass(ProjectClassLoader.java:111) - waiting to lock <0x00000000902d8d20> (a org.drools.core.common.ProjectClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.drools.core.base.ClassFieldAccessorCache.getClass(ClassFieldAccessorCache.java:121) at org.drools.core.base.ClassFieldAccessorCache.getClassObjectType(ClassFieldAccessorCache.java:45) at org.drools.core.reteoo.ClassObjectTypeConf.<init>(ClassObjectTypeConf.java:85) at org.drools.core.common.ObjectTypeConfigurationRegistry.getObjectTypeConf(ObjectTypeConfigurationRegistry.java:72) at org.drools.core.impl.StatefulKnowledgeSessionImpl.initInitialFact(StatefulKnowledgeSessionImpl.java:689) at org.drools.core.impl.StatefulKnowledgeSessionImpl.<init>(StatefulKnowledgeSessionImpl.java:388) at org.drools.core.impl.StatefulKnowledgeSessionImpl.<init>(StatefulKnowledgeSessionImpl.java:278) "tomcat-http--36": at java.lang.ClassLoader.loadClass(ClassLoader.java:404) - waiting to lock <0x000000009044e5b8> (a org.drools.core.common.ProjectClassLoader$InternalTypesClassLoader) at org.drools.core.common.ProjectClassLoader$InternalTypesClassLoader.loadType(ProjectClassLoader.java:293) at org.drools.core.common.ProjectClassLoader$InternalTypesClassLoader.access$200(ProjectClassLoader.java:258) at org.drools.core.common.ProjectClassLoader.loadType(ProjectClassLoader.java:140) at org.drools.core.common.ProjectClassLoader.loadClass(ProjectClassLoader.java:115) - locked <0x00000000902d8d20> (a org.drools.core.common.ProjectClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.drools.core.rule.JavaDialectRuntimeData$PackageClassLoader.loadClass(JavaDialectRuntimeData.java:640) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.mvel2.ParserConfiguration.checkForDynamicImport(ParserConfiguration.java:163) at org.mvel2.ParserConfiguration.hasImport(ParserConfiguration.java:191) at org.mvel2.ParserContext.hasImport(ParserContext.java:363) at org.mvel2.compiler.AbstractParser.createPropertyToken(AbstractParser.java:1342) at org.mvel2.compiler.AbstractParser.nextToken(AbstractParser.java:846) at org.mvel2.compiler.ExpressionCompiler._compile(ExpressionCompiler.java:129) at org.mvel2.compiler.ExpressionCompiler.compile(ExpressionCompiler.java:63) at org.mvel2.MVEL.compileExpression(MVEL.java:810) at org.drools.core.base.mvel.MVELCompilationUnit.compile(MVELCompilationUnit.java:429) at org.drools.core.base.mvel.MVELCompilationUnit.getCompiledExpression(MVELCompilationUnit.java:240) at org.drools.core.rule.constraint.MvelConstraint.createMvelConditionEvaluator(MvelConstraint.java:222) at org.drools.core.rule.constraint.MvelConstraint.evaluate(MvelConstraint.java:206) at org.drools.core.rule.constraint.MvelConstraint.isAllowed(MvelConstraint.java:173) at org.drools.core.reteoo.AlphaNode.assertObject(AlphaNode.java:134) at org.drools.core.reteoo.CompositeObjectSinkAdapter.doPropagateAssertObject(CompositeObjectSinkAdapter.java:502) at org.drools.core.reteoo.CompositeObjectSinkAdapter.propagateAssertObject(CompositeObjectSinkAdapter.java:377) at org.drools.core.reteoo.ObjectTypeNode.assertObject(ObjectTypeNode.java:288) at org.drools.core.reteoo.EntryPointNode.assertObject(EntryPointNode.java:251) at org.drools.core.common.NamedEntryPoint.insert(NamedEntryPoint.java:367) at org.drools.core.common.NamedEntryPoint.insert(NamedEntryPoint.java:286) at org.drools.core.common.NamedEntryPoint.insert(NamedEntryPoint.java:139) at org.drools.core.command.runtime.rule.InsertElementsCommand.execute(InsertElementsCommand.java:93) at org.drools.core.command.runtime.rule.InsertElementsCommand.execute(InsertElementsCommand.java:40) at org.drools.core.command.runtime.BatchExecutionCommandImpl.execute(BatchExecutionCommandImpl.java:155) at org.drools.core.command.runtime.BatchExecutionCommandImpl.execute(BatchExecutionCommandImpl.java:76) at org.drools.core.impl.StatelessKnowledgeSessionImpl.execute(StatelessKnowledgeSessionImpl.java:321) at com.amdocs.dc.ruleengine.service.RuleEngineServiceImpl.evaluateRules(RuleEngineServiceImpl.java:452) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.amdocs.dc.applicationcontext.BusinessResponseProcessorServiceInvoker.invoke(BusinessResponseProcessorServiceInvoker.java:53) at com.amdocs.dc.applicationcontext.spring.BusinessResponseProcessorInvocationHandler.invoke(BusinessResponseProcessorInvocationHandler.java:85) at com.amdocs.dc.applicationcontext.spring.$Proxy121.evaluateRules(Unknown Source) "Metrics Collection" #21 daemon prio=5 os_prio=0 tid=0x00007fae7883a000 nid=0x7276 waiting for monitor entry [0x00007fae27ffd000] java.lang.Thread.State: BLOCKED (on object monitor) at org.drools.core.common.ProjectClassLoader$InternalTypesClassLoader.loadClass(ProjectClassLoader.java:284) - waiting to lock <0x00000000902d8d20> (a org.drools.core.common.ProjectClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at com.mercury.diagnostics.capture.metrics.CollectorControl.instantiateCollector(CollectorControl.java:470) at com.mercury.diagnostics.capture.metrics.CollectorControl.initialize(CollectorControl.java:369) at com.mercury.diagnostics.capture.metrics.CollectorAgent.validateInitialization(CollectorAgent.java:1020) at com.mercury.diagnostics.capture.metrics.CollectorAgent.run(CollectorAgent.java:803) at java.lang.Thread.run(Thread.java:745) and because of above, other HTTP Connector thread are stuck at "tomcat-http--42" #217 daemon prio=5 os_prio=0 tid=0x00007fae54153000 nid=0x7619 waiting for monitor entry [0x00007fae12d72000] java.lang.Thread.State: BLOCKED (on object monitor) at org.drools.core.common.ProjectClassLoader.loadClass(ProjectClassLoader.java:111) - waiting to lock <0x00000000902e3e88> (a org.drools.core.common.ProjectClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.drools.core.base.ClassFieldAccessorCache.getClass(ClassFieldAccessorCache.java:121) at org.drools.core.base.ClassFieldAccessorCache.getClassObjectType(ClassFieldAccessorCache.java:45) at org.drools.core.reteoo.ClassObjectTypeConf.<init>(ClassObjectTypeConf.java:85) at org.drools.core.common.ObjectTypeConfigurationRegistry.getObjectTypeConf(ObjectTypeConfigurationRegistry.java:72) at org.drools.core.impl.StatefulKnowledgeSessionImpl.initInitialFact(StatefulKnowledgeSessionImpl.java:689) at org.drools.core.impl.StatefulKnowledgeSessionImpl.<init>(StatefulKnowledgeSessionImpl.java:388) at org.drools.core.impl.StatefulKnowledgeSessionImpl.<init>(StatefulKnowledgeSessionImpl.java:278) at org.drools.core.common.PhreakWorkingMemoryFactory.createWorkingMemory(PhreakWorkingMemoryFactory.java:21) at org.drools.core.impl.StatelessKnowledgeSessionImpl.newWorkingMemory(StatelessKnowledgeSessionImpl.java:133) at org.drools.core.impl.StatelessKnowledgeSessionImpl.execute(StatelessKnowledgeSessionImpl.java:309)