Thread deadlock in org.jboss.aop.asintegration.jboss5.ToClassInvoker.toClass() ------------------------------------------------------------------------------ Key: JBAS-8652 URL: https://issues.jboss.org/browse/JBAS-8652 Project: Application Server 3 4 5 and 6 Issue Type: Bug Security Level: Public(Everyone can see) Components: AOP Affects Versions: JBossAS-5.1.0.GA Environment: JBossAS 5.1.0.GA, JAVA 1.6.0_17, Sun Solaris Reporter: martin walla Fix For: No Release On startup of JBoss 5.1.0-GA sometimes (seen it several times) it comes to a Thread deadlock between the main thread and an other Thread. We use XA datasources and we have configured in the file conf/jbossts-properties.xml the Arjuna Transaction Manager Recovery Module. <properties depends="arjuna" name="jta"> <property name="com.arjuna.ats.jta.recovery.XAResourceRecoveryJDBC" value="com.arjuna.ats.internal.jbossatx.jta.AppServerJDBCXARecovery;jndiname=nxcore/resource/DS"/> This RecoveryModule is triggered during startup in a Thread which causes together with the "main" Thread a deadlock in the method toClass() of the class org.jboss.aop.asintegration.jboss5.ToClassInvoker. 1.) The "main" Thread has already locked the org.jboss.classloader.spi.base.BaseClassLoader and waits to lock the "tmplock" Object in class ToClassInvoker. 2.) The other Thread ("Thread-20") has already locked the "tmplock" Object and waits to lock the BaseClassLoader for the call of the synchronized method loadClass() (myloader.loadClass()). Source can be seen here: http://grepcode.com/file/repository.jboss.com/maven2/org.jboss.aop/jboss-... IMHO a locking of the BaseClassLoader "synchronized (myloader)" before the locking of the "tmplock" Object should prevent this deadlock situation. "main": at org.jboss.aop.asintegration.jboss5.ToClassInvoker.toClass(ToClassInvoker.java:72) - waiting to lock <0xffffffff64f63d00> (a java.lang.Object) at org.jboss.aop.classpool.jbosscl.JBossClDelegatingClassPool.toClass(JBossClDelegatingClassPool.java:81) at javassist.CtClass.toClass(CtClass.java:1094) at org.jboss.aop.instrument.TransformerCommon$ToClassAction$2.toClass(TransformerCommon.java:331) at org.jboss.aop.instrument.TransformerCommon.compileOrLoadClass(TransformerCommon.java:103) at org.jboss.aop.instrument.TransformerCommon.compileOrLoadClass(TransformerCommon.java:65) at org.jboss.aop.instrument.MethodJoinPointGenerator$BaseClassGenerator.generate(MethodJoinPointGenerator.java:239) at org.jboss.aop.instrument.MethodJoinPointGenerator.createJoinpointBaseClass(MethodJoinPointGenerator.java:172) at org.jboss.aop.instrument.GeneratedAdvisorMethodExecutionTransformer.createJoinpointClass(GeneratedAdvisorMethodExecutionTransformer.java:95) at org.jboss.aop.instrument.GeneratedAdvisorMethodExecutionTransformer.addJoinpoint(GeneratedAdvisorMethodExecutionTransformer.java:83) at org.jboss.aop.instrument.GeneratedAdvisorMethodExecutionTransformer.addMethodInfoFieldToGenAdvisor(GeneratedAdvisorMethodExecutionTransformer.java:58) at org.jboss.aop.instrument.GeneratedAdvisorMethodExecutionTransformer.transformMethod(GeneratedAdvisorMethodExecutionTransformer.java:304) at org.jboss.aop.instrument.MethodExecutionTransformer.instrument(MethodExecutionTransformer.java:146) at org.jboss.aop.instrument.Instrumentor.transform(Instrumentor.java:744) at org.jboss.aop.instrument.GeneratedAdvisorInstrumentor.transform(GeneratedAdvisorInstrumentor.java:119) at org.jboss.aop.SuperClassesFirstWeavingStrategy.instrumentClass(SuperClassesFirstWeavingStrategy.java:202) at org.jboss.aop.SuperClassesFirstWeavingStrategy.translate(SuperClassesFirstWeavingStrategy.java:69) at org.jboss.aop.AspectManager.translate(AspectManager.java:1071) at org.jboss.aop.AspectManager.transform(AspectManager.java:1015) at org.jboss.aop.standalone.AOPTransformer.aspectTransform(AOPTransformer.java:87) at org.jboss.aop.standalone.AOPTransformer.transform(AOPTransformer.java:75) at sun.instrument.TransformerManager.transform(TransformerManager.java:169) at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:365) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:616) at org.jboss.classloader.spi.base.BaseClassLoader.access$200(BaseClassLoader.java:63) at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:572) at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:532) at java.security.AccessController.doPrivileged(Native Method) at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:530) - locked <0xffffffff60aba990> (a org.jboss.classloader.spi.base.BaseClassLoader) at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:507) at org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134) at org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131) at org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452) at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:251) at org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:150) at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:265) at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1119) at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:798) at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:441) - locked <0xffffffff60aba990> (a org.jboss.classloader.spi.base.BaseClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316) - locked <0xffffffff60aba990> (a org.jboss.classloader.spi.base.BaseClassLoader) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:616) at org.jboss.classloader.spi.base.BaseClassLoader.access$200(BaseClassLoader.java:63) at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:572) at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:532) at java.security.AccessController.doPrivileged(Native Method) at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:530) - locked <0xffffffff60aba990> (a org.jboss.classloader.spi.base.BaseClassLoader) at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:507) at org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134) at org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131) at org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452) at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:251) at org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:150) at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:265) at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1119) at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:798) at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:441) - locked <0xffffffff65eda3b0> (a org.jboss.classloader.spi.base.BaseClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316) - locked <0xffffffff65eda3b0> (a org.jboss.classloader.spi.base.BaseClassLoader) at org.jboss.resource.deployers.builder.ConnectionManagerBuilder.getCode(ConnectionManagerBuilder.java:245) at org.jboss.resource.deployers.builder.AbstractBuilder.buildService(AbstractBuilder.java:54) at org.jboss.resource.deployers.builder.AbstractBuilder.build(AbstractBuilder.java:65) at org.jboss.resource.deployers.ManagedConnectionFactoryDeployer.deploy(ManagedConnectionFactoryDeployer.java:177) at org.jboss.resource.deployers.ManagedConnectionFactoryDeployer.deploy(ManagedConnectionFactoryDeployer.java:52) at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:62) at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:50) at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:171) at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1439) at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1157) at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1098) at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1631) at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:934) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984) at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:822) at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553) at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:781) at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:702) at org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process(MainDeployerAdapter.java:117) at org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(ProfileDeployAction.java:70) at org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(AbstractProfileAction.java:53) at org.jboss.system.server.profileservice.repository.AbstractProfileService.install(AbstractProfileService.java:361) at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1631) at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:934) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984) at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:822) at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553) at org.jboss.system.server.profileservice.repository.AbstractProfileService.activateProfile(AbstractProfileService.java:306) at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:271) at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:461) at org.jboss.Main.boot(Main.java:221) at org.jboss.Main$1.run(Main.java:556) at java.lang.Thread.run(Thread.java:619) "Thread-20": at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:437) - waiting to lock <0xffffffff60aba990> (a org.jboss.classloader.spi.base.BaseClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at org.jboss.aop.asintegration.jboss5.ToClassInvoker.toClass(ToClassInvoker.java:88) - locked <0xffffffff64f63d00> (a java.lang.Object) at org.jboss.aop.classpool.jbosscl.JBossClDelegatingClassPool.toClass(JBossClDelegatingClassPool.java:81) at javassist.CtClass.toClass(CtClass.java:1094) at org.jboss.aop.instrument.TransformerCommon$ToClassAction$2.toClass(TransformerCommon.java:331) at org.jboss.aop.instrument.TransformerCommon.toClass(TransformerCommon.java:139) at org.jboss.aop.instrument.JoinPointGenerator.toClass(JoinPointGenerator.java:315) at org.jboss.aop.instrument.JoinPointGenerator.generateJoinpointClass(JoinPointGenerator.java:380) at org.jboss.aop.instrument.JoinPointGenerator.doGenerateJoinPointClass(JoinPointGenerator.java:284) at org.jboss.aop.instrument.JoinPointGenerator.access$300(JoinPointGenerator.java:76) at org.jboss.aop.instrument.JoinPointGenerator$GenerateJoinPointClassAction$2.generateJoinPointClass(JoinPointGenerator.java:1729) at org.jboss.aop.instrument.JoinPointGenerator.generateJoinPointClass(JoinPointGenerator.java:249) - locked <0xfffffffe7d0d0538> (a org.jboss.aop.instrument.MethodJoinPointGenerator) at org.jboss.aop.GeneratedClassAdvisor.generateJoinPointClass(GeneratedClassAdvisor.java:1044) at org.jboss.jmx.connector.invoker.AuthenticationInterceptor$AuthenticationInterceptorAdvisor.invoke_N_5788773513025575775(AuthenticationInterceptor$AuthenticationInterceptorAdvisor.java) at org.jboss.jmx.connector.invoker.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java) at org.jboss.mx.server.Invocation.invoke(Invocation.java:90) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668) at org.jboss.invocation.jrmp.server.JRMPProxyFactory.invoke(JRMPProxyFactory.java:180) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668) at org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:169) at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:118) at org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:209) at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:195) at org.jboss.jmx.connector.invoker.client.InvokerAdaptorClientInterceptor.invoke(InvokerAdaptorClientInterceptor.java:66) at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:68) at org.jboss.proxy.ClientMethodInterceptor.invoke(ClientMethodInterceptor.java:74) at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:101) at $Proxy247.invoke(Unknown Source) at com.arjuna.ats.internal.jbossatx.jta.AppServerJDBCXARecovery.createDataSource(AppServerJDBCXARecovery.java:189) at com.arjuna.ats.internal.jbossatx.jta.AppServerJDBCXARecovery.hasMoreResources(AppServerJDBCXARecovery.java:144) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.resourceInitiatedRecovery(XARecoveryModule.java:659) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:200) at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:799) at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:412) Found 1 deadlock.