[jboss-jira] [JBoss JIRA] Commented: (JBAS-8652) Thread deadlock in org.jboss.aop.asintegration.jboss5.ToClassInvoker.toClass()
niroshnk (JIRA)
jira-events at lists.jboss.org
Wed Jan 19 21:50:49 EST 2011
[ https://issues.jboss.org/browse/JBAS-8652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12576677#comment-12576677 ]
niroshnk commented on JBAS-8652:
--------------------------------
Hi,
I also experienced a similar issue when there are lot of concurrent requests coming on.
So I tried to find a fix for this and came up with following fix
Instead of putting the synchronized (myloader) just before the synchronized (tmplock), what I did is put the synchronized block only to the piece of code “factory.putFile(outputURL, classBytes)” as follows.
------------------------------------------------
if (trace) logger.trace(this + " " + pool + ".toClass() myloader:" + myloader +" writing bytes to " + tempURL);
ByteArrayOutputStream byteout = new ByteArrayOutputStream();
BufferedOutputStream out = new BufferedOutputStream(byteout);
out.write(cc.toBytecode());
out.flush();
out.close();
byte[] classBytes = byteout.toByteArray();
MemoryContextFactory factory = MemoryContextFactory.getInstance();
synchronized (tmplock) {
factory.putFile(outputURL, classBytes);
}
if (myloader instanceof RealClassLoader) {
((RealClassLoader) myloader).clearBlackList(classFileName);
}
Class<?> clazz = myloader.loadClass(cc.getName());
if (trace) logger.trace(this + " " + pool + " myloader:" + myloader + " created class:" + clazz);
return clazz;
------------------------------------------------
The thinking behind this change is
1. By defaults maps are not synchronized. So this line need to be a synchronized one.
2. Adding another sync block makes even more vulnerable to thread blocking.
3. BufferedOutputStream class write method is synchronized therefore BufferedOutputStream write method and relevant piece of code doesn’t need to be in a another sync block.
4. What I am really doubtful is do we need the myloader.loadClass within the synchronized block. If we keep that then the error is still there. If so we have to use double synchronized as suggested in the previous comment.
synchronized (myloader) {
synchronized (tmplock) {
After my fix the application works fine without any error. But I may be wrong in the thinking and this class is heavily used in the JVM.
So feedback on this change are highly appreciated.
Thanks
Nirosh
> Thread deadlock in org.jboss.aop.asintegration.jboss5.ToClassInvoker.toClass()
> ------------------------------------------------------------------------------
>
> Key: JBAS-8652
> URL: https://issues.jboss.org/browse/JBAS-8652
> Project: JBoss Application Server
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Components: AOP
> Affects Versions: JBossAS-5.1.0.GA
> Environment: JBossAS 5.1.0.GA, JAVA 1.6.0_17, Sun Solaris
> Reporter: martin walla
> Assignee: Kabir Khan
>
> On startup of JBoss 5.1.0-GA sometimes (seen it several times) it comes to a Thread deadlock between the main thread and an other Thread.
> We use XA datasources and we have configured in the file conf/jbossts-properties.xml the Arjuna Transaction Manager Recovery Module.
> <properties depends="arjuna" name="jta">
> <property name="com.arjuna.ats.jta.recovery.XAResourceRecoveryJDBC" value="com.arjuna.ats.internal.jbossatx.jta.AppServerJDBCXARecovery;jndiname=nxcore/resource/DS"/>
> This RecoveryModule is triggered during startup in a Thread which causes together with the "main" Thread a deadlock
> in the method toClass() of the class org.jboss.aop.asintegration.jboss5.ToClassInvoker.
> 1.) The "main" Thread has already locked the org.jboss.classloader.spi.base.BaseClassLoader and waits to lock the "tmplock" Object in class ToClassInvoker.
> 2.) The other Thread ("Thread-20") has already locked the "tmplock" Object and waits to lock the BaseClassLoader for the call
> of the synchronized method loadClass() (myloader.loadClass()).
> Source can be seen here:
> http://grepcode.com/file/repository.jboss.com/maven2/org.jboss.aop/jboss-aop-asintegration-mc/2.1.1.GA/org/jboss/aop/asintegration/jboss5/ToClassInvoker.java#ToClassInvoker
> IMHO a locking of the BaseClassLoader "synchronized (myloader)" before the locking of the "tmplock" Object should prevent this deadlock situation.
> "main":
> at org.jboss.aop.asintegration.jboss5.ToClassInvoker.toClass(ToClassInvoker.java:72)
> - waiting to lock <0xffffffff64f63d00> (a java.lang.Object)
> at org.jboss.aop.classpool.jbosscl.JBossClDelegatingClassPool.toClass(JBossClDelegatingClassPool.java:81)
> at javassist.CtClass.toClass(CtClass.java:1094)
> at org.jboss.aop.instrument.TransformerCommon$ToClassAction$2.toClass(TransformerCommon.java:331)
> at org.jboss.aop.instrument.TransformerCommon.compileOrLoadClass(TransformerCommon.java:103)
> at org.jboss.aop.instrument.TransformerCommon.compileOrLoadClass(TransformerCommon.java:65)
> at org.jboss.aop.instrument.MethodJoinPointGenerator$BaseClassGenerator.generate(MethodJoinPointGenerator.java:239)
> at org.jboss.aop.instrument.MethodJoinPointGenerator.createJoinpointBaseClass(MethodJoinPointGenerator.java:172)
> at org.jboss.aop.instrument.GeneratedAdvisorMethodExecutionTransformer.createJoinpointClass(GeneratedAdvisorMethodExecutionTransformer.java:95)
> at org.jboss.aop.instrument.GeneratedAdvisorMethodExecutionTransformer.addJoinpoint(GeneratedAdvisorMethodExecutionTransformer.java:83)
> at org.jboss.aop.instrument.GeneratedAdvisorMethodExecutionTransformer.addMethodInfoFieldToGenAdvisor(GeneratedAdvisorMethodExecutionTransformer.java:58)
> at org.jboss.aop.instrument.GeneratedAdvisorMethodExecutionTransformer.transformMethod(GeneratedAdvisorMethodExecutionTransformer.java:304)
> at org.jboss.aop.instrument.MethodExecutionTransformer.instrument(MethodExecutionTransformer.java:146)
> at org.jboss.aop.instrument.Instrumentor.transform(Instrumentor.java:744)
> at org.jboss.aop.instrument.GeneratedAdvisorInstrumentor.transform(GeneratedAdvisorInstrumentor.java:119)
> at org.jboss.aop.SuperClassesFirstWeavingStrategy.instrumentClass(SuperClassesFirstWeavingStrategy.java:202)
> at org.jboss.aop.SuperClassesFirstWeavingStrategy.translate(SuperClassesFirstWeavingStrategy.java:69)
> at org.jboss.aop.AspectManager.translate(AspectManager.java:1071)
> at org.jboss.aop.AspectManager.transform(AspectManager.java:1015)
> at org.jboss.aop.standalone.AOPTransformer.aspectTransform(AOPTransformer.java:87)
> at org.jboss.aop.standalone.AOPTransformer.transform(AOPTransformer.java:75)
> at sun.instrument.TransformerManager.transform(TransformerManager.java:169)
> at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:365)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
> at org.jboss.classloader.spi.base.BaseClassLoader.access$200(BaseClassLoader.java:63)
> at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:572)
> at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:532)
> at java.security.AccessController.doPrivileged(Native Method)
> at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:530)
> - locked <0xffffffff60aba990> (a org.jboss.classloader.spi.base.BaseClassLoader)
> at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:507)
> at org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
> at org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131)
> at org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452)
> at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:251)
> at org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:150)
> at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:265)
> at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1119)
> at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:798)
> at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:441)
> - locked <0xffffffff60aba990> (a org.jboss.classloader.spi.base.BaseClassLoader)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
> - locked <0xffffffff60aba990> (a org.jboss.classloader.spi.base.BaseClassLoader)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
> at org.jboss.classloader.spi.base.BaseClassLoader.access$200(BaseClassLoader.java:63)
> at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:572)
> at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:532)
> at java.security.AccessController.doPrivileged(Native Method)
> at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:530)
> - locked <0xffffffff60aba990> (a org.jboss.classloader.spi.base.BaseClassLoader)
> at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:507)
> at org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
> at org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131)
> at org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452)
> at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:251)
> at org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:150)
> at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:265)
> at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1119)
> at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:798)
> at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:441)
> - locked <0xffffffff65eda3b0> (a org.jboss.classloader.spi.base.BaseClassLoader)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
> - locked <0xffffffff65eda3b0> (a org.jboss.classloader.spi.base.BaseClassLoader)
> at org.jboss.resource.deployers.builder.ConnectionManagerBuilder.getCode(ConnectionManagerBuilder.java:245)
> at org.jboss.resource.deployers.builder.AbstractBuilder.buildService(AbstractBuilder.java:54)
> at org.jboss.resource.deployers.builder.AbstractBuilder.build(AbstractBuilder.java:65)
> at org.jboss.resource.deployers.ManagedConnectionFactoryDeployer.deploy(ManagedConnectionFactoryDeployer.java:177)
> at org.jboss.resource.deployers.ManagedConnectionFactoryDeployer.deploy(ManagedConnectionFactoryDeployer.java:52)
> at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:62)
> at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:50)
> at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:171)
> at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1439)
> at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1157)
> at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1098)
> at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
> at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1631)
> at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:934)
> at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)
> at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984)
> at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:822)
> at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553)
> at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:781)
> at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:702)
> at org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process(MainDeployerAdapter.java:117)
> at org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(ProfileDeployAction.java:70)
> at org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(AbstractProfileAction.java:53)
> at org.jboss.system.server.profileservice.repository.AbstractProfileService.install(AbstractProfileService.java:361)
> at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
> at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1631)
> at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:934)
> at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)
> at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984)
> at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:822)
> at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553)
> at org.jboss.system.server.profileservice.repository.AbstractProfileService.activateProfile(AbstractProfileService.java:306)
> at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:271)
> at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:461)
> at org.jboss.Main.boot(Main.java:221)
> at org.jboss.Main$1.run(Main.java:556)
> at java.lang.Thread.run(Thread.java:619)
> "Thread-20":
> at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:437)
> - waiting to lock <0xffffffff60aba990> (a org.jboss.classloader.spi.base.BaseClassLoader)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> at org.jboss.aop.asintegration.jboss5.ToClassInvoker.toClass(ToClassInvoker.java:88)
> - locked <0xffffffff64f63d00> (a java.lang.Object)
> at org.jboss.aop.classpool.jbosscl.JBossClDelegatingClassPool.toClass(JBossClDelegatingClassPool.java:81)
> at javassist.CtClass.toClass(CtClass.java:1094)
> at org.jboss.aop.instrument.TransformerCommon$ToClassAction$2.toClass(TransformerCommon.java:331)
> at org.jboss.aop.instrument.TransformerCommon.toClass(TransformerCommon.java:139)
> at org.jboss.aop.instrument.JoinPointGenerator.toClass(JoinPointGenerator.java:315)
> at org.jboss.aop.instrument.JoinPointGenerator.generateJoinpointClass(JoinPointGenerator.java:380)
> at org.jboss.aop.instrument.JoinPointGenerator.doGenerateJoinPointClass(JoinPointGenerator.java:284)
> at org.jboss.aop.instrument.JoinPointGenerator.access$300(JoinPointGenerator.java:76)
> at org.jboss.aop.instrument.JoinPointGenerator$GenerateJoinPointClassAction$2.generateJoinPointClass(JoinPointGenerator.java:1729)
> at org.jboss.aop.instrument.JoinPointGenerator.generateJoinPointClass(JoinPointGenerator.java:249)
> - locked <0xfffffffe7d0d0538> (a org.jboss.aop.instrument.MethodJoinPointGenerator)
> at org.jboss.aop.GeneratedClassAdvisor.generateJoinPointClass(GeneratedClassAdvisor.java:1044)
> at org.jboss.jmx.connector.invoker.AuthenticationInterceptor$AuthenticationInterceptorAdvisor.invoke_N_5788773513025575775(AuthenticationInterceptor$AuthenticationInterceptorAdvisor.java)
> at org.jboss.jmx.connector.invoker.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java)
> at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
> at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
> at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
> at org.jboss.invocation.jrmp.server.JRMPProxyFactory.invoke(JRMPProxyFactory.java:180)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
> at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
> at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
> at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
> at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
> at org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:169)
> at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:118)
> at org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:209)
> at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:195)
> at org.jboss.jmx.connector.invoker.client.InvokerAdaptorClientInterceptor.invoke(InvokerAdaptorClientInterceptor.java:66)
> at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:68)
> at org.jboss.proxy.ClientMethodInterceptor.invoke(ClientMethodInterceptor.java:74)
> at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:101)
> at $Proxy247.invoke(Unknown Source)
> at com.arjuna.ats.internal.jbossatx.jta.AppServerJDBCXARecovery.createDataSource(AppServerJDBCXARecovery.java:189)
> at com.arjuna.ats.internal.jbossatx.jta.AppServerJDBCXARecovery.hasMoreResources(AppServerJDBCXARecovery.java:144)
> at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.resourceInitiatedRecovery(XARecoveryModule.java:659)
> at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:200)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:799)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:412)
> Found 1 deadlock.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
More information about the jboss-jira
mailing list