[
https://issues.jboss.org/browse/JBAS-8652?page=com.atlassian.jira.plugin....
]
niroshnk commented on JBAS-8652:
--------------------------------
Hi,
I also experienced a similar issue when there are lot of concurrent requests coming on.
So I tried to find a fix for this and came up with following fix
Instead of putting the synchronized (myloader) just before the synchronized (tmplock),
what I did is put the synchronized block only to the piece of code
“factory.putFile(outputURL, classBytes)” as follows.
------------------------------------------------
if (trace) logger.trace(this + " " + pool + ".toClass() myloader:" +
myloader +" writing bytes to " + tempURL);
ByteArrayOutputStream byteout = new ByteArrayOutputStream();
BufferedOutputStream out = new BufferedOutputStream(byteout);
out.write(cc.toBytecode());
out.flush();
out.close();
byte[] classBytes = byteout.toByteArray();
MemoryContextFactory factory = MemoryContextFactory.getInstance();
synchronized (tmplock) {
factory.putFile(outputURL, classBytes);
}
if (myloader instanceof RealClassLoader) {
((RealClassLoader) myloader).clearBlackList(classFileName);
}
Class<?> clazz = myloader.loadClass(cc.getName());
if (trace) logger.trace(this + " " + pool + " myloader:" + myloader +
" created class:" + clazz);
return clazz;
------------------------------------------------
The thinking behind this change is
1. By defaults maps are not synchronized. So this line need to be a synchronized one.
2. Adding another sync block makes even more vulnerable to thread blocking.
3. BufferedOutputStream class write method is synchronized therefore BufferedOutputStream
write method and relevant piece of code doesn’t need to be in a another sync block.
4. What I am really doubtful is do we need the myloader.loadClass within the
synchronized block. If we keep that then the error is still there. If so we have to use
double synchronized as suggested in the previous comment.
synchronized (myloader) {
synchronized (tmplock) {
After my fix the application works fine without any error. But I may be wrong in the
thinking and this class is heavily used in the JVM.
So feedback on this change are highly appreciated.
Thanks
Nirosh
Thread deadlock in
org.jboss.aop.asintegration.jboss5.ToClassInvoker.toClass()
------------------------------------------------------------------------------
Key: JBAS-8652
URL:
https://issues.jboss.org/browse/JBAS-8652
Project: JBoss Application Server
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: AOP
Affects Versions: JBossAS-5.1.0.GA
Environment: JBossAS 5.1.0.GA, JAVA 1.6.0_17, Sun Solaris
Reporter: martin walla
Assignee: Kabir Khan
On startup of JBoss 5.1.0-GA sometimes (seen it several times) it comes to a Thread
deadlock between the main thread and an other Thread.
We use XA datasources and we have configured in the file conf/jbossts-properties.xml the
Arjuna Transaction Manager Recovery Module.
<properties depends="arjuna" name="jta">
<property name="com.arjuna.ats.jta.recovery.XAResourceRecoveryJDBC"
value="com.arjuna.ats.internal.jbossatx.jta.AppServerJDBCXARecovery;jndiname=nxcore/resource/DS"/>
This RecoveryModule is triggered during startup in a Thread which causes together with
the "main" Thread a deadlock
in the method toClass() of the class org.jboss.aop.asintegration.jboss5.ToClassInvoker.
1.) The "main" Thread has already locked the
org.jboss.classloader.spi.base.BaseClassLoader and waits to lock the "tmplock"
Object in class ToClassInvoker.
2.) The other Thread ("Thread-20") has already locked the "tmplock"
Object and waits to lock the BaseClassLoader for the call
of the synchronized method loadClass() (myloader.loadClass()).
Source can be seen here:
http://grepcode.com/file/repository.jboss.com/maven2/org.jboss.aop/jboss-...
IMHO a locking of the BaseClassLoader "synchronized (myloader)" before the
locking of the "tmplock" Object should prevent this deadlock situation.
"main":
at org.jboss.aop.asintegration.jboss5.ToClassInvoker.toClass(ToClassInvoker.java:72)
- waiting to lock <0xffffffff64f63d00> (a java.lang.Object)
at
org.jboss.aop.classpool.jbosscl.JBossClDelegatingClassPool.toClass(JBossClDelegatingClassPool.java:81)
at javassist.CtClass.toClass(CtClass.java:1094)
at
org.jboss.aop.instrument.TransformerCommon$ToClassAction$2.toClass(TransformerCommon.java:331)
at
org.jboss.aop.instrument.TransformerCommon.compileOrLoadClass(TransformerCommon.java:103)
at
org.jboss.aop.instrument.TransformerCommon.compileOrLoadClass(TransformerCommon.java:65)
at
org.jboss.aop.instrument.MethodJoinPointGenerator$BaseClassGenerator.generate(MethodJoinPointGenerator.java:239)
at
org.jboss.aop.instrument.MethodJoinPointGenerator.createJoinpointBaseClass(MethodJoinPointGenerator.java:172)
at
org.jboss.aop.instrument.GeneratedAdvisorMethodExecutionTransformer.createJoinpointClass(GeneratedAdvisorMethodExecutionTransformer.java:95)
at
org.jboss.aop.instrument.GeneratedAdvisorMethodExecutionTransformer.addJoinpoint(GeneratedAdvisorMethodExecutionTransformer.java:83)
at
org.jboss.aop.instrument.GeneratedAdvisorMethodExecutionTransformer.addMethodInfoFieldToGenAdvisor(GeneratedAdvisorMethodExecutionTransformer.java:58)
at
org.jboss.aop.instrument.GeneratedAdvisorMethodExecutionTransformer.transformMethod(GeneratedAdvisorMethodExecutionTransformer.java:304)
at
org.jboss.aop.instrument.MethodExecutionTransformer.instrument(MethodExecutionTransformer.java:146)
at org.jboss.aop.instrument.Instrumentor.transform(Instrumentor.java:744)
at
org.jboss.aop.instrument.GeneratedAdvisorInstrumentor.transform(GeneratedAdvisorInstrumentor.java:119)
at
org.jboss.aop.SuperClassesFirstWeavingStrategy.instrumentClass(SuperClassesFirstWeavingStrategy.java:202)
at
org.jboss.aop.SuperClassesFirstWeavingStrategy.translate(SuperClassesFirstWeavingStrategy.java:69)
at org.jboss.aop.AspectManager.translate(AspectManager.java:1071)
at org.jboss.aop.AspectManager.transform(AspectManager.java:1015)
at org.jboss.aop.standalone.AOPTransformer.aspectTransform(AOPTransformer.java:87)
at org.jboss.aop.standalone.AOPTransformer.transform(AOPTransformer.java:75)
at sun.instrument.TransformerManager.transform(TransformerManager.java:169)
at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:365)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
at org.jboss.classloader.spi.base.BaseClassLoader.access$200(BaseClassLoader.java:63)
at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:572)
at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:532)
at java.security.AccessController.doPrivileged(Native Method)
at
org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:530)
- locked <0xffffffff60aba990> (a org.jboss.classloader.spi.base.BaseClassLoader)
at
org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:507)
at
org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
at
org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131)
at
org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452)
at
org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:251)
at
org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:150)
at
org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:265)
at
org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1119)
at
org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:798)
at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:441)
- locked <0xffffffff60aba990> (a org.jboss.classloader.spi.base.BaseClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
- locked <0xffffffff60aba990> (a org.jboss.classloader.spi.base.BaseClassLoader)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
at org.jboss.classloader.spi.base.BaseClassLoader.access$200(BaseClassLoader.java:63)
at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:572)
at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:532)
at java.security.AccessController.doPrivileged(Native Method)
at
org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:530)
- locked <0xffffffff60aba990> (a org.jboss.classloader.spi.base.BaseClassLoader)
at
org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:507)
at
org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
at
org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131)
at
org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452)
at
org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:251)
at
org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:150)
at
org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:265)
at
org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1119)
at
org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:798)
at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:441)
- locked <0xffffffff65eda3b0> (a org.jboss.classloader.spi.base.BaseClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
- locked <0xffffffff65eda3b0> (a org.jboss.classloader.spi.base.BaseClassLoader)
at
org.jboss.resource.deployers.builder.ConnectionManagerBuilder.getCode(ConnectionManagerBuilder.java:245)
at
org.jboss.resource.deployers.builder.AbstractBuilder.buildService(AbstractBuilder.java:54)
at org.jboss.resource.deployers.builder.AbstractBuilder.build(AbstractBuilder.java:65)
at
org.jboss.resource.deployers.ManagedConnectionFactoryDeployer.deploy(ManagedConnectionFactoryDeployer.java:177)
at
org.jboss.resource.deployers.ManagedConnectionFactoryDeployer.deploy(ManagedConnectionFactoryDeployer.java:52)
at
org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:62)
at
org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:50)
at
org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:171)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1439)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1157)
at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1098)
at
org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
at
org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1631)
at
org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:934)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984)
at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:822)
at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553)
at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:781)
at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:702)
at
org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process(MainDeployerAdapter.java:117)
at
org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(ProfileDeployAction.java:70)
at
org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(AbstractProfileAction.java:53)
at
org.jboss.system.server.profileservice.repository.AbstractProfileService.install(AbstractProfileService.java:361)
at
org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
at
org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1631)
at
org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:934)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984)
at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:822)
at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553)
at
org.jboss.system.server.profileservice.repository.AbstractProfileService.activateProfile(AbstractProfileService.java:306)
at
org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:271)
at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:461)
at org.jboss.Main.boot(Main.java:221)
at org.jboss.Main$1.run(Main.java:556)
at java.lang.Thread.run(Thread.java:619)
"Thread-20":
at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:437)
- waiting to lock <0xffffffff60aba990> (a
org.jboss.classloader.spi.base.BaseClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at org.jboss.aop.asintegration.jboss5.ToClassInvoker.toClass(ToClassInvoker.java:88)
- locked <0xffffffff64f63d00> (a java.lang.Object)
at
org.jboss.aop.classpool.jbosscl.JBossClDelegatingClassPool.toClass(JBossClDelegatingClassPool.java:81)
at javassist.CtClass.toClass(CtClass.java:1094)
at
org.jboss.aop.instrument.TransformerCommon$ToClassAction$2.toClass(TransformerCommon.java:331)
at org.jboss.aop.instrument.TransformerCommon.toClass(TransformerCommon.java:139)
at org.jboss.aop.instrument.JoinPointGenerator.toClass(JoinPointGenerator.java:315)
at
org.jboss.aop.instrument.JoinPointGenerator.generateJoinpointClass(JoinPointGenerator.java:380)
at
org.jboss.aop.instrument.JoinPointGenerator.doGenerateJoinPointClass(JoinPointGenerator.java:284)
at org.jboss.aop.instrument.JoinPointGenerator.access$300(JoinPointGenerator.java:76)
at
org.jboss.aop.instrument.JoinPointGenerator$GenerateJoinPointClassAction$2.generateJoinPointClass(JoinPointGenerator.java:1729)
at
org.jboss.aop.instrument.JoinPointGenerator.generateJoinPointClass(JoinPointGenerator.java:249)
- locked <0xfffffffe7d0d0538> (a
org.jboss.aop.instrument.MethodJoinPointGenerator)
at
org.jboss.aop.GeneratedClassAdvisor.generateJoinPointClass(GeneratedClassAdvisor.java:1044)
at
org.jboss.jmx.connector.invoker.AuthenticationInterceptor$AuthenticationInterceptorAdvisor.invoke_N_5788773513025575775(AuthenticationInterceptor$AuthenticationInterceptorAdvisor.java)
at
org.jboss.jmx.connector.invoker.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
at org.jboss.invocation.jrmp.server.JRMPProxyFactory.invoke(JRMPProxyFactory.java:180)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
at
org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:169)
at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:118)
at org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:209)
at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:195)
at
org.jboss.jmx.connector.invoker.client.InvokerAdaptorClientInterceptor.invoke(InvokerAdaptorClientInterceptor.java:66)
at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:68)
at org.jboss.proxy.ClientMethodInterceptor.invoke(ClientMethodInterceptor.java:74)
at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:101)
at $Proxy247.invoke(Unknown Source)
at
com.arjuna.ats.internal.jbossatx.jta.AppServerJDBCXARecovery.createDataSource(AppServerJDBCXARecovery.java:189)
at
com.arjuna.ats.internal.jbossatx.jta.AppServerJDBCXARecovery.hasMoreResources(AppServerJDBCXARecovery.java:144)
at
com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.resourceInitiatedRecovery(XARecoveryModule.java:659)
at
com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:200)
at
com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:799)
at
com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:412)
Found 1 deadlock.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira