[jboss-jira] [JBoss JIRA] (JBMESSAGING-1941) Messaging deadlocks on AspectManager
Aaron Ogburn (JIRA)
jira-events at lists.jboss.org
Wed Feb 6 17:34:51 EST 2013
[ https://issues.jboss.org/browse/JBMESSAGING-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Ogburn updated JBMESSAGING-1941:
--------------------------------------
Workaround Description: -Try downgrading the JDK to 1.6_u22 or earlier (was: -Downgrade the JDK)
> Messaging deadlocks on AspectManager
> ------------------------------------
>
> Key: JBMESSAGING-1941
> URL: https://issues.jboss.org/browse/JBMESSAGING-1941
> Project: JBoss Messaging
> Issue Type: Bug
> Affects Versions: 1.4.8.SP9
> Environment: -JBoss Enterprise Application Platform (EAP) 5
> Reporter: Aaron Ogburn
> Attachments: 00779442testpatch.diff
>
>
> Messaging can stall JBoss at start up. The hang occurs as the main thread blocks in different locations in JMS trying to acquire the AspectManager instance lock:
> "main" prio=10 tid=0x00007fe69c1e1000 nid=0x3613 waiting for monitor entry [0x00007fe6828e5000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.jboss.aop.AspectManager.getAdvisor(AspectManager.java:728)
> - waiting to lock <0x00000000a5e43e50> (a org.jboss.aop.AspectManager)
> at org.jboss.jms.server.endpoint.advised.SessionAdvised.<clinit>(SessionAdvised.java)
> at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.createSessionDelegate(ServerConnectionEndpoint.java:273)
> - locked <0x00000000a5600f48> (a org.jboss.aop.asintegration.jboss5.ScopedVFSClassLoaderDomain)
> at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.createSessionDelegate(ServerConnectionEndpoint.java:229)
> Since the main thread is stalled, so too is start up. This lock is always held by another making a similar call like so:
> "WorkManager(2)-1" daemon prio=10 tid=0x00007fe62dd96000 nid=0x363e in
> Object.wait() [0x00007fe68029a000]
> java.lang.Thread.State: RUNNABLE
> at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.createSessionDelegate(ServerConnectionEndpoint.java:273)
> - locked <0x00000000a5e43e50> (a org.jboss.aop.AspectManager)
> at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.createSessionDelegate(ServerConnectionEndpoint.java:229)
> The code there at line 273 is simple enough and shouldn't cause a long stall:
> synchronized (AspectManager.instance())
> {
> advised = new SessionAdvised(ep); // line 273
> }
> Nonetheless, it does clearly hang at this line as WorkManager(2)-1 never progresses. There's no GC/CPU issues or anything of that nature, but we get a slight in the thread dump as WorkManager(2)-1 is shown to be in Object.wait(). Checking the thread through pstack, we can see the issue. It is waiting in native/JDK level operations required to initialize the SessionAdvised class:
> Thread 2 (Thread 0x7fe68029d700 (LWP 13886)):
> #0 0x0000003b3e40b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007fe6a12d447e in os:latformEvent:ark() () from /opt/app/jvm/jdk1.6.0_38/jre/lib/amd64/server/libjvm.so
> #2 0x00007fe6a12c54fa in ObjectMonitor::wait(long, bool, Thread*) () from /opt/app/jvm/jdk1.6.0_38/jre/lib/amd64/server/libjvm.so
> #3 0x00007fe6a13ab5f0 in ObjectSynchronizer::waitUninterruptibly(Handle, long, Thread*) () from /opt/app/jvm/jdk1.6.0_38/jre/lib/amd64/server/libjvm.so
> #4 0x00007fe6a107115b in instanceKlass::initialize_impl(instanceKlassHandle, Thread*) () from /opt/app/jvm/jdk1.6.0_38/jre/lib/amd64/server/libjvm.so
> #5 0x00007fe6a107077a in instanceKlass::initialize(Thread*) () from /opt/app/jvm/jdk1.6.0_38/jre/lib/amd64/server/libjvm.so
> #6 0x00007fe6a1097c5c in InterpreterRuntime::_new(JavaThread*, constantPoolOopDesc*, int) () from /opt/app/jvm/jdk1.6.0_38/jre/lib/amd64/server/libjvm.so
> These calls are naturally synchronized internally in the JVM so only one thread can proceed through initialization of a class at a given time. Checking for any threads in process of initializing SessionAdvised, we can see only the main thread is. But the main thread is still blocked for the AspectManager lock.
> So we have a deadlock here. The main thread holds the initialization lock while waiting for the AspectManager lock, while the WorkManager(2)-1 thread holds the AspectManager lock while waiting for the initialization lock. But really that shouldn't even be able to occur occur. The initialization of SessionAdvised is wrapped in the synchronized block so the main thread should not have been able to proceed into the initialization at all without already possessing the AspectManager lock. But for some reason it does proceed without the lock and then tries to grab it in the middle of executing that synchronized block to allow for the deadlock.
> It's been noted that this occurs on the latest JDK updates but was resolved downgrading to u21. 1.6u23 upgraded to HotSpot 19 (http://www.oracle.com/technetwork/java/javase/6u23releasenotes-191058.html), which would have brought in a whole bunch of lock optimisation changes and could change how the JVM decides to handle that synchronized block. If ep is a local variable and the SessionAdvised constructor does not access any statics, then the lock elision and escape analysis algorithms may be determining that it can safely move the lock point to after the constructor call. It would then be able to remove the lock entirely and replace with a simple memory barrier instruction because it deduced that the lock can't actually protect anything that isn't local to the thread because the reference from the constructor can't escape to other threads until after that point.
> JDK 1.6 is going end of life so this likely can't be addressed at the JVM level; we may need to offer a resolution within JBoss to avoid this with people using latest 1.6 SunJDK releases.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
More information about the jboss-jira
mailing list