[jboss-jira] [JBoss JIRA] (JBMESSAGING-1941) Messaging deadlocks on AspectManager

Messaging deadlocks on AspectManager ------------------------------------ Key: JBMESSAGING-1941 URL: https://issues.jboss.org/browse/JBMESSAGING-1941 Project: JBoss Messaging Issue Type: Bug Affects Versions: 1.4.8.SP9 Environment: -JBoss Enterprise Application Platform (EAP) 5 Reporter: Aaron Ogburn Assignee: Yong Hao Gao Fix For: 1.4.8.SP10 Attachments: 00779442testpatch.diff, JBMESSAGING-1941.diff Messaging can stall JBoss at start up. The hang occurs as the main thread blocks in different locations in JMS trying to acquire the AspectManager instance lock: {noformat} "main" prio=10 tid=0x00007fe69c1e1000 nid=0x3613 waiting for monitor entry [0x00007fe6828e5000] java.lang.Thread.State: BLOCKED (on object monitor) at org.jboss.aop.AspectManager.getAdvisor(AspectManager.java:728) - waiting to lock <0x00000000a5e43e50> (a org.jboss.aop.AspectManager) at org.jboss.jms.server.endpoint.advised.SessionAdvised.<clinit>(SessionAdvised.java) at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.createSessionDelegate(ServerConnectionEndpoint.java:273) - locked <0x00000000a5600f48> (a org.jboss.aop.asintegration.jboss5.ScopedVFSClassLoaderDomain) at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.createSessionDelegate(ServerConnectionEndpoint.java:229) {noformat} Since the main thread is stalled, so too is start up. This lock is always held by another making a similar call like so: {noformat} "WorkManager(2)-1" daemon prio=10 tid=0x00007fe62dd96000 nid=0x363e in Object.wait() [0x00007fe68029a000] java.lang.Thread.State: RUNNABLE at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.createSessionDelegate(ServerConnectionEndpoint.java:273) - locked <0x00000000a5e43e50> (a org.jboss.aop.AspectManager) at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.createSessionDelegate(ServerConnectionEndpoint.java:229) {noformat} The code there at line 273 is simple enough and shouldn't cause a long stall: {code:title=org.jboss.jms.server.endpoint.ServerConnectionEndpoint.java|borderStyle=solid} synchronized (AspectManager.instance()) { advised = new SessionAdvised(ep); // line 273 } {code} Nonetheless, it does clearly hang at this line as WorkManager(2)-1 never progresses. There's no GC/CPU issues or anything of that nature, but we get a slight in the thread dump as WorkManager(2)-1 is shown to be in Object.wait(). Checking the thread through pstack, we can see the issue. It is waiting in native/JDK level operations required to initialize the SessionAdvised class: {noformat} Thread 2 (Thread 0x7fe68029d700 (LWP 13886)): #0 0x0000003b3e40b43c in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe6a12d447e in os:latformEvent:ark() () from /opt/app/jvm/jdk1.6.0_38/jre/lib/amd64/server/libjvm.so #2 0x00007fe6a12c54fa in ObjectMonitor::wait(long, bool, Thread*) () from /opt/app/jvm/jdk1.6.0_38/jre/lib/amd64/server/libjvm.so #3 0x00007fe6a13ab5f0 in ObjectSynchronizer::waitUninterruptibly(Handle, long, Thread*) () from /opt/app/jvm/jdk1.6.0_38/jre/lib/amd64/server/libjvm.so #4 0x00007fe6a107115b in instanceKlass::initialize_impl(instanceKlassHandle, Thread*) () from /opt/app/jvm/jdk1.6.0_38/jre/lib/amd64/server/libjvm.so #5 0x00007fe6a107077a in instanceKlass::initialize(Thread*) () from /opt/app/jvm/jdk1.6.0_38/jre/lib/amd64/server/libjvm.so #6 0x00007fe6a1097c5c in InterpreterRuntime::_new(JavaThread*, constantPoolOopDesc*, int) () from /opt/app/jvm/jdk1.6.0_38/jre/lib/amd64/server/libjvm.so {noformat} These calls are naturally synchronized internally in the JVM so only one thread can proceed through initialization of a class at a given time. Checking for any threads in process of initializing SessionAdvised, we can see only the main thread is. But the main thread is still blocked for the AspectManager lock. So we have a deadlock here. The main thread holds the initialization lock while waiting for the AspectManager lock, while the WorkManager(2)-1 thread holds the AspectManager lock while waiting for the initialization lock. The initialization of SessionAdvised is wrapped in the synchronized block so the main thread should not have been able to proceed into the initialization at all without already possessing the AspectManager lock. It locks the ScopedVFSClassLoaderDomain, which extends the AspectManager. This is different from the actual AspectManager though that main tries to lock in the clinit call and that it deadlocks on.