[jboss-dev-forums] [Design of Messaging on JBoss (Messaging/JBoss)] - Re: Another DeadLock, something around InVM

[clebert.suconic@jboss.com]

I believe (unless I"m missing something here) this is being caused by the recent changes on shutting down the executors:

I have found three threads waiting the same lock (00007fc50a611708):


On the thread dump (1):

- "Thread-1" waiting to the lock, before a call to failover, while the executor is being used.

- Another thread named "LocalThread i = 10" is holding the lock and waiting to shutdown the executor.


I would consider this a dead lock, even though the VM didn't report it so as it didn't explicity used to lock objects.


I am adding PingStressTest into stressTest (which is the test i previously reported here with a few modifications). The test is currently disabled. Change getNumberOfIterations to something like 10 or 100 to replicate this hang.


The test passed for me after reverting r5990.

I will run it a few more times and report the results later.



___________________________________________________________________________
*1 - Thread dump


  | "LocalThread i = 10" prio=10 tid=0x00007fc50035c000 nid=0x6484 waiting on condition [0x0000000040849000..0x0000000040849c00]
  |    java.lang.Thread.State: TIMED_WAITING (parking)
  | 	at sun.misc.Unsafe.park(Native Method)
  | 	- parking to wait for  <0x00007fc50a5eebe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  | 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  | 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
  | 	at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1244)
  | 	at org.jboss.messaging.integration.transports.netty.NettyConnector.close(NettyConnector.java:363)
  | 	- locked <0x00007fc50a6114d0> (a org.jboss.messaging.integration.transports.netty.NettyConnector)
  | 	at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.checkCloseConnections(ConnectionManagerImpl.java:761)
  | 	at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.returnConnection(ConnectionManagerImpl.java:852)
  | 	at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:387)
  | 	- locked <0x00007fc50a611708> (a java.lang.Object)
  | 	- locked <0x00007fc50a6116f8> (a java.lang.Object)
  | 	at org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1313)
  | 	at org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:761)
  | 	at org.jboss.messaging.tests.stress.remote.PingStressTest$1LocalThread.run(PingStressTest.java:248)
  | 
  | 
  | "Thread-1 (group:jbm-pinger-threads-896472140)" daemon prio=10 tid=0x00007fc500385c00 nid=0x6463 waiting for monitor entry [0x0000000045ece000..0x0000000045ecec00]
  |    java.lang.Thread.State: BLOCKED (on object monitor)
  | 	at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failover(ConnectionManagerImpl.java:493)
  | 	- waiting to lock <0x00007fc50a611708> (a java.lang.Object)
  | 	at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.connectionFailed(ConnectionManagerImpl.java:411)
  | 	at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callListeners(RemotingConnectionImpl.java:530)
  | 	at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:421)
  | 	at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$Pinger.run(RemotingConnectionImpl.java:1537)
  | 	- locked <0x00007fc50a614508> (a org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$Pinger)
  | 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
  | 	at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
  | 	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
  | 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
  | 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
  | 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
  | 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
  | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  | 	at java.lang.Thread.run(Thread.java:619)
  | 
  | 
  | 
  | "LocalThread i = 10" prio=10 tid=0x00007fc50035c000 nid=0x6484 waiting on condition [0x0000000040849000..0x0000000040849c00]
  |    java.lang.Thread.State: TIMED_WAITING (parking)
  | 	at sun.misc.Unsafe.park(Native Method)
  | 	- parking to wait for  <0x00007fc50a5eebe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  | 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  | 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
  | 	at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1244)
  | 	at org.jboss.messaging.integration.transports.netty.NettyConnector.close(NettyConnector.java:363)
  | 	- locked <0x00007fc50a6114d0> (a org.jboss.messaging.integration.transports.netty.NettyConnector)
  | 	at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.checkCloseConnections(ConnectionManagerImpl.java:761)
  | 	at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.returnConnection(ConnectionManagerImpl.java:852)
  | 	at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:387)
  | 	- locked <0x00007fc50a611708> (a java.lang.Object)
  | 	- locked <0x00007fc50a6116f8> (a java.lang.Object)
  | 	at org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1313)
  | 	at org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:761)
  | 	at org.jboss.messaging.tests.stress.remote.PingStressTest$1LocalThread.run(PingStressTest.java:248)
  | 
  | 



View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4215066#4215066

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4215066