[jboss-jira] [JBoss JIRA] (WFCORE-196) Hang in HostController shutdown when CTRL-C is used

[Brian Stansberry (JIRA)]

Brian Stansberry created WFCORE-196:
---------------------------------------

             Summary: Hang in HostController shutdown when CTRL-C is used
                 Key: WFCORE-196
                 URL: https://issues.jboss.org/browse/WFCORE-196
             Project: WildFly Core
          Issue Type: Bug
          Components: Domain Management
    Affects Versions: 1.0.0.Alpha10
            Reporter: Brian Stansberry
            Assignee: Brian Stansberry
             Fix For: 1.0.0.Alpha11


It is fairly easy to produce a hang in domain mode shutdown by launching domain.sh and then soft killing the process with CTRL-C.

If this hang happens, a thread dump will show this

{code}
[Host Controller] "Remoting "localhost:MANAGEMENT" task-16" prio=5 tid=0x00007fa07423b800 nid=0x3b0b waiting for monitor entry [0x000000010bce6000]
[Host Controller]    java.lang.Thread.State: BLOCKED (on object monitor)
[Host Controller] 	at org.jboss.as.host.controller.ManagedServer.callbackUnregistered(ManagedServer.java:417)
[Host Controller] 	- waiting to lock <0x00000007f7946a30> (a org.jboss.as.host.controller.ManagedServer)
[Host Controller] 	at org.jboss.as.host.controller.ServerInventoryImpl$1.handleClose(ServerInventoryImpl.java:461)
[Host Controller] 	at org.jboss.as.host.controller.ServerInventoryImpl$1.handleClose(ServerInventoryImpl.java:456)
[Host Controller] 	at org.jboss.remoting3.spi.SpiUtils.safeHandleClose(SpiUtils.java:54)
[Host Controller] 	at org.jboss.remoting3.spi.AbstractHandleableCloseable$CloseHandlerTask.run(AbstractHandleableCloseable.java:501)
[Host Controller] 	at org.jboss.remoting3.spi.AbstractHandleableCloseable.runCloseTask(AbstractHandleableCloseable.java:406)
[Host Controller] 	at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeComplete(AbstractHandleableCloseable.java:277)
[Host Controller] 	at org.jboss.remoting3.remote.RemoteConnectionChannel.closeAction(RemoteConnectionChannel.java:532)
[Host Controller] 	at org.jboss.remoting3.spi.AbstractHandleableCloseable.close(AbstractHandleableCloseable.java:153)
[Host Controller] 	at org.jboss.as.protocol.mgmt.ManagementChannelReceiver.handleEnd(ManagementChannelReceiver.java:127)
[Host Controller] 	at org.jboss.remoting3.remote.RemoteConnectionChannel$2.run(RemoteConnectionChannel.java:282)
[Host Controller] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[Host Controller] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[Host Controller] 	at java.lang.Thread.run(Thread.java:744)


[Host Controller] "Host Controller Service Threads - 13" prio=5 tid=0x00007fa072cbc800 nid=0xa903 waiting on condition [0x000000010d93a000]
[Host Controller]    java.lang.Thread.State: WAITING (parking)
[Host Controller] 	at sun.misc.Unsafe.park(Native Method)
[Host Controller] 	- parking to wait for  <0x00000007f61e5480> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[Host Controller] 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
[Host Controller] 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[Host Controller] 	at org.jboss.as.controller.remote.BlockingQueueOperationListener.retrievePreparedOperation(BlockingQueueOperationListener.java:88)
[Host Controller] 	at org.jboss.as.controller.remote.TransactionalProtocolHandlers.executeBlocking(TransactionalProtocolHandlers.java:76)
[Host Controller] 	at org.jboss.as.host.controller.ManagedServer$ServerStopTask.execute(ManagedServer.java:846)
[Host Controller] 	at org.jboss.as.host.controller.ManagedServer.internalSetState(ManagedServer.java:531)
[Host Controller] 	at org.jboss.as.host.controller.ManagedServer.stop(ManagedServer.java:259)
[Host Controller] 	- locked <0x00000007f7946a30> (a org.jboss.as.host.controller.ManagedServer)
[Host Controller] 	at org.jboss.as.host.controller.ServerInventoryImpl.stopServers(ServerInventoryImpl.java:339)
[Host Controller] 	at org.jboss.as.host.controller.ServerInventoryImpl.shutdown(ServerInventoryImpl.java:437)
[Host Controller] 	at org.jboss.as.host.controller.ServerInventoryService$1.run(ServerInventoryService.java:124)
[Host Controller] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[Host Controller] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[Host Controller] 	at java.lang.Thread.run(Thread.java:744)
[Host Controller] 	at org.jboss.threads.JBossThread.run(JBossThread.java:320)
{code}

The 2nd thread is blocking waiting for a prepared response from one of the servers. The first thread is trying to send a notification that the channel connecting to the server has been closed. The first thread will never unblock because the 2nd thread can't proceed and deliver the information that the channel is closed.

This happens because all the processes (PC, HC, servers) respond to the CTRL-C. So the server is shutting down in parallel and has closed the channel.

In theory this could happen for other reasons too, though. Anything that results in closing the channel while the ManagedServer lock is held.



--
This message was sent by Atlassian JIRA
(v6.3.1#6329)