[jboss-jira] [JBoss JIRA] (WFCORE-202) Deadlock when shutdown Wildfly server during CLI client connection
Chao Wang (JIRA)
issues at jboss.org
Tue Oct 28 05:30:35 EDT 2014
[ https://issues.jboss.org/browse/WFCORE-202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015471#comment-13015471 ]
Chao Wang commented on WFCORE-202:
----------------------------------
I think the CLIModelControllerClient.handleClose action should not be blocked under this condition, as "main" is waiting on some message receive to complete, however, server is already down at this moment.
What if adding a new boolean variable to indicate channel creation completion. When server is shutdown before finishing channel creation, ChannelCloseHandler can directly handle the close action. Something may like [242507|http://pastebin.test.redhat.com/242507] .And CLIModelControllerClient.getOrCreateChannel() will see "IOException: WFLYPRT0054: Channel closed" as a consequence
> Deadlock when shutdown Wildfly server during CLI client connection
> ------------------------------------------------------------------
>
> Key: WFCORE-202
> URL: https://issues.jboss.org/browse/WFCORE-202
> Project: WildFly Core
> Issue Type: Bug
> Components: CLI
> Affects Versions: 1.0.0.Alpha10
> Reporter: Chao Wang
> Assignee: Chao Wang
>
> Creating upstream issue as the same deadlock can be found in WFLY.
> Descirption as comment 5 in [BZ1142538|https://bugzilla.redhat.com/show_bug.cgi?id=1142538]
> {noformat}
> The following deadlock still exists.
> Steps to Reproduce:
> 1. Prepare a running EAP instance with secured management port - optionally in VM
> 2. Execute export JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,address=8787,server=y,suspend=y"
> 3. In the same terminal, execute "bin/jboss-cli.sh --connect --controller=$EAP_IP --user=admin --password=password ':read-resource'"
> 4. From yet another terminal, execute "jdb -attach localhost:8787"
> 5. In JDB:
> 5.a. Create breakpoint: "stop in org.xnio.channels.FramedMessageChannel.receive(java.nio.ByteBuffer)"
> 5.b. Resume all threads: "resume"
> 5.c. [Execute five times] Wait until breakpoint is hit and execute "resume". Either set timeout or be fast so that timeout does not occur first
> 6. Execute "kill -9 $EAP_PID" - optionally in VM
> 7. In JDB:
> 8.a. Remove breakpoint: "clear org.xnio.channels.FramedMessageChannel.receive(java.nio.ByteBuffer)"
> 8.b. Resume all threads: "resume"
> 9. Now dump the stack trace of jboss-cli.sh: "kill -3 $JBOSS_CLI_PID"
> Found one Java-level deadlock:
> =============================
> "Remoting "cli-client" read-1":
> waiting to lock monitor 0x00007ff9c829b558 (object 0x0000000783433408, a java.lang.String),
> which is held by "main"
> "main":
> waiting to lock monitor 0x00007ff9c8333c48 (object 0x00000007851ae6e0, a java.util.ArrayDeque),
> which is held by "Remoting "cli-client" read-1"
> Java stack information for the threads listed above:
> ===================================================
> "Remoting "cli-client" read-1":
> at org.jboss.as.cli.impl.CLIModelControllerClient$ChannelCloseHandler.handleClose(CLIModelControllerClient.java:286)
> - waiting to lock <0x0000000783433408> (a java.lang.String)
> at org.jboss.as.cli.impl.CLIModelControllerClient$ChannelCloseHandler.handleClose(CLIModelControllerClient.java:269)
> at org.jboss.remoting3.spi.SpiUtils.safeHandleClose(SpiUtils.java:54)
> at org.jboss.remoting3.spi.AbstractHandleableCloseable$CloseHandlerTask.run(AbstractHandleableCloseable.java:501)
> at org.jboss.remoting3.spi.AbstractHandleableCloseable.runCloseTask(AbstractHandleableCloseable.java:406)
> at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeComplete(AbstractHandleableCloseable.java:277)
> at org.jboss.remoting3.remote.RemoteConnectionChannel.closeAction(RemoteConnectionChannel.java:532)
> at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:359)
> at org.jboss.remoting3.remote.RemoteConnectionHandler.closeAllChannels(RemoteConnectionHandler.java:392)
> - locked <0x00000007851ae6e0> (a java.util.ArrayDeque)
> at org.jboss.remoting3.remote.RemoteConnectionHandler.handleConnectionClose(RemoteConnectionHandler.java:109)
> at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:81)
> - locked <0x00000007851ae6e0> (a java.util.ArrayDeque)
> at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:45)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
> at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)
> at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
> at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)
> at org.xnio.ssl.JsseConnectedSslStreamChannel.handleReadable(JsseConnectedSslStreamChannel.java:183)
> at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
> at org.xnio.nio.NioHandle.run(NioHandle.java:90)
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:198)
> "main":
> at org.jboss.remoting3.remote.RemoteConnectionChannel.receiveMessage(RemoteConnectionChannel.java:361)
> - waiting to lock <0x00000007851ae6e0> (a java.util.ArrayDeque)
> at org.jboss.as.protocol.mgmt.FutureManagementChannel$Establishing.connectionOpened(FutureManagementChannel.java:217)
> at org.jboss.as.protocol.ProtocolConnectionManager.connect(ProtocolConnectionManager.java:78)
> - locked <0x0000000784978a00> (a org.jboss.as.protocol.ProtocolConnectionManager)
> at org.jboss.as.protocol.mgmt.FutureManagementChannel$Establishing.getChannel(FutureManagementChannel.java:204)
> at org.jboss.as.cli.impl.CLIModelControllerClient.getOrCreateChannel(CLIModelControllerClient.java:160)
> - locked <0x0000000783433408> (a java.lang.String)
> at org.jboss.as.cli.impl.CLIModelControllerClient$2.getChannel(CLIModelControllerClient.java:120)
> at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:123)
> at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:98)
> at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeRequest(AbstractModelControllerClient.java:236)
> at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:141)
> at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:127)
> at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:71)
> at org.jboss.as.cli.impl.CommandContextImpl.tryConnection(CommandContextImpl.java:980)
> at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:841)
> at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:817)
> at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:282)
> at org.jboss.as.cli.impl.CliLauncher.main(CliLauncher.java:250)
> at org.jboss.as.cli.CommandLineMain.main(CommandLineMain.java:34)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at org.jboss.modules.Module.run(Module.java:312)
> at org.jboss.modules.Main.main(Main.java:460)
> {noformat}
> It can be easily reproduce with Eclipse as:
> {noformat}
> 1 start Wildfly
> 2 uncomment "JAVA_OPTS="$JAVA_OPTS -agentlib:jdwp=transport=dt_socket,address=8787,server=y,suspend=y" in jboss-cli.sh and connect to server
> 3 add two break points at
> CLIModelControllerClient$ChannelCloseHandler [line: 285] - handleClose(Channel, IOException)
> RemoteConnectionChannel [line: 360] - receiveMessage(Receiver)
> 4 connect to 8787 from eclipse debug
> 5 shutdown Wildfly
> A deadlock between "main" and "cli-client" is in Eclipse debug stack.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.1#6329)
More information about the jboss-jira
mailing list