[jboss-jira] [JBoss JIRA] (WFCORE-202) Deadlock when shutdown Wildfly server during CLI client connection
Chao Wang (JIRA)
issues at jboss.org
Fri Oct 24 06:04:35 EDT 2014
[ https://issues.jboss.org/browse/WFCORE-202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014821#comment-13014821 ]
Chao Wang commented on WFCORE-202:
----------------------------------
I have investigated this issue a bit since it was firstly found in EAP 6.3.2.CP.CR1 by QE.
The deadlock happens when CLI creates connection to server, meanwhile, before the first thread finishes its creation, another thread tries to close channel due to server shutdown or process kill.
"main" thread firstly locks *strategy lock* to continue channel creation, it's blocked by *connection lock* (inside remoting3) during message receive.
"cli-client" worker thread locks firstly *connection lock*, and it's blocked by the *strategy lock* during handleClose().
Not sure if this is adequate, I think we may move *_return strategy.getChannel();_* outside the synchronized block [CLIModelControllerClient.java#L166|https://github.com/wildfly/wildfly-core/blob/b7241fed52e776a93d7d65c53ba4e42b3a2afddc/cli/src/main/java/org/jboss/as/cli/impl/CLIModelControllerClient.java#L166], in that case, it won't lock the strategy "lock" for further close action, if close happens during that connection period, CLI client will get an IOException due to Channel closed since server is shutdown.
[~aloubyansky] Is this a possible change for CLIModelControllerClient?
> Deadlock when shutdown Wildfly server during CLI client connection
> ------------------------------------------------------------------
>
> Key: WFCORE-202
> URL: https://issues.jboss.org/browse/WFCORE-202
> Project: WildFly Core
> Issue Type: Bug
> Components: CLI
> Affects Versions: 1.0.0.Alpha10
> Reporter: Chao Wang
> Assignee: Chao Wang
>
> Creating upstream issue as the same deadlock can be found in WFLY.
> Descirption as comment 5 in [BZ1142538|https://bugzilla.redhat.com/show_bug.cgi?id=1142538]
> {noformat}
> The following deadlock still exists.
> Steps to Reproduce:
> 1. Prepare a running EAP instance with secured management port - optionally in VM
> 2. Execute export JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,address=8787,server=y,suspend=y"
> 3. In the same terminal, execute "bin/jboss-cli.sh --connect --controller=$EAP_IP --user=admin --password=password ':read-resource'"
> 4. From yet another terminal, execute "jdb -attach localhost:8787"
> 5. In JDB:
> 5.a. Create breakpoint: "stop in org.xnio.channels.FramedMessageChannel.receive(java.nio.ByteBuffer)"
> 5.b. Resume all threads: "resume"
> 5.c. [Execute five times] Wait until breakpoint is hit and execute "resume". Either set timeout or be fast so that timeout does not occur first
> 6. Execute "kill -9 $EAP_PID" - optionally in VM
> 7. In JDB:
> 8.a. Remove breakpoint: "clear org.xnio.channels.FramedMessageChannel.receive(java.nio.ByteBuffer)"
> 8.b. Resume all threads: "resume"
> 9. Now dump the stack trace of jboss-cli.sh: "kill -3 $JBOSS_CLI_PID"
> Found one Java-level deadlock:
> =============================
> "Remoting "cli-client" read-1":
> waiting to lock monitor 0x00007ff9c829b558 (object 0x0000000783433408, a java.lang.String),
> which is held by "main"
> "main":
> waiting to lock monitor 0x00007ff9c8333c48 (object 0x00000007851ae6e0, a java.util.ArrayDeque),
> which is held by "Remoting "cli-client" read-1"
> Java stack information for the threads listed above:
> ===================================================
> "Remoting "cli-client" read-1":
> at org.jboss.as.cli.impl.CLIModelControllerClient$ChannelCloseHandler.handleClose(CLIModelControllerClient.java:286)
> - waiting to lock <0x0000000783433408> (a java.lang.String)
> at org.jboss.as.cli.impl.CLIModelControllerClient$ChannelCloseHandler.handleClose(CLIModelControllerClient.java:269)
> at org.jboss.remoting3.spi.SpiUtils.safeHandleClose(SpiUtils.java:54)
> at org.jboss.remoting3.spi.AbstractHandleableCloseable$CloseHandlerTask.run(AbstractHandleableCloseable.java:501)
> at org.jboss.remoting3.spi.AbstractHandleableCloseable.runCloseTask(AbstractHandleableCloseable.java:406)
> at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeComplete(AbstractHandleableCloseable.java:277)
> at org.jboss.remoting3.remote.RemoteConnectionChannel.closeAction(RemoteConnectionChannel.java:532)
> at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:359)
> at org.jboss.remoting3.remote.RemoteConnectionHandler.closeAllChannels(RemoteConnectionHandler.java:392)
> - locked <0x00000007851ae6e0> (a java.util.ArrayDeque)
> at org.jboss.remoting3.remote.RemoteConnectionHandler.handleConnectionClose(RemoteConnectionHandler.java:109)
> at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:81)
> - locked <0x00000007851ae6e0> (a java.util.ArrayDeque)
> at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:45)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
> at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)
> at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
> at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)
> at org.xnio.ssl.JsseConnectedSslStreamChannel.handleReadable(JsseConnectedSslStreamChannel.java:183)
> at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
> at org.xnio.nio.NioHandle.run(NioHandle.java:90)
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:198)
> "main":
> at org.jboss.remoting3.remote.RemoteConnectionChannel.receiveMessage(RemoteConnectionChannel.java:361)
> - waiting to lock <0x00000007851ae6e0> (a java.util.ArrayDeque)
> at org.jboss.as.protocol.mgmt.FutureManagementChannel$Establishing.connectionOpened(FutureManagementChannel.java:217)
> at org.jboss.as.protocol.ProtocolConnectionManager.connect(ProtocolConnectionManager.java:78)
> - locked <0x0000000784978a00> (a org.jboss.as.protocol.ProtocolConnectionManager)
> at org.jboss.as.protocol.mgmt.FutureManagementChannel$Establishing.getChannel(FutureManagementChannel.java:204)
> at org.jboss.as.cli.impl.CLIModelControllerClient.getOrCreateChannel(CLIModelControllerClient.java:160)
> - locked <0x0000000783433408> (a java.lang.String)
> at org.jboss.as.cli.impl.CLIModelControllerClient$2.getChannel(CLIModelControllerClient.java:120)
> at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:123)
> at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:98)
> at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeRequest(AbstractModelControllerClient.java:236)
> at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:141)
> at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:127)
> at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:71)
> at org.jboss.as.cli.impl.CommandContextImpl.tryConnection(CommandContextImpl.java:980)
> at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:841)
> at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:817)
> at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:282)
> at org.jboss.as.cli.impl.CliLauncher.main(CliLauncher.java:250)
> at org.jboss.as.cli.CommandLineMain.main(CommandLineMain.java:34)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at org.jboss.modules.Module.run(Module.java:312)
> at org.jboss.modules.Main.main(Main.java:460)
> {noformat}
> It can be easily reproduce with Eclipse as:
> {noformat}
> 1 start Wildfly
> 2 uncomment "JAVA_OPTS="$JAVA_OPTS -agentlib:jdwp=transport=dt_socket,address=8787,server=y,suspend=y" in jboss-cli.sh and connect to server
> 3 add two break points at
> CLIModelControllerClient$ChannelCloseHandler [line: 285] - handleClose(Channel, IOException)
> RemoteConnectionChannel [line: 360] - receiveMessage(Receiver)
> 4 connect to 8787 from eclipse debug
> 5 shutdown Wildfly
> A deadlock between "main" and "cli-client" is in Eclipse debug stack.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.1#6329)
More information about the jboss-jira
mailing list