[jboss-jira] [JBoss JIRA] (WFLY-5994) RejectedExecutionException: XNIO007007 on WFLY 10 CR5 shutdown

[David Lloyd (JIRA)]

    [ https://issues.jboss.org/browse/WFLY-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13148896#comment-13148896 ] 

David Lloyd commented on WFLY-5994:
-----------------------------------

OK here's the problem.  The Remoting management endpoint ("RME" for short) shutdown is proceeding correctly, however, the CLI is connecting via HTTP, which uses its own management XNIO worker, instead of the RME's internal worker.

So what happens is, shutdown is being initiated, and the RME (which is managing the CLI connection) is properly waiting for all of its connections to terminate.  However, since the CLI's underlying XNIO connection is using an XNIO worker other than the RME's, we have a problem: the XNIO worker that the CLI is using gets shut down with no regard for the RME or its connections, causing the shutdown to fail.

I have verified that this is the case by hacking in the following dependency to the RME:

{code:diff}
diff --git a/remoting/subsystem/src/main/java/org/jboss/as/remoting/RemotingServices.java b/remoting/subsystem/src/main/java/org/jboss/as/remoting/RemotingServices.java
index b11c0ed..20bfdbe 100644
--- a/remoting/subsystem/src/main/java/org/jboss/as/remoting/RemotingServices.java
+++ b/remoting/subsystem/src/main/java/org/jboss/as/remoting/RemotingServices.java
@@ -96,6 +96,7 @@ public class RemotingServices {
                                                          final String hostName, final EndpointService.EndpointType type, final OptionMap options) {
         ManagementEndpointService endpointService = new ManagementEndpointService(hostName, type, options);
         serviceTarget.addService(endpointName, endpointService)
+                .addDependency(ServiceName.JBOSS.append("serverManagement", "controller", "management", "http"))
                 .setInitialMode(ACTIVE)
                 .install();
     }
{code}

This causes shutdown to proceed in the proper orderly fashion - however, this naïve strategy obviously fails in the event that there is no HTTP management endpoint.  Even adding this dependency conditionally fails, because the HTTP management interface could be added, removed, or modified at any time.

A better solution would be to use a shared internal management-only XNIO worker for both of these things, with a proper dependency relationship from both the management endpoint and the HTTP endpoint.

P.S. The reason this appeared recently is that our recent performance enhancements which use the write thread to write all data cause more/earlier exection to the XNIO worker's various threads, increasing the chances of a failure in this case.

> RejectedExecutionException: XNIO007007 on WFLY 10 CR5 shutdown
> --------------------------------------------------------------
>
>                 Key: WFLY-5994
>                 URL: https://issues.jboss.org/browse/WFLY-5994
>             Project: WildFly
>          Issue Type: Bug
>          Components: Server
>    Affects Versions: 10.0.0.CR5
>            Reporter: Rostislav Svoboda
>            Assignee: David Lloyd
>            Priority: Blocker
>
> WFLY 10 CR5 logs Error message on shutdown (both standalone and domain mode). This happens regularly, platforms like Fedora 23, RHEL 6, RHEL 7. WFLY 10 CR5 does not have clean start & stop. 
> *reproduce*
> \# start WFLY 10 CR5
> {noformat}
> $ ./standalone.sh
> ...
> 10:30:46,536 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 10.0.0.CR5 (WildFly Core 2.0.5.Final) started in 3999ms - Started 267 of 569 services (386 services are lazy, passive or on-demand)
> {noformat}
> \# shutdown via CLI
> {noformat}
> $ ./jboss-cli.sh -c shutdown
> {noformat}
> \# check console where WFLY 10 CR5 was started
> {noformat}
> 10:30:53,937 ERROR [org.jboss.remoting.resource] (MSC service thread 1-3) Close action for Connection handler for Remoting connection 4181b134 to /127.0.0.1:51703 failed to execute (resource may be left in an indeterminate state): java.util.concurrent.RejectedExecutionException: XNIO007007: Thread is terminating
> 	at org.xnio.nio.WorkerThread.execute(WorkerThread.java:588)
> 	at org.jboss.remoting3.remote.RemoteConnection$RemoteWriteListener.send(RemoteConnection.java:327)
> 	at org.jboss.remoting3.remote.RemoteConnection.send(RemoteConnection.java:142)
> 	at org.jboss.remoting3.remote.RemoteConnectionHandler.sendCloseRequestBody(RemoteConnectionHandler.java:289)
> 	at org.jboss.remoting3.remote.RemoteConnectionHandler.sendCloseRequestBody(RemoteConnectionHandler.java:278)
> 	at org.jboss.remoting3.remote.RemoteConnectionHandler.sendCloseRequest(RemoteConnectionHandler.java:235)
> 	at org.jboss.remoting3.remote.RemoteConnectionHandler.closeAction(RemoteConnectionHandler.java:413)
> 	at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:372)
> 	at org.jboss.remoting3.ConnectionImpl.closeAction(ConnectionImpl.java:54)
> 	at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:372)
> 	at org.jboss.remoting3.EndpointImpl.closeAction(EndpointImpl.java:225)
> 	at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:372)
> 	at org.jboss.as.remoting.EndpointService.stop(EndpointService.java:98)
> 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
> 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> {noformat}
> {noformat}
> 10:30:53,936 ERROR [org.jboss.remoting.resource] (Remoting "rs:MANAGEMENT" task-8) Close action for Channel ID 1158edde (inbound) of Remoting connection 4181b134 to /127.0.0.1:51703 failed to execute (resource may be left in an indeterminate state): java.util.concurrent.RejectedExecutionException: XNIO007007: Thread is terminating
> 	at org.xnio.nio.WorkerThread.execute(WorkerThread.java:588)
> 	at org.jboss.remoting3.remote.RemoteConnection$RemoteWriteListener.send(RemoteConnection.java:327)
> 	at org.jboss.remoting3.remote.RemoteConnection.send(RemoteConnection.java:138)
> 	at org.jboss.remoting3.remote.RemoteConnectionChannel.closeReadsAndWrites(RemoteConnectionChannel.java:261)
> 	at org.jboss.remoting3.remote.RemoteConnectionChannel.closeAction(RemoteConnectionChannel.java:534)
> 	at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:372)
> 	at org.jboss.remoting3.remote.RemoteConnectionHandler.closeAllChannels(RemoteConnectionHandler.java:437)
> 	at org.jboss.remoting3.remote.RemoteConnectionHandler.handleConnectionClose(RemoteConnectionHandler.java:114)
> 	at org.jboss.remoting3.remote.RemoteReadListener$1$1.run(RemoteReadListener.java:57)
> 	at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:717)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.11#64026)