9:44 a.m.
[
https://issues.jboss.org/browse/WFLY-5994?page=com.atlassian.jira.plugin....
]
David Lloyd commented on WFLY-5994:
-----------------------------------
OK here's the problem. The Remoting management endpoint ("RME" for short)
shutdown is proceeding correctly, however, the CLI is connecting via HTTP, which uses its
own management XNIO worker, instead of the RME's internal worker.
So what happens is, shutdown is being initiated, and the RME (which is managing the CLI
connection) is properly waiting for all of its connections to terminate. However, since
the CLI's underlying XNIO connection is using an XNIO worker other than the RME's,
we have a problem: the XNIO worker that the CLI is using gets shut down with no regard for
the RME or its connections, causing the shutdown to fail.
I have verified that this is the case by hacking in the following dependency to the RME:
{code:diff}
diff --git a/remoting/subsystem/src/main/java/org/jboss/as/remoting/RemotingServices.java
b/remoting/subsystem/src/main/java/org/jboss/as/remoting/RemotingServices.java
index b11c0ed..20bfdbe 100644
--- a/remoting/subsystem/src/main/java/org/jboss/as/remoting/RemotingServices.java
+++ b/remoting/subsystem/src/main/java/org/jboss/as/remoting/RemotingServices.java
@@ -96,6 +96,7 @@ public class RemotingServices {
final String hostName, final
EndpointService.EndpointType type, final OptionMap options) {
ManagementEndpointService endpointService = new
ManagementEndpointService(hostName, type, options);
serviceTarget.addService(endpointName, endpointService)
+ .addDependency(ServiceName.JBOSS.append("serverManagement",
"controller", "management", "http"))
.setInitialMode(ACTIVE)
.install();
}
{code}
This causes shutdown to proceed in the proper orderly fashion - however, this naïve
strategy obviously fails in the event that there is no HTTP management endpoint. Even
adding this dependency conditionally fails, because the HTTP management interface could be
added, removed, or modified at any time.
A better solution would be to use a shared internal management-only XNIO worker for both
of these things, with a proper dependency relationship from both the management endpoint
and the HTTP endpoint.
P.S. The reason this appeared recently is that our recent performance enhancements which
use the write thread to write all data cause more/earlier exection to the XNIO
worker's various threads, increasing the chances of a failure in this case.
RejectedExecutionException: XNIO007007 on WFLY 10 CR5 shutdown
--------------------------------------------------------------
Key: WFLY-5994
URL: https://issues.jboss.org/browse/WFLY-5994
Project: WildFly
Issue Type: Bug
Components: Server
Affects Versions: 10.0.0.CR5
Reporter: Rostislav Svoboda
Assignee: David Lloyd
Priority: Blocker
WFLY 10 CR5 logs Error message on shutdown (both standalone and domain mode). This
happens regularly, platforms like Fedora 23, RHEL 6, RHEL 7. WFLY 10 CR5 does not have
clean start & stop.
*reproduce*
\# start WFLY 10 CR5
{noformat}
$ ./standalone.sh
...
10:30:46,536 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full
10.0.0.CR5 (WildFly Core 2.0.5.Final) started in 3999ms - Started 267 of 569 services (386
services are lazy, passive or on-demand)
{noformat}
\# shutdown via CLI
{noformat}
$ ./jboss-cli.sh -c shutdown
{noformat}
\# check console where WFLY 10 CR5 was started
{noformat}
10:30:53,937 ERROR [org.jboss.remoting.resource] (MSC service thread 1-3) Close action
for Connection handler for Remoting connection 4181b134 to /127.0.0.1:51703 failed to
execute (resource may be left in an indeterminate state):
java.util.concurrent.RejectedExecutionException: XNIO007007: Thread is terminating
at org.xnio.nio.WorkerThread.execute(WorkerThread.java:588)
at
org.jboss.remoting3.remote.RemoteConnection$RemoteWriteListener.send(RemoteConnection.java:327)
at org.jboss.remoting3.remote.RemoteConnection.send(RemoteConnection.java:142)
at
org.jboss.remoting3.remote.RemoteConnectionHandler.sendCloseRequestBody(RemoteConnectionHandler.java:289)
at
org.jboss.remoting3.remote.RemoteConnectionHandler.sendCloseRequestBody(RemoteConnectionHandler.java:278)
at
org.jboss.remoting3.remote.RemoteConnectionHandler.sendCloseRequest(RemoteConnectionHandler.java:235)
at
org.jboss.remoting3.remote.RemoteConnectionHandler.closeAction(RemoteConnectionHandler.java:413)
at
org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:372)
at org.jboss.remoting3.ConnectionImpl.closeAction(ConnectionImpl.java:54)
at
org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:372)
at org.jboss.remoting3.EndpointImpl.closeAction(EndpointImpl.java:225)
at
org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:372)
at org.jboss.as.remoting.EndpointService.stop(EndpointService.java:98)
at
org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
at
org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
{noformat}
10:30:53,936 ERROR [org.jboss.remoting.resource] (Remoting "rs:MANAGEMENT"
task-8) Close action for Channel ID 1158edde (inbound) of Remoting connection 4181b134 to
/127.0.0.1:51703 failed to execute (resource may be left in an indeterminate state):
java.util.concurrent.RejectedExecutionException: XNIO007007: Thread is terminating
at org.xnio.nio.WorkerThread.execute(WorkerThread.java:588)
at
org.jboss.remoting3.remote.RemoteConnection$RemoteWriteListener.send(RemoteConnection.java:327)
at org.jboss.remoting3.remote.RemoteConnection.send(RemoteConnection.java:138)
at
org.jboss.remoting3.remote.RemoteConnectionChannel.closeReadsAndWrites(RemoteConnectionChannel.java:261)
at
org.jboss.remoting3.remote.RemoteConnectionChannel.closeAction(RemoteConnectionChannel.java:534)
at
org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:372)
at
org.jboss.remoting3.remote.RemoteConnectionHandler.closeAllChannels(RemoteConnectionHandler.java:437)
at
org.jboss.remoting3.remote.RemoteConnectionHandler.handleConnectionClose(RemoteConnectionHandler.java:114)
at org.jboss.remoting3.remote.RemoteReadListener$1$1.run(RemoteReadListener.java:57)
at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:717)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
3764
days inactive
3764
days old
0 comments
1 participants
participants (1)
-
David Lloyd (JIRA)