10:56 a.m.
Joachim Petrich created WFLY-12016:
--------------------------------------
Summary: Outbound channel count will become divergent from actually open
channels when remote service is not yet registered
Key: WFLY-12016
URL: https://issues.jboss.org/browse/WFLY-12016
Project: WildFly
Issue Type: Bug
Components: Remoting
Affects Versions: 15.0.1.Final
Environment: Wildfly 15.0.1.Final
Windows 10
standalone_full_ha configuration
server nodes as Wildfly cluster with 2 nodes,
various client nodes
server and client nodes a running a standalone Wildfly
Reporter: Joachim Petrich
Assignee: Flavia Rainone
Requesting the opening of an outbound channel via
org.jboss.remoting3.remote.RemoteConnectionHandler.open(String, Result<Channel>,
OptionMap) for a service type that is not registered on the remote node results in a
divergence between the number of channels actually opened and the internal counter.
In the worst case this will result in an unusable connection to the remote node that will
never recover and can only be fixed by restarting the application server from which the
outbound channel should have been opened.
In our case, the problem came up after restarting the remote application server, resulting
in the following repeated exception:
org.jboss.ejb.client.RequestSendFailedException: org.jboss.remoting3.ProtocolException:
Too many channels open@remote+http://172.30.15.150:8080
at
org.jboss.ejb.protocol.remote.RemoteEJBReceiver$1.lambda$handleDone$0(RemoteEJBReceiver.java:83)
at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:720)
at
org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
at java.lang.Thread.run(Unknown Source)
Suppressed: org.jboss.ejb.client.RequestSendFailedException:
org.jboss.remoting3.ProtocolException: Too many channels
open@remote+http://172.30.15.150:8080
... 7 more
The following excerpt from checking the Service MBean to dump the channel status showed a
maxed outbound channel counter but no open channels:
Connection LOCAL-NODE/172.30.15.151:59765 <-> /172.30.15.150:8080
Raw: org.xnio.ssl.JsseSslStreamConnection@31166dfc
* Flags: supports-message-close auth-cap
* 0 (max 40) inbound channels
* 40 (max 40) outbound channels
* Channels:
Connection LOCAL-NODE/172.30.15.151:57690 <-> /172.30.15.152:8080
Reproducing the problem with activated org.jboss.remoting traces showed, that the remote
application server was refusing the channel open request (for the jboss.ejb service
type):
14:00:26,672 TRACE [org.jboss.remoting.remote] (default I/O-4) Refusing service on channel
2d4e99c8: Unknown service name
Meanwhile, the local application server was trying to open channels until the maximum of
40 channels was reached.
The propes fix (as we did locally):
When handling a service error response in
org.jboss.remoting3.remote.RemoteReadListener.handleEvent(ConduitStreamSourceChannel)
properly decrement the internal counter by calling
org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelClosed() to match
the previously succeeded call to
org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelOpen() in
org.jboss.remoting3.remote.RemoteConnectionHandler.open(String, Result<Channel>,
OptionMap).
diff --git a/src/main/java/org/jboss/remoting3/remote/RemoteReadListener.java
b/src/main/java/org/jboss/remoting3/remote/RemoteReadListener.java
index ceb0e32e..bccd046a 100644
--- a/src/main/java/org/jboss/remoting3/remote/RemoteReadListener.java
+++ b/src/main/java/org/jboss/remoting3/remote/RemoteReadListener.java
@@ -424,6 +424,9 @@ final class RemoteReadListener implements
ChannelListener<ConduitStreamSourceCha
// invalid
break;
}
+ // match the handleOutboundChannelOpened() call in
RemoteConnectionHandler.open(String, Result<Channel>, OptionMap)
+ handler.handleOutboundChannelClosed();
+ log.tracef("Properly decremented outbound channel
counter after closing pending channel %08x.", channelId);
String reason = new String(Buffers.take(buffer),
StandardCharsets.UTF_8);
pendingChannel.getResult().setException(new
ServiceOpenException(reason));
break;
--
This message was sent by Atlassian Jira
(v7.12.1#712002)