[jboss-jira] [JBoss JIRA] (WFLY-12016) Outbound channel count will become divergent from actually open channels when remote service is not yet registered

[Joachim Petrich (Jira)]

Joachim Petrich created WFLY-12016:
--------------------------------------

             Summary: Outbound channel count will become divergent from actually open channels when remote service is not yet registered
                 Key: WFLY-12016
                 URL: https://issues.jboss.org/browse/WFLY-12016
             Project: WildFly
          Issue Type: Bug
          Components: Remoting
    Affects Versions: 15.0.1.Final
         Environment: Wildfly 15.0.1.Final
Windows 10
standalone_full_ha configuration
server nodes as Wildfly cluster with 2 nodes,
various client nodes
server and client nodes a running a standalone Wildfly 
            Reporter: Joachim Petrich
            Assignee: Flavia Rainone


Requesting the opening of an outbound channel via org.jboss.remoting3.remote.RemoteConnectionHandler.open(String, Result<Channel>, OptionMap) for a service type that is not registered on the remote node results in a divergence between the number of channels actually opened and the internal counter.

In the worst case this will result in an unusable connection to the remote node that will never recover and can only be fixed by restarting the application server from which the outbound channel should have been opened.

In our case, the problem came up after restarting the remote application server, resulting in the following repeated exception:
org.jboss.ejb.client.RequestSendFailedException: org.jboss.remoting3.ProtocolException: Too many channels open at remote+http://172.30.15.150:8080
	at org.jboss.ejb.protocol.remote.RemoteEJBReceiver$1.lambda$handleDone$0(RemoteEJBReceiver.java:83)
	at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:720)
	at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
	at java.lang.Thread.run(Unknown Source)
	Suppressed: org.jboss.ejb.client.RequestSendFailedException: org.jboss.remoting3.ProtocolException: Too many channels open at remote+http://172.30.15.150:8080
		... 7 more

The following excerpt from checking the Service MBean to dump the channel status showed a maxed outbound channel counter but no open channels:
    Connection LOCAL-NODE/172.30.15.151:59765 <-> /172.30.15.150:8080
    Raw: org.xnio.ssl.JsseSslStreamConnection at 31166dfc
    * Flags: supports-message-close auth-cap 
    * 0 (max 40) inbound channels
    * 40 (max 40) outbound channels
    * Channels:
    Connection LOCAL-NODE/172.30.15.151:57690 <-> /172.30.15.152:8080

Reproducing the problem with activated org.jboss.remoting traces showed, that the remote application server was refusing the channel open request (for the jboss.ejb service type):

14:00:26,672 TRACE [org.jboss.remoting.remote] (default I/O-4) Refusing service on channel 2d4e99c8: Unknown service name

Meanwhile, the local application server was trying to open channels until the maximum of 40 channels was reached.

The propes fix (as we did locally):

When handling a service error response in org.jboss.remoting3.remote.RemoteReadListener.handleEvent(ConduitStreamSourceChannel) properly decrement the internal counter by calling org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelClosed() to match the previously succeeded call to
org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelOpen() in org.jboss.remoting3.remote.RemoteConnectionHandler.open(String, Result<Channel>, OptionMap).

diff --git a/src/main/java/org/jboss/remoting3/remote/RemoteReadListener.java b/src/main/java/org/jboss/remoting3/remote/RemoteReadListener.java
index ceb0e32e..bccd046a 100644
--- a/src/main/java/org/jboss/remoting3/remote/RemoteReadListener.java
+++ b/src/main/java/org/jboss/remoting3/remote/RemoteReadListener.java
@@ -424,6 +424,9 @@ final class RemoteReadListener implements ChannelListener<ConduitStreamSourceCha
                                 // invalid
                                 break;
                             }
+                            // match the handleOutboundChannelOpened() call in RemoteConnectionHandler.open(String, Result<Channel>, OptionMap)
+                            handler.handleOutboundChannelClosed();
+                            log.tracef("Properly decremented outbound channel counter after closing pending channel %08x.", channelId);
                             String reason = new String(Buffers.take(buffer), StandardCharsets.UTF_8);
                             pendingChannel.getResult().setException(new ServiceOpenException(reason));
                             break;




--
This message was sent by Atlassian Jira
(v7.12.1#712002)