Joachim Petrich created WFLY-12016: -------------------------------------- Summary: Outbound channel count will become divergent from actually open channels when remote service is not yet registered Key: WFLY-12016 URL: https://issues.jboss.org/browse/WFLY-12016 Project: WildFly Issue Type: Bug Components: Remoting Affects Versions: 15.0.1.Final Environment: Wildfly 15.0.1.Final Windows 10 standalone_full_ha configuration server nodes as Wildfly cluster with 2 nodes, various client nodes server and client nodes a running a standalone Wildfly Reporter: Joachim Petrich Assignee: Flavia Rainone Requesting the opening of an outbound channel via org.jboss.remoting3.remote.RemoteConnectionHandler.open(String, Result<Channel>, OptionMap) for a service type that is not registered on the remote node results in a divergence between the number of channels actually opened and the internal counter. In the worst case this will result in an unusable connection to the remote node that will never recover and can only be fixed by restarting the application server from which the outbound channel should have been opened. In our case, the problem came up after restarting the remote application server, resulting in the following repeated exception: org.jboss.ejb.client.RequestSendFailedException: org.jboss.remoting3.ProtocolException: Too many channels open@remote+http://172.30.15.150:8080 at org.jboss.ejb.protocol.remote.RemoteEJBReceiver$1.lambda$handleDone$0(RemoteEJBReceiver.java:83) at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:720) at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) at java.lang.Thread.run(Unknown Source) Suppressed: org.jboss.ejb.client.RequestSendFailedException: org.jboss.remoting3.ProtocolException: Too many channels open@remote+http://172.30.15.150:8080 ... 7 more The following excerpt from checking the Service MBean to dump the channel status showed a maxed outbound channel counter but no open channels: Connection LOCAL-NODE/172.30.15.151:59765 <-> /172.30.15.150:8080 Raw: org.xnio.ssl.JsseSslStreamConnection@31166dfc * Flags: supports-message-close auth-cap * 0 (max 40) inbound channels * 40 (max 40) outbound channels * Channels: Connection LOCAL-NODE/172.30.15.151:57690 <-> /172.30.15.152:8080 Reproducing the problem with activated org.jboss.remoting traces showed, that the remote application server was refusing the channel open request (for the jboss.ejb service type): 14:00:26,672 TRACE [org.jboss.remoting.remote] (default I/O-4) Refusing service on channel 2d4e99c8: Unknown service name Meanwhile, the local application server was trying to open channels until the maximum of 40 channels was reached. The propes fix (as we did locally): When handling a service error response in org.jboss.remoting3.remote.RemoteReadListener.handleEvent(ConduitStreamSourceChannel) properly decrement the internal counter by calling org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelClosed() to match the previously succeeded call to org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelOpen() in org.jboss.remoting3.remote.RemoteConnectionHandler.open(String, Result<Channel>, OptionMap). diff --git a/src/main/java/org/jboss/remoting3/remote/RemoteReadListener.java b/src/main/java/org/jboss/remoting3/remote/RemoteReadListener.java index ceb0e32e..bccd046a 100644 --- a/src/main/java/org/jboss/remoting3/remote/RemoteReadListener.java +++ b/src/main/java/org/jboss/remoting3/remote/RemoteReadListener.java @@ -424,6 +424,9 @@ final class RemoteReadListener implements ChannelListener<ConduitStreamSourceCha // invalid break; } + // match the handleOutboundChannelOpened() call in RemoteConnectionHandler.open(String, Result<Channel>, OptionMap) + handler.handleOutboundChannelClosed(); + log.tracef("Properly decremented outbound channel counter after closing pending channel %08x.", channelId); String reason = new String(Buffers.take(buffer), StandardCharsets.UTF_8); pendingChannel.getResult().setException(new ServiceOpenException(reason)); break; -- This message was sent by Atlassian Jira (v7.12.1#712002)