Remoting channel has been closed errors after a node in cluster goes down (legacy ejb client) --------------------------------------------------------------------------------------------- Key: WFLY-9909 URL: https://issues.jboss.org/browse/WFLY-9909 Project: WildFly Issue Type: Bug Components: EJB, Remoting Affects Versions: JBoss AS7 7.2.0.Final, 12.0.0.Beta1 Reporter: Richard Janík We've been seeing this in our clustered ejb failover tests with the legacy (and only legacy) ejb client: {noformat} 2018/02/20 10:09:51:011 EST [ERROR][Runner - 950] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Error getting response. <javax.ejb.EJBException: java.io.IOException: Channel Channel ID 88395849 (outbound) of Remoting connection 7f615772 to /10.16.90.54:8080 of endpoint "config-based-ejb-client-endpoint" <6949126f> has been closed> javax.ejb.EJBException: java.io.IOException: Channel Channel ID 88395849 (outbound) of Remoting connection 7f615772 to /10.16.90.54:8080 of endpoint "config-based-ejb-client-endpoint" <6949126f> has been closed at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:240) at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:185) at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:148) at com.sun.proxy.$Proxy2.getSerialAndIncrement(Unknown Source) at org.jboss.smartfrog.clustering.ejb3.StatefulSBProcessorFactoryImplLegacyClient$EJB3RequestProcessor.processRequest(StatefulSBProcessorFactoryImplLegacyClient.java:79) at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52) at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Channel Channel ID 88395849 (outbound) of Remoting connection 7f615772 to /10.16.90.54:8080 of endpoint "config-based-ejb-client-endpoint" <6949126f> has been closed at org.jboss.ejb.client.remoting.ChannelAssociation$1.handleClose(ChannelAssociation.java:123) at org.jboss.ejb.client.remoting.ChannelAssociation$1.handleClose(ChannelAssociation.java:115) at org.jboss.remoting3.spi.SpiUtils.safeHandleClose(SpiUtils.java:50) at org.jboss.remoting3.spi.AbstractHandleableCloseable$CloseHandlerTask.run(AbstractHandleableCloseable.java:520) at org.jboss.remoting3.spi.AbstractHandleableCloseable.runCloseTask(AbstractHandleableCloseable.java:425) at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeComplete(AbstractHandleableCloseable.java:286) at org.jboss.remoting3.remote.RemoteConnectionChannel.closeAction(RemoteConnectionChannel.java:508) at org.jboss.remoting3.spi.AbstractHandleableCloseable.close(AbstractHandleableCloseable.java:150) at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.disassociate(RemotingConnectionEJBReceiver.java:273) at org.jboss.ejb.client.EJBClientContext.unregisterEJBReceiver(EJBClientContext.java:458) at org.jboss.ejb.client.EJBClientContext.unregisterNodeEJBReceivers(EJBClientContext.java:497) at org.jboss.ejb.client.remoting.EJBClientContextConnectionReconnectHandler.reconnect(EJBClientContextConnectionReconnectHandler.java:65) at org.jboss.ejb.client.EJBClientContext$ReconnectAttempt.run(EJBClientContext.java:1474) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more {noformat} After having a look in the EAP 7.1 jobs, this was the case there as well, so it's not new. The pattern is similar, which is the reason it fell through our checkers, but there are no related NoSuchEjb exceptions here. These occur fairly often - that is hundreds of times for 2000 clients each making invocations every 4 seconds.