EjbClient: No cluster node manager found for node XY during server restart --------------------------------------------------------------------------- Key: WFLY-3839 URL: https://issues.jboss.org/browse/WFLY-3839 Project: WildFly Issue Type: Bug Components: Clustering, EJB Affects Versions: 8.1.0.Final Reporter: Dominik Pospisil Assignee: Tomas Hofman Situation: 4-node cluster, one node is failed at a time, during the whole test, clients are accessing stateless clustered EJB3. The failure type in this particular case is simulating server crash: JVM is killed using kill -9. When the server was starting, being brought back, we saw this error: 16:05:29,966 ERROR [org.jboss.ejb.client.ClusterContext] (ejb-client-cluster-node-connection-creation-2-thread-17) Cannot create EJBReceiver since no cluster node manager found for node perf20 in cluster context for cluster ejb Here is the log output again with context, the server was starting: {noformat} 16:05:29,966 INFO [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-2) Retrying invocation which failed on node perf20 with exception: javax.ejb.NoSuchEJBException: No such EJB[appname=clusterbench-ee6,modulename=clusterbench-ee6-ejb,distinctname=,beanname=RemoteStatelessSBImpl] at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64) at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:366) at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:458) at org.jboss.remoting3.remote.RemoteConnectionChannel$4.run(RemoteConnectionChannel.java:373) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 16:05:29,966 ERROR [org.jboss.ejb.client.ClusterContext] (ejb-client-cluster-node-connection-creation-2-thread-17) Cannot create EJBReceiver since no cluster node manager found for node perf20 in cluster context for cluster ejb 16:05:29,974 INFO [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-2) Retrying invocation which failed on node perf20 with exception: javax.ejb.NoSuchEJBException: No such EJB[appname=clusterbench-ee6,modulename=clusterbench-ee6-ejb,distinctname=,beanname=RemoteStatelessSBImpl] at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64) at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:366) at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:458) at org.jboss.remoting3.remote.RemoteConnectionChannel$4.run(RemoteConnectionChannel.java:373) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 16:05:29,974 ERROR [org.jboss.ejb.client.ClusterContext] (ejb-client-cluster-node-connection-creation-2-thread-18) Cannot create EJBReceiver since no cluster node manager found for node perf20 in cluster context for cluster ejb 16:05:29,975 INFO [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-2) Retrying invocation which failed on node perf20 with exception: javax.ejb.NoSuchEJBException: No such EJB[appname=clusterbench-ee6,modulename=clusterbench-ee6-ejb,distinctname=,beanname=RemoteStatelessSBImpl] at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64) at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:366) at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:458) at org.jboss.remoting3.remote.RemoteConnectionChannel$4.run(RemoteConnectionChannel.java:373) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 16:05:29,975 INFO [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-2) Retrying invocation which failed on node perf20 with exception: javax.ejb.NoSuchEJBException: No such EJB[appname=clusterbench-ee6,modulename=clusterbench-ee6-ejb,distinctname=,beanname=RemoteStatelessSBImpl] at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64) at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:366) at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:458) at org.jboss.remoting3.remote.RemoteConnectionChannel$4.run(RemoteConnectionChannel.java:373) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) {noformat} During the whole test, there were 4 server crashes and startups (1 for each node), but only 3 occurences of the above mentioned error. These cluster nodes are perf18, perf19, perf20, perf21, but this error was seen only for perf20 (two occurences) and perf18 (one occurence). I did not find anything suspicious in the server.log of perf20. Cache: REPL_ASYNC Versions: EAP 6.1.0.ER6, ejb-client 1.0.19.Final Link to hudson job: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-... Server log: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-...