Server with HA profile bound to 0.0.0.0 sends cluster topology containing this address to client ------------------------------------------------------------------------------------------------ Key: WFLY-11235 URL: https://issues.jboss.org/browse/WFLY-11235 Project: WildFly Issue Type: Bug Components: EJB Reporter: Lin Gao Assignee: Cheng Fang Priority: Minor Labels: downstream_dependency Attachments: WFLY-11235-reproducer.zip Original Estimate: 4 days Remaining Estimate: 4 days Scenario: - server-1 -- Doesn't matter which profile -- Doesn't matter which bind address -- Has a remote-outbound-connection pointing at server-2 through server-2's public IP address - server-2 -- On a different machine than server-1 -- Bound to 0.0.0.0 (important) -- Running standalone-ha profile (important) - Standalone EJB client invokes a bean server-1, which as a result invokes another bean on server-2. Repeat this multiple times. The first invocation goes through correctly, but it can be seen in the log of server-1 that server-2 (10.8.246.90) has sent a cluster topology containing 0.0.0.0: {noformat} 04:04:24,651 TRACE [org.jboss.ejb.client.remoting] (default task-5) Running callback org.jboss.ejb.protocol.remote.RemoteEJBService$1$1$$Lambda$876/915175573@68823b7b(org.jboss.ejb.protocol.remote.EJBServerChannel(a)707ad04e, org.jboss.as.ejb3.remote.AssociationImpl@3eb54e9a) 04:04:24,919 INFO [org.wildfly.naming] (default task-7) WildFly Naming version 1.0.7.Final-redhat-1 04:04:24,930 INFO [org.jboss.ejb.client] (default task-7) JBoss EJB Client version 4.0.9.Final-redhat-1 04:04:24,941 DEBUG [org.jboss.ejb.client.invocation] (default task-7) Calling invoke(module = /server-side/WhoAmIBean, strong affinity = None, weak affinity = None): 04:04:24,948 TRACE [org.jboss.ejb.client.invocation] (default task-7) Calling executeDiscovery(locator = StatelessEJBLocator for "/server-side/WhoAmIBean", view is interface example.ejb.WhoAmIBeanRemote, affinity is None, weak affinity = None) 04:04:24,949 TRACE [org.jboss.ejb.client.invocation] (default task-7) Performing any discovery(locator = StatelessEJBLocator for "/server-side/WhoAmIBean", view is interface example.ejb.WhoAmIBeanRemote, affinity is None, weak affinity = None, filter spec = (ejb-module=server-side)) 04:04:24,952 TRACE [org.jboss.ejb.client.invocation] (default task-7) EJB discovery provider: attempting to connect to configured connection remote+http://10.8.246.90:8080 04:04:25,317 DEBUG [org.jboss.ejb.client.invocation] (default task-18) Received CLUSTER_TOPOLOGY(15) message, registering cluster ejb to node localhost 04:04:25,319 DEBUG [org.jboss.ejb.client.invocation] (default task-18) Received CLUSTER_TOPOLOGY(15) message block, registering block ::/0 to address /0.0.0.0:8080 04:04:25,321 DEBUG [org.jboss.ejb.client.invocation] (default task-19) Received MODULE_AVAILABLE(8) message for module /server-side 04:04:25,324 TRACE [org.jboss.ejb.client.invocation] (default task-7) Performed first-match discovery(target affinity(node) = localhost, destination = remote+http://10.8.246.90:8080) 04:04:25,325 TRACE [org.jboss.ejb.client] (default task-7) Transitioning org.jboss.ejb.client.EJBClientInvocationContext@5ffb94cb from SENDING to SENT (org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:450)) 04:04:25,326 TRACE [org.jboss.ejb.client] (default task-7) Transitioning org.jboss.ejb.client.EJBClientInvocationContext@5ffb94cb from SENT to WAITING (org.jboss.ejb.client.EJBClientInvocationContext.sendRequestInitial(EJBClientInvocationContext.java:339)) 04:04:25,345 TRACE [org.jboss.ejb.client] (default task-22) Transitioning org.jboss.ejb.client.EJBClientInvocationContext@5ffb94cb from WAITING to READY (org.jboss.ejb.client.EJBClientInvocationContext.resultReady(EJBClientInvocationContext.java:646)) 04:04:25,345 TRACE [org.jboss.ejb.client] (default task-7) Transitioning org.jboss.ejb.client.EJBClientInvocationContext@5ffb94cb from READY to CONSUMING (org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:546)) 04:04:25,346 TRACE [org.jboss.ejb.client] (default task-7) Transitioning org.jboss.ejb.client.EJBClientInvocationContext@5ffb94cb from CONSUMING to DONE (org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:573)) {noformat} After the first invocation, the bean proxy on server-1 sets it affinity to a ClusterAffinity, which means that the next invocation's destination is picked based on the cluster topology (which contains only 0.0.0.0). It will most likely fail, and after this, the affinity seems to be reset back to None, therefore we have a loop where always one invocation works correctly and the following one fails. The following is the log observed on server-1 when an invocation tries to go to 0.0.0.0 (so it either fails with connection refused, or server-1 tries to invoke itself in case that it's listening on 0.0.0.0 too) {noformat} 04:04:26,485 DEBUG [org.jboss.ejb.client.invocation] (default task-24) Calling invoke(module = /server-side/WhoAmIBean, strong affinity = Cluster "ejb", weak affinity = None): 04:04:26,485 TRACE [org.jboss.ejb.client.invocation] (default task-24) Calling executeDiscovery(locator = StatelessEJBLocator for "/server-side/WhoAmIBean", view is interface example.ejb.WhoAmIBeanRemote, affinity is Cluster "ejb", weak affinity = None) 04:04:26,486 TRACE [org.jboss.ejb.client.invocation] (default task-24) Performing cluster discovery(locator = StatelessEJBLocator for "/server-side/WhoAmIBean", view is interface example.ejb.WhoAmIBeanRemote, affinity is Cluster "ejb", weak affinity = None, filter spec = (&(cluster=ejb))) 04:04:26,486 TRACE [org.jboss.ejb.client.invocation] (default task-24) EJB discovery provider: attempting to connect to configured connection remote+http://10.8.246.90:8080 04:04:26,488 TRACE [org.jboss.ejb.client.invocation] (default task-24) EJB discovery provider: attempting to connect to cluster ejb connection remote+http://0.0.0.0:8080 04:04:26,492 TRACE [org.jboss.ejb.client.invocation] (default task-24) Performed cluster discovery (target affinity = Node "localhost", destination = remote+http://0.0.0.0:8080) 04:04:26,493 TRACE [org.jboss.ejb.client] (default task-24) Transitioning org.jboss.ejb.client.EJBClientInvocationContext@386576b9 from SENDING to SENT (org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:450)) 04:04:26,495 TRACE [org.jboss.ejb.client] (default task-24) Transitioning org.jboss.ejb.client.EJBClientInvocationContext@386576b9 from SENT to WAITING (org.jboss.ejb.client.EJBClientInvocationContext.sendRequestInitial(EJBClientInvocationContext.java:339)) 04:04:26,497 TRACE [org.jboss.ejb.client] (default task-32) Invocation marked failed, state is currently: WAITING 04:04:26,498 TRACE [org.jboss.ejb.client] (default task-32) Transitioning org.jboss.ejb.client.EJBClientInvocationContext@386576b9 from WAITING to CONSUMING (org.jboss.ejb.client.EJBClientInvocationContext.failed(EJBClientInvocationContext.java:975)) 04:04:26,498 TRACE [org.jboss.ejb.client.invocation] (default task-32) Blacklisting destination (locator = StatelessEJBLocator for "/server-side/WhoAmIBean", view is interface example.ejb.WhoAmIBeanRemote, affinity is Cluster "ejb", weak affinity = None, missing target = remote+http://0.0.0.0:8080) 04:04:26,499 TRACE [org.jboss.ejb.client] (default task-32) Transitioning org.jboss.ejb.client.EJBClientInvocationContext@386576b9 from CONSUMING to SENDING (org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:600)) 04:04:26,499 TRACE [org.jboss.ejb.client.invocation] (default task-32) Calling executeDiscovery(locator = StatelessEJBLocator for "/server-side/WhoAmIBean", view is interface example.ejb.WhoAmIBeanRemote, affinity is Cluster "ejb", weak affinity = None) 04:04:26,499 TRACE [org.jboss.ejb.client.invocation] (default task-32) Performing cluster discovery(locator = StatelessEJBLocator for "/server-side/WhoAmIBean", view is interface example.ejb.WhoAmIBeanRemote, affinity is Cluster "ejb", weak affinity = None, filter spec = (&(cluster=ejb))) 04:04:26,501 TRACE [org.jboss.ejb.client.invocation] (default task-32) Performed cluster discovery, nodes is empty; trying an initial 04:04:26,502 TRACE [org.jboss.ejb.client] (default task-32) Transitioning org.jboss.ejb.client.EJBClientInvocationContext@386576b9 from SENDING to SENT (org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:442)) 04:04:26,503 TRACE [org.jboss.ejb.client] (default task-32) Transitioning org.jboss.ejb.client.EJBClientInvocationContext@386576b9 from SENT to READY (org.jboss.ejb.client.EJBClientInvocationContext.sendRequestInitial(EJBClientInvocationContext.java:380)) 04:04:26,503 TRACE [org.jboss.ejb.client] (default task-24) Transitioning org.jboss.ejb.client.EJBClientInvocationContext@386576b9 from READY to CONSUMING (org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:546)) 04:04:26,503 TRACE [org.jboss.ejb.client] (default task-24) Will not retry (requested = false, remaining = 0) 04:04:26,504 TRACE [org.jboss.ejb.client] (default task-24) Transitioning org.jboss.ejb.client.EJBClientInvocationContext@386576b9 from CONSUMING to DONE (org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:613)) 04:04:26,505 ERROR [org.jboss.as.ejb3.invocation] (default task-24) WFLYEJB0034: EJB Invocation failed on component Intermediate for method public abstract java.lang.String example.ejb.WhoAmIBeanRemote.whoAmI(): javax.ejb.NoSuchEJBException: EJBCLIENT000079: Unable to discover destination for request for EJB StatelessEJBLocator for "/server-side/WhoAmIBean", view is interface example.ejb.WhoAmIBeanRemote, affinity is Cluster "ejb" (... rest of the error omitted, but it is a "connection refused") {noformat}