[jboss-jira] [JBoss JIRA] (WFLY-8033) Clients are not able to do failback in colocated HA topology with HTTP connectors

Clients are not able to do failback in colocated HA topology with HTTP connectors --------------------------------------------------------------------------------- Key: WFLY-8033 URL: https://issues.jboss.org/browse/WFLY-8033 Project: WildFly Issue Type: Bug Components: JMS Affects Versions: 10.1.0.Final Reporter: Jeff Mesnil Assignee: Jeff Mesnil Priority: Critical *Scenario*: * There are two EAPs configured in old way colocated HA topology * Start both servers * Connect clients to the first server * Kill first server * Wait until clients do failover * Start first server * Wait until clients do failback *Description of problem:* Sometimes happens that after the first server is started again, clients don't do failback. Producers are blocked because they don't have credits for sending. Receivers don't receive any messages. In trace logs I can see following messages: {code} 10:22:52,915 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:792] Trying reconnection attempt 403/-1 10:22:52,915 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:792] Trying reconnection attempt 403/-1 10:22:53,241 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1070] Trying to connect with connectorFactory = org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@6f7ca1c1, connectorConfig=TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor-backup&activemqServerName=backup&httpUpgradeEnabled=true&port=9080&host=localhost 10:22:53,241 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:542] Started Netty Connector version 4.0.35.Final-redhat-1 10:22:53,242 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:602] Remote destination: localhost/127.0.0.1:9080 10:22:52,743 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:792] Trying reconnection attempt 403/-1 10:22:53,242 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1070] Trying to connect with connectorFactory = org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@25572f4, connectorConfig=TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor-backup&activemqServerName=backup&httpUpgradeEnabled=true&port=9080&host=localhost 10:22:53,242 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:542] Started Netty Connector version 4.0.35.Final-redhat-1 10:22:53,242 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:602] Remote destination: localhost/127.0.0.1:9080 10:22:53,242 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:672] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1) GET HTTP/1.1 Host: localhost Upgrade: activemq-remoting Connection: Upgrade activemqServerName: backup httpUpgradeEndpoint: acceptor-backup Sec-ActiveMQRemoting-Key: bvTu4ZbJbAqWX5n3TwgNGA== 10:22:53,241 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1070] Trying to connect with connectorFactory = org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@1c5a232f, connectorConfig=TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor-backup&activemqServerName=backup&httpUpgradeEnabled=true&port=9080&host=localhost 10:22:53,242 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:542] Started Netty Connector version 4.0.35.Final-redhat-1 10:22:53,242 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:602] Remote destination: localhost/127.0.0.1:9080 10:22:53,241 main INFO [org.jboss.qa.hornetq.apps.clients.QueueClientsTransAck:178] Producer: Thread[Thread-22,5,main] is still alive. 10:22:53,243 main INFO [org.jboss.qa.hornetq.apps.clients.QueueClientsTransAck:178] Producer: Thread[Thread-23,5,main] is still alive. 10:22:53,243 main INFO [org.jboss.qa.hornetq.apps.clients.QueueClientsTransAck:191] Reciever: Thread[Thread-24,5,] is finished. 10:22:53,243 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:672] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1) GET HTTP/1.1 Host: localhost Upgrade: activemq-remoting Connection: Upgrade activemqServerName: backup httpUpgradeEndpoint: acceptor-backup Sec-ActiveMQRemoting-Key: vxNO5uMO4KK6etjAlAtorA== 10:22:53,243 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:672] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1) GET HTTP/1.1 Host: localhost Upgrade: activemq-remoting Connection: Upgrade activemqServerName: backup httpUpgradeEndpoint: acceptor-backup Sec-ActiveMQRemoting-Key: aUC+bE6WaMmU1A6KGcssww== 10:22:53,243 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1040] Connector towards NettyConnector [host=localhost, port=9080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed 10:22:53,244 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1081] Trying backup config = TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor-backup&activemqServerName=backup&httpUpgradeEnabled=true&port=8080&host=localhost 10:22:53,244 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:542] Started Netty Connector version 4.0.35.Final-redhat-1 10:22:53,244 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:602] Remote destination: localhost/127.0.0.1:8080 10:22:53,244 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1040] Connector towards NettyConnector [host=localhost, port=9080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed 10:22:53,244 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1040] Connector towards NettyConnector [host=localhost, port=9080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed 10:22:53,244 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:672] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1) GET HTTP/1.1 Host: localhost Upgrade: activemq-remoting Connection: Upgrade activemqServerName: backup httpUpgradeEndpoint: acceptor-backup Sec-ActiveMQRemoting-Key: plpRSFUDmbA3Xky/5BWx6g== 10:22:53,244 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1081] Trying backup config = TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor-backup&activemqServerName=backup&httpUpgradeEnabled=true&port=8080&host=localhost 10:22:53,244 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:542] Started Netty Connector version 4.0.35.Final-redhat-1 10:22:53,244 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:602] Remote destination: localhost/127.0.0.1:8080 10:22:53,244 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1081] Trying backup config = TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor-backup&activemqServerName=backup&httpUpgradeEnabled=true&port=8080&host=localhost 10:22:53,245 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:542] Started Netty Connector version 4.0.35.Final-redhat-1 10:22:53,245 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:602] Remote destination: localhost/127.0.0.1:8080 10:22:53,245 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1040] Connector towards NettyConnector [host=localhost, port=8080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed 10:22:53,245 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:672] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1) GET HTTP/1.1 Host: localhost Upgrade: activemq-remoting Connection: Upgrade activemqServerName: backup httpUpgradeEndpoint: acceptor-backup Sec-ActiveMQRemoting-Key: f2nehkPE0VDmGlWAcx3Ltg== 10:22:53,246 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:672] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1) GET HTTP/1.1 Host: localhost Upgrade: activemq-remoting Connection: Upgrade activemqServerName: backup httpUpgradeEndpoint: acceptor-backup Sec-ActiveMQRemoting-Key: RIaHZ2gy6JEm9nz6XOdyVA== 10:22:53,246 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1040] Connector towards NettyConnector [host=localhost, port=8080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed 10:22:53,246 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1104] Backup is not active. 10:22:53,246 Thread-1 (ActiveMQ-client-global-threads-1221943800) TRACE [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1258] Neither backup or live were active, will just give up now 10:22:53,245 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1104] Backup is not active. 10:22:53,246 Thread-2 (ActiveMQ-client-global-threads-1221943800) TRACE [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1258] Neither backup or live were active, will just give up now 10:22:53,246 Thread-2 (ActiveMQ-client-global-threads-1221943800) TRACE [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:815] Waiting 2000 milliseconds before next retry. RetryInterval=2000 and multiplier=1.0 {code} You can notice in trace logs that clients try to reconnect only to Artemis backup instances. They don't try any default (live) instance. At the time both Artemis Backup instances are inactive. I think if clients are initially connected to Artemis default instance on first server, they should try to reconnect only to this instance or to its backup. Note: The same scenario works with remote connectors/acceptors. *Customer impact:* Clients may hang after failback in colocated HA topology with HTTP connectors. If it happens, client's application must be restarted.