[jboss-jira] [JBoss JIRA] (WFLY-8033) Clients are not able to do failback in colocated HA topology with HTTP connectors
Jeff Mesnil (JIRA)
issues at jboss.org
Mon Feb 6 08:52:00 EST 2017
[ https://issues.jboss.org/browse/WFLY-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Mesnil moved JBEAP-8682 to WFLY-8033:
------------------------------------------
Project: WildFly (was: JBoss Enterprise Application Platform)
Key: WFLY-8033 (was: JBEAP-8682)
Workflow: GIT Pull Request workflow (was: CDW with loose statuses v1)
Component/s: JMS
(was: JMS)
Affects Version/s: 10.1.0.Final
(was: 7.1.0.DR10)
(was: 7.1.0.DR11)
> Clients are not able to do failback in colocated HA topology with HTTP connectors
> ---------------------------------------------------------------------------------
>
> Key: WFLY-8033
> URL: https://issues.jboss.org/browse/WFLY-8033
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Affects Versions: 10.1.0.Final
> Reporter: Jeff Mesnil
> Assignee: Jeff Mesnil
> Priority: Critical
>
> *Scenario*:
> * There are two EAPs configured in old way colocated HA topology
> * Start both servers
> * Connect clients to the first server
> * Kill first server
> * Wait until clients do failover
> * Start first server
> * Wait until clients do failback
> *Description of problem:*
> Sometimes happens that after the first server is started again, clients don't do failback. Producers are blocked because they don't have credits for sending. Receivers don't receive any messages.
> In trace logs I can see following messages:
> {code}
> 10:22:52,915 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:792] Trying reconnection attempt 403/-1
> 10:22:52,915 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:792] Trying reconnection attempt 403/-1
> 10:22:53,241 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1070] Trying to connect with connectorFactory = org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory at 6f7ca1c1, connectorConfig=TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor-backup&activemqServerName=backup&httpUpgradeEnabled=true&port=9080&host=localhost
> 10:22:53,241 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:542] Started Netty Connector version 4.0.35.Final-redhat-1
> 10:22:53,242 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:602] Remote destination: localhost/127.0.0.1:9080
> 10:22:52,743 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:792] Trying reconnection attempt 403/-1
> 10:22:53,242 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1070] Trying to connect with connectorFactory = org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory at 25572f4, connectorConfig=TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor-backup&activemqServerName=backup&httpUpgradeEnabled=true&port=9080&host=localhost
> 10:22:53,242 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:542] Started Netty Connector version 4.0.35.Final-redhat-1
> 10:22:53,242 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:602] Remote destination: localhost/127.0.0.1:9080
> 10:22:53,242 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:672] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
> GET HTTP/1.1
> Host: localhost
> Upgrade: activemq-remoting
> Connection: Upgrade
> activemqServerName: backup
> httpUpgradeEndpoint: acceptor-backup
> Sec-ActiveMQRemoting-Key: bvTu4ZbJbAqWX5n3TwgNGA==
> 10:22:53,241 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1070] Trying to connect with connectorFactory = org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory at 1c5a232f, connectorConfig=TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor-backup&activemqServerName=backup&httpUpgradeEnabled=true&port=9080&host=localhost
> 10:22:53,242 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:542] Started Netty Connector version 4.0.35.Final-redhat-1
> 10:22:53,242 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:602] Remote destination: localhost/127.0.0.1:9080
> 10:22:53,241 main INFO [org.jboss.qa.hornetq.apps.clients.QueueClientsTransAck:178] Producer: Thread[Thread-22,5,main] is still alive.
> 10:22:53,243 main INFO [org.jboss.qa.hornetq.apps.clients.QueueClientsTransAck:178] Producer: Thread[Thread-23,5,main] is still alive.
> 10:22:53,243 main INFO [org.jboss.qa.hornetq.apps.clients.QueueClientsTransAck:191] Reciever: Thread[Thread-24,5,] is finished.
> 10:22:53,243 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:672] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
> GET HTTP/1.1
> Host: localhost
> Upgrade: activemq-remoting
> Connection: Upgrade
> activemqServerName: backup
> httpUpgradeEndpoint: acceptor-backup
> Sec-ActiveMQRemoting-Key: vxNO5uMO4KK6etjAlAtorA==
> 10:22:53,243 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:672] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
> GET HTTP/1.1
> Host: localhost
> Upgrade: activemq-remoting
> Connection: Upgrade
> activemqServerName: backup
> httpUpgradeEndpoint: acceptor-backup
> Sec-ActiveMQRemoting-Key: aUC+bE6WaMmU1A6KGcssww==
> 10:22:53,243 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1040] Connector towards NettyConnector [host=localhost, port=9080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
> 10:22:53,244 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1081] Trying backup config = TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor-backup&activemqServerName=backup&httpUpgradeEnabled=true&port=8080&host=localhost
> 10:22:53,244 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:542] Started Netty Connector version 4.0.35.Final-redhat-1
> 10:22:53,244 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:602] Remote destination: localhost/127.0.0.1:8080
> 10:22:53,244 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1040] Connector towards NettyConnector [host=localhost, port=9080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
> 10:22:53,244 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1040] Connector towards NettyConnector [host=localhost, port=9080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
> 10:22:53,244 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:672] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
> GET HTTP/1.1
> Host: localhost
> Upgrade: activemq-remoting
> Connection: Upgrade
> activemqServerName: backup
> httpUpgradeEndpoint: acceptor-backup
> Sec-ActiveMQRemoting-Key: plpRSFUDmbA3Xky/5BWx6g==
> 10:22:53,244 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1081] Trying backup config = TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor-backup&activemqServerName=backup&httpUpgradeEnabled=true&port=8080&host=localhost
> 10:22:53,244 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:542] Started Netty Connector version 4.0.35.Final-redhat-1
> 10:22:53,244 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:602] Remote destination: localhost/127.0.0.1:8080
> 10:22:53,244 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1081] Trying backup config = TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor-backup&activemqServerName=backup&httpUpgradeEnabled=true&port=8080&host=localhost
> 10:22:53,245 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:542] Started Netty Connector version 4.0.35.Final-redhat-1
> 10:22:53,245 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:602] Remote destination: localhost/127.0.0.1:8080
> 10:22:53,245 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1040] Connector towards NettyConnector [host=localhost, port=8080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
> 10:22:53,245 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:672] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
> GET HTTP/1.1
> Host: localhost
> Upgrade: activemq-remoting
> Connection: Upgrade
> activemqServerName: backup
> httpUpgradeEndpoint: acceptor-backup
> Sec-ActiveMQRemoting-Key: f2nehkPE0VDmGlWAcx3Ltg==
> 10:22:53,246 Thread-3 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:672] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
> GET HTTP/1.1
> Host: localhost
> Upgrade: activemq-remoting
> Connection: Upgrade
> activemqServerName: backup
> httpUpgradeEndpoint: acceptor-backup
> Sec-ActiveMQRemoting-Key: RIaHZ2gy6JEm9nz6XOdyVA==
> 10:22:53,246 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1040] Connector towards NettyConnector [host=localhost, port=8080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
> 10:22:53,246 Thread-1 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1104] Backup is not active.
> 10:22:53,246 Thread-1 (ActiveMQ-client-global-threads-1221943800) TRACE [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1258] Neither backup or live were active, will just give up now
> 10:22:53,245 Thread-2 (ActiveMQ-client-global-threads-1221943800) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1104] Backup is not active.
> 10:22:53,246 Thread-2 (ActiveMQ-client-global-threads-1221943800) TRACE [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1258] Neither backup or live were active, will just give up now
> 10:22:53,246 Thread-2 (ActiveMQ-client-global-threads-1221943800) TRACE [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:815] Waiting 2000 milliseconds before next retry. RetryInterval=2000 and multiplier=1.0
> {code}
> You can notice in trace logs that clients try to reconnect only to Artemis backup instances. They don't try any default (live) instance. At the time both Artemis Backup instances are inactive.
> I think if clients are initially connected to Artemis default instance on first server, they should try to reconnect only to this instance or to its backup.
> Note: The same scenario works with remote connectors/acceptors.
> *Customer impact:* Clients may hang after failback in colocated HA topology with HTTP connectors. If it happens, client's application must be restarted.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
More information about the jboss-jira
mailing list