[JBoss JIRA] (WFWIP-27) [Artemis 2.x Upgrade] Failover does not work with HTTP conncetors/acceptors
by Andy Taylor (JIRA)
[ https://issues.jboss.org/browse/WFWIP-27?page=com.atlassian.jira.plugin.s... ]
Andy Taylor commented on WFWIP-27:
----------------------------------
Ive debugged this with Jeffs 2.x upgrade branch and Im seeing an issue with the upgrade handler, basically the following happens
1. Live is started
2. backup is started and the upgrade handler is registered in HTTPUpgradeService::installService
3. live crashes
4. backup broker becomes live
5. live is restarted
6. failback occurs and th elive is restarted as a backup
when 6 happens HTTPUpgradeService::stop is called and the handler is removed. From this point on a 200 status is returned to any connecting client, even if the backup becomes live again it is never reregistered.
> [Artemis 2.x Upgrade] Failover does not work with HTTP conncetors/acceptors
> ---------------------------------------------------------------------------
>
> Key: WFWIP-27
> URL: https://issues.jboss.org/browse/WFWIP-27
> Project: WildFly WIP
> Issue Type: Bug
> Components: Artemis
> Reporter: Erich Duda
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: activemq, feature-branch-blocker
> Attachments: clients.log, server1.log, server2.log, standalone-full-ha.xml, standalone-full-ha.xml, test.log
>
>
> Failover does not work with HTTP connectors/acceptors.
> *Scenario:*
> # There are two Wildfly servers configured as Live-Backup pair
> # There is one JMS producer and one JMS receiver which sends/receives messages
> # Live server is several times killed and restarted.
> *Expectation:* Always when the Live server is killed or restarted, clients do failover or failback.
> *Reality*: Sometimes happens that clients don't do failover.
> *Users impact:* One of basics feature of HA, failover, does not work properly.
> *Blocker* priority was set because this is regression against previous releases of Wildfly.
> *Detail description of issue*
> In the trace logs it can be seen that clients send HTTP handshake request to active backup, but the handshake fails. All checks (and logs) say that the backup is active in this time period. I tried to run the test with Netty connectors/acceptors and I didn't see this issue.
> {code:title=Backup log}
> 07:52:47,856 WARN [org.apache.activemq.artemis.core.server] (Thread-2 (ActiveMQ-IO-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@1f01f347)) AMQ222029: Could not locate page transaction 3 995, ignoring message on position PagePositionImpl [pageNr=2, messageNr=55, recordID=-1] on address=jms.queue.testQueue queue=jms.queue.testQueue
> 07:52:47,856 WARN [org.apache.activemq.artemis.core.server] (Thread-2 (ActiveMQ-IO-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@1f01f347)) AMQ222029: Could not locate page transaction 3 995, ignoring message on position PagePositionImpl [pageNr=2, messageNr=56, recordID=-1] on address=jms.queue.testQueue queue=jms.queue.testQueue
> 07:52:47,863 INFO [org.apache.activemq.artemis.core.server] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=null) AMQ221005: Deleting pending large message as it was not completed: Pair[a=2147485068, b=2147485067]
> 07:52:47,863 INFO [org.apache.activemq.artemis.core.server] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=null) AMQ221005: Deleting pending large message as it was not completed: Pair[a=2147485076, b=2147485075]
> 07:52:47,864 INFO [org.apache.activemq.artemis.core.server] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=null) AMQ221005: Deleting pending large message as it was not completed: Pair[a=2147484450, b=2147484449]
> 07:52:47,864 INFO [org.apache.activemq.artemis.core.server] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=null) AMQ221005: Deleting pending large message as it was not completed: Pair[a=2147485072, b=2147485071]
> 07:52:47,864 INFO [org.apache.activemq.artemis.core.server] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=null) AMQ221005: Deleting pending large message as it was not completed: Pair[a=2147484454, b=2147484453]
> 07:52:47,866 WARN [org.apache.activemq.artemis.core.client] (activemq-discovery-group-thread-dg-group1) AMQ212034: There are more than one servers on the network broadcasting the same node id. You will see this message exactly once (per node) if a node is restarted, in which case it can be safely ignored. But if it is logged continuously it means you really do have more than one node on the same network active concurrently with the same node id. This could occur if you have a backup node active at the same time as its live node. nodeID=0dee81c9-2d9d-11e8-ba3f-cc3d825f79a4
> 07:52:47,867 INFO [org.apache.activemq.artemis.core.server] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=null) AMQ221007: Server is now live
> {code}
> {code:title=Clients log}
> 07:52:49,274 Thread-80 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:726] Remote destination: localhost/127.0.0.1:9080
> 07:52:49,274 Thread-79 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1040] Connector towards NettyConnector [host=localhost, port=8080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true, activemqServerName=default, httpUpgradeEndpoint=acceptor] failed
> 07:52:49,275 Thread-79 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1081] Trying backup config = TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor&activemqServerName=default&httpUpgradeEnabled=true&port=9080&host=localhost
> 07:52:49,275 Thread-79 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:447] Connector NettyConnector [host=localhost, port=9080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true, activemqServerName=default, httpUpgradeEndpoint=acceptor] using native epoll
> 07:52:49,275 Thread-80 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:797] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
> GET HTTP/1.1
> host: localhost
> upgrade: activemq-remoting
> connection: upgrade
> activemqServerName: default
> httpUpgradeEndpoint: acceptor
> Sec-ActiveMQRemoting-Key: QWV3bCfgh75NjWH3pZV5Ew==
> 07:52:49,275 Thread-79 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:601] AMQ211002: Started EPOLL Netty Connector version 4.1.16.Final to localhost:9080
> 07:52:49,275 Thread-79 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:726] Remote destination: localhost/127.0.0.1:9080
> 07:52:49,276 Thread-79 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:797] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
> GET HTTP/1.1
> host: localhost
> upgrade: activemq-remoting
> connection: upgrade
> activemqServerName: default
> httpUpgradeEndpoint: acceptor
> Sec-ActiveMQRemoting-Key: P9xBwRk1eZP5QjDWjqYuIg==
> 07:52:49,276 Thread-2 (ActiveMQ-client-netty-threads) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$HttpUpgradeHandler:876] Received msg=DefaultHttpResponse(decodeResult: success, version: HTTP/1.1)
> HTTP/1.1 200 OK
> Connection: keep-alive
> Last-Modified: Thu, 22 Mar 2018 06:47:03 GMT
> Content-Length: 2426
> Content-Type: text/html
> Accept-Ranges: bytes
> Date: Thu, 22 Mar 2018 06:52:49 GMT
> 07:52:49,276 Thread-2 (ActiveMQ-client-netty-threads) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$HttpUpgradeHandler:903] AMQ214023: HTTP Handshake failed, received DefaultHttpResponse(decodeResult: success, version: HTTP/1.1)
> HTTP/1.1 200 OK
> Connection: keep-alive
> Last-Modified: Thu, 22 Mar 2018 06:47:03 GMT
> Content-Length: 2426
> Content-Type: text/html
> Accept-Ranges: bytes
> Date: Thu, 22 Mar 2018 06:52:49 GMT
> 07:52:49,276 Thread-2 (ActiveMQ-client-netty-threads) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$HttpUpgradeHandler:876] Received msg=DefaultHttpContent(data: PooledSlicedByteBuf(ridx: 0, widx: 829, cap: 829/829, unwrapped: PooledUnsafeDirectByteBuf(ridx: 1024, widx: 1024, cap: 1024)), decoderResult: success)
> 07:52:49,276 Thread-2 (ActiveMQ-client-netty-threads) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$HttpUpgradeHandler:903] AMQ214023: HTTP Handshake failed, received DefaultHttpContent(data: PooledSlicedByteBuf(ridx: 0, widx: 829, cap: 829/829, unwrapped: PooledUnsafeDirectByteBuf(ridx: 1024, widx: 1024, cap: 1024)), decoderResult: success)
> 07:52:49,277 Thread-80 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1040] Connector towards NettyConnector [host=localhost, port=9080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true, activemqServerName=default, httpUpgradeEndpoint=acceptor] failed
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (WFLY-10629) Server startup wrongly deploys all xml files and never stops
by Kasimir Bobst (JIRA)
[ https://issues.jboss.org/browse/WFLY-10629?page=com.atlassian.jira.plugin... ]
Kasimir Bobst commented on WFLY-10629:
--------------------------------------
I had the same issue happening to me today. After having a look at the standalone.xml I noticed a second deployment-scanner that was added by jboss-tools:
<deployment-scanner name="jbosstoolsscanner1" path="C:\dev\appserver\wildfly\wildfly-10.1.0.Final\user_projects\domains\myDomain" scan-interval="5000" deployment-timeout="600"/>
After removing the scanner that scans the domain directory, wildfly would stop trying to deploy the standalone.xml file. I noticed a setting in the jboss-tools server overview under deployment scanners: There are two checkboxes
1) Add missing deployment scanners after server startup
2) Remove added deployment scanners before shutdown
After unchecking these no second deployment-scanner was added to my standalone.xml file anymore.
Regards
> Server startup wrongly deploys all xml files and never stops
> ------------------------------------------------------------
>
> Key: WFLY-10629
> URL: https://issues.jboss.org/browse/WFLY-10629
> Project: WildFly
> Issue Type: Bug
> Affects Versions: 13.0.0.Final
> Reporter: Arnie Morein
> Assignee: Jason Greene
>
> I have downloaded WF 12 and 13, then imported each into Eclipse (Oxygen.3a Release (4.7.3a)), OS is Win 7/x64; then used add-user to set up admin user. Then from both the command line and from within Eclipse, when the software starts up all of the xml configuration files get deployed at the same time despite the fact that the --server-config=standalone.xml parameter is present in either environment. What's worse, it starts versioning them and re-deploying them over and over and over again.
> {quote}
> 15:45:42,229 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 66) WFLYCLINF0002: Started client-mappings cache from ejb container
> 15:45:42,319 INFO [org.jboss.as.server] (ServerService Thread Pool -- 39) WFLYSRV0010: Deployed "standalone.xml" (runtime-name : "standalone.xml")
> 15:45:42,320 INFO [org.jboss.as.server] (ServerService Thread Pool -- 39) WFLYSRV0010: Deployed "standalone-load-balancer.xml" (runtime-name : "standalone-load-balancer.xml")
> 15:45:42,324 INFO [org.jboss.as.server] (ServerService Thread Pool -- 39) WFLYSRV0010: Deployed "standalone-ha.xml" (runtime-name : "standalone-ha.xml")
> 15:45:42,324 INFO [org.jboss.as.server] (ServerService Thread Pool -- 39) WFLYSRV0010: Deployed "standalone-full.xml" (runtime-name : "standalone-full.xml")
> 15:45:42,325 INFO [org.jboss.as.server] (ServerService Thread Pool -- 39) WFLYSRV0010: Deployed "standalone-full-ha.xml" (runtime-name : "standalone-full-ha.xml")
> 15:45:42,326 INFO [org.jboss.as.server] (ServerService Thread Pool -- 39) WFLYSRV0010: Deployed "standalone-ee8.xml" (runtime-name : "standalone-ee8.xml")
> 15:45:42,407 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
> 15:45:42,410 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:9990/management
> 15:45:42,411 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:9990
> 15:45:42,411 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 13.0.0.Final (WildFly Core 5.0.0.Final) started in 6285ms - Started 549 of 746 services (326 se
> rvices are lazy, passive or on-demand)
> 15:45:42,575 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) WFLYSRV0027: Starting deployment of "standalone.last.xml" (runtime-name: "standalone.last.xml")
> 15:45:42,578 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0027: Starting deployment of "standalone.boot.xml" (runtime-name: "standalone.boot.xml")
> 15:45:42,581 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) WFLYSRV0027: Starting deployment of "standalone.initial.xml" (runtime-name: "standalone.initial.xml")
> 15:45:42,724 INFO [org.jboss.as.server] (DeploymentScanner-threads - 1) WFLYSRV0010: Deployed "standalone.last.xml" (runtime-name : "standalone.last.xml")
> 15:45:42,725 INFO [org.jboss.as.server] (DeploymentScanner-threads - 1) WFLYSRV0010: Deployed "standalone.initial.xml" (runtime-name : "standalone.initial.xml")
> 15:45:42,726 INFO [org.jboss.as.server] (DeploymentScanner-threads - 1) WFLYSRV0010: Deployed "standalone.boot.xml" (runtime-name : "standalone.boot.xml")
> 15:45:47,921 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) WFLYSRV0028: Stopped deployment standalone.xml (runtime-name: standalone.xml) in 28ms
> 15:45:47,928 INFO [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0027: Starting deployment of "standalone.xml" (runtime-name: "standalone.xml")
> 15:45:47,930 INFO [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0027: Starting deployment of "standalone.v1.xml" (runtime-name: "standalone.v1.xml")
> 15:45:47,982 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) WFLYSRV0028: Stopped deployment standalone.last.xml (runtime-name: standalone.last.xml) in 55ms
> 15:45:47,984 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) WFLYSRV0027: Starting deployment of "standalone.last.xml" (runtime-name: "standalone.last.xml")
> 15:45:48,059 INFO [org.jboss.as.server] (DeploymentScanner-threads - 1) WFLYSRV0016: Replaced deployment "standalone.last.xml" with deployment "standalone.last.xml"
> 15:45:48,060 INFO [org.jboss.as.server] (DeploymentScanner-threads - 1) WFLYSRV0010: Deployed "standalone.v1.xml" (runtime-name : "standalone.v1.xml")
> 15:45:48,060 INFO [org.jboss.as.server] (DeploymentScanner-threads - 1) WFLYSRV0016: Replaced deployment "standalone.xml" with deployment "standalone.xml"
> 15:45:53,233 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) WFLYSRV0028: Stopped deployment standalone.xml (runtime-name: standalone.xml) in 13ms
> 15:45:53,236 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) WFLYSRV0027: Starting deployment of "standalone.v2.xml" (runtime-name: "standalone.v2.xml")
> 15:45:53,237 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) WFLYSRV0027: Starting deployment of "standalone.xml" (runtime-name: "standalone.xml")
> 15:45:53,249 INFO [org.jboss.as.server.deployment] (MSC service thread 1-5) WFLYSRV0028: Stopped deployment standalone.last.xml (runtime-name: standalone.last.xml) in 15ms
> 15:45:53,252 INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0027: Starting deployment of "standalone.last.xml" (runtime-name: "standalone.last.xml")
> 15:45:53,340 INFO [org.jboss.as.server] (DeploymentScanner-threads - 1) WFLYSRV0016: Replaced deployment "standalone.last.xml" with deployment "standalone.last.xml"
> 15:45:53,341 INFO [org.jboss.as.server] (DeploymentScanner-threads - 1) WFLYSRV0010: Deployed "standalone.v2.xml" (runtime-name : "standalone.v2.xml")
> 15:45:53,342 INFO [org.jboss.as.server] (DeploymentScanner-threads - 1) WFLYSRV0016: Replaced deployment "standalone.xml" with deployment "standalone.xml"{quote}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (JGRP-2227) Use of AUTH does not result in a SecurityException, but instead nodes create separate clusters with the same name
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-2227?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on JGRP-2227:
--------------------------------
Excellent, happy to see this finally works! Note that you'll only see an {{Exception}} in the latest release; {{SecurityException}} is only thrown by code in master (not released yet).
> Use of AUTH does not result in a SecurityException, but instead nodes create separate clusters with the same name
> -----------------------------------------------------------------------------------------------------------------
>
> Key: JGRP-2227
> URL: https://issues.jboss.org/browse/JGRP-2227
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 4.0.6
> Reporter: Robert Cernak
> Assignee: Bela Ban
> Fix For: 4.0.13
>
> Attachments: jgroupsDoesNotThrowSecurityExceptionWithJgroups4012.zip, jgroupsLogs.zip, traceLogs.zip
>
>
> I implemented method org.jgroups.auth.AuthToken#authenticate(AuthToken token, Message msg) in my class and its body contained only one line: return false;
> In this way authentication should be false and I should get SecurityException.
> When I started joining of nodes together to form a cluster, instead of getting SecurityException, nodes formed 2 different clusters with the same name.
> I am sure method was evaluated, since I tried to run it also with breakpoint, which was triggered during joining process.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (JGRP-2227) Use of AUTH does not result in a SecurityException, but instead nodes create separate clusters with the same name
by Robert Cernak (JIRA)
[ https://issues.jboss.org/browse/JGRP-2227?page=com.atlassian.jira.plugin.... ]
Robert Cernak commented on JGRP-2227:
-------------------------------------
[~belaban] Thanks for your support. After changing configuration according your notes from previous comment, I was finally able to catch SecurityException.
> Use of AUTH does not result in a SecurityException, but instead nodes create separate clusters with the same name
> -----------------------------------------------------------------------------------------------------------------
>
> Key: JGRP-2227
> URL: https://issues.jboss.org/browse/JGRP-2227
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 4.0.6
> Reporter: Robert Cernak
> Assignee: Bela Ban
> Fix For: 4.0.13
>
> Attachments: jgroupsDoesNotThrowSecurityExceptionWithJgroups4012.zip, jgroupsLogs.zip, traceLogs.zip
>
>
> I implemented method org.jgroups.auth.AuthToken#authenticate(AuthToken token, Message msg) in my class and its body contained only one line: return false;
> In this way authentication should be false and I should get SecurityException.
> When I started joining of nodes together to form a cluster, instead of getting SecurityException, nodes formed 2 different clusters with the same name.
> I am sure method was evaluated, since I tried to run it also with breakpoint, which was triggered during joining process.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years