[jboss-jira] [JBoss JIRA] (WFLY-12214) JGRP000029: failed sending message: java.net.ConnectException: Connection refused

Fri Jun 21 09:25:00 EDT 2019

     [ https://issues.jboss.org/browse/WFLY-12214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tommasso Borgato updated WFLY-12214:
------------------------------------
    Description: 
The error is observed in fail-over clustering tests where fail-over is "shutdown" and the jgroups stack is TCP.

The error was *not* observed in:

* wildfly-17.0.0.Final.zip
* wildfly-16.0.0.Final.zip

Right after one node (wildfly1) is shut down and restarted and the next node (wildfly2) is shut down we see:

{noformat}
2019-06-20 10:55:07,021 INFO  [org.infinispan.CLUSTER] (thread-84,ejb,wildfly1) ISPN000094: Received new cluster view for channel ejb: [wildfly3|6] (3) [wildfly3, wildfly4, wildfly1]
2019-06-20 10:55:07,022 INFO  [org.infinispan.CLUSTER] (thread-84,ejb,wildfly1) ISPN100001: Node wildfly2 left the cluster
2019-06-20 10:55:07,109 ERROR [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,wildfly1) JGRP000029: wildfly1: failed sending message to wildfly2 (133 bytes): java.net.ConnectException: Connection refused (Connection refused), headers: FORK: ejb:web, UNICAST3: DATA, seqno=44773, TP: [cluster=ejb]
2019-06-20 10:55:07,251 ERROR [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,wildfly1) JGRP000029: wildfly1: failed sending message to wildfly2 (133 bytes): java.net.ConnectException: Connection refused (Connection refused), headers: FORK: ejb:web, UNICAST3: DATA, seqno=44773, TP: [cluster=ejb]
2019-06-20 10:55:07,342 ERROR [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,wildfly1) JGRP000029: wildfly1: failed sending message to wildfly2 (133 bytes): java.net.ConnectException: Connection refused (Connection refused), headers: FORK: ejb:web, UNICAST3: DATA, seqno=44773, TP: [cluster=ejb]
2019-06-20 10:55:07,437 ERROR [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,wildfly1) JGRP000029: wildfly1: failed sending message to wildfly2 (133 bytes): java.net.ConnectException: Connection refused (Connection refused), headers: FORK: ejb:web, UNICAST3: DATA, seqno=44773, TP: [cluster=ejb]
{noformat}

Complete logs:
* [eap-7.x-clustering-db-session-shutdown-repl-mssql-2016#2|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/EAP7/view/EAP7-Clustering/view/EAP7-Clustering-Database/job/eap-7.x-clustering-db-session-shutdown-repl-mssql-2016/2/artifact/report/wildfly/wlf_20194320-104347-wildfly-service-1-server.log/*view*/]
* [eap-7.x-clustering-http-session-granular-dist-jvmkill-haproxy#4|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/EAP7/view/EAP7-Clustering/view/EAP7-Clustering-HTTP-Session-Granular/job/eap-7.x-clustering-http-session-granular-dist-jvmkill-haproxy/4/artifact/report/wildfly/wlf_20193120-123102-wildfly-service-1-server.log/*view*/]
* [eap-7.x-clustering-http-session-shutdown-scattered#10|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/EAP7/view/EAP7-Clustering/view/EAP7-Clustering-HTTP-Session/job/eap-7.x-clustering-http-session-shutdown-scattered/10/artifact/report/wildfly/wlf_20193321-123338-wildfly-service-1-server.log/*view*/]

The number of errors in the log files is elevated: about 1000-2000 per node;

Overall fail-rate is still close to 0%.

  was:
The error is observed in fail-over clustering tests where fail-over is "shutdown" and the jgroups stack is TCP.

The error was *not* observed in:

* wildfly-17.0.0.Final.zip
* wildfly-16.0.0.Final.zip

Right after one node (wildfly1) is shut down and restarted and the next node (wildfly2) is shut down we see:

{noformat}
2019-06-20 10:55:07,021 INFO  [org.infinispan.CLUSTER] (thread-84,ejb,wildfly1) ISPN000094: Received new cluster view for channel ejb: [wildfly3|6] (3) [wildfly3, wildfly4, wildfly1]
2019-06-20 10:55:07,022 INFO  [org.infinispan.CLUSTER] (thread-84,ejb,wildfly1) ISPN100001: Node wildfly2 left the cluster
2019-06-20 10:55:07,109 ERROR [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,wildfly1) JGRP000029: wildfly1: failed sending message to wildfly2 (133 bytes): java.net.ConnectException: Connection refused (Connection refused), headers: FORK: ejb:web, UNICAST3: DATA, seqno=44773, TP: [cluster=ejb]
2019-06-20 10:55:07,251 ERROR [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,wildfly1) JGRP000029: wildfly1: failed sending message to wildfly2 (133 bytes): java.net.ConnectException: Connection refused (Connection refused), headers: FORK: ejb:web, UNICAST3: DATA, seqno=44773, TP: [cluster=ejb]
2019-06-20 10:55:07,342 ERROR [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,wildfly1) JGRP000029: wildfly1: failed sending message to wildfly2 (133 bytes): java.net.ConnectException: Connection refused (Connection refused), headers: FORK: ejb:web, UNICAST3: DATA, seqno=44773, TP: [cluster=ejb]
2019-06-20 10:55:07,437 ERROR [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,wildfly1) JGRP000029: wildfly1: failed sending message to wildfly2 (133 bytes): java.net.ConnectException: Connection refused (Connection refused), headers: FORK: ejb:web, UNICAST3: DATA, seqno=44773, TP: [cluster=ejb]
{noformat}

Complete logs:
* [eap-7.x-clustering-db-session-shutdown-repl-mssql-2016#2|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/EAP7/view/EAP7-Clustering/view/EAP7-Clustering-Database/job/eap-7.x-clustering-db-session-shutdown-repl-mssql-2016/2/artifact/report/wildfly/wlf_20194320-104347-wildfly-service-1-server.log/*view*/]
* [eap-7.x-clustering-http-session-granular-dist-jvmkill-haproxy#4|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/EAP7/view/EAP7-Clustering/view/EAP7-Clustering-HTTP-Session-Granular/job/eap-7.x-clustering-http-session-granular-dist-jvmkill-haproxy/4/artifact/report/wildfly/wlf_20193120-123102-wildfly-service-1-server.log/*view*/]
* [eap-7.x-clustering-http-session-shutdown-scattered#10|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/EAP7/view/EAP7-Clustering/view/EAP7-Clustering-HTTP-Session/job/eap-7.x-clustering-http-session-shutdown-scattered/10/artifact/report/wildfly/wlf_20193321-123338-wildfly-service-1-server.log/*view*/]

The number of errors in the log files is *elevated*: about 1000-2000 per node;

Overall fail-rate is still close to 0%.

> JGRP000029: failed sending message: java.net.ConnectException: Connection refused
> ---------------------------------------------------------------------------------
>
>                 Key: WFLY-12214
>                 URL: https://issues.jboss.org/browse/WFLY-12214
>             Project: WildFly
>          Issue Type: Bug
>          Components: Clustering
>    Affects Versions: 18.0.0.Beta1
>            Reporter: Tommasso Borgato
>            Assignee: Paul Ferraro
>            Priority: Major
>
> The error is observed in fail-over clustering tests where fail-over is "shutdown" and the jgroups stack is TCP.
> The error was *not* observed in:
> * wildfly-17.0.0.Final.zip
> * wildfly-16.0.0.Final.zip
> Right after one node (wildfly1) is shut down and restarted and the next node (wildfly2) is shut down we see:
> {noformat}
> 2019-06-20 10:55:07,021 INFO  [org.infinispan.CLUSTER] (thread-84,ejb,wildfly1) ISPN000094: Received new cluster view for channel ejb: [wildfly3|6] (3) [wildfly3, wildfly4, wildfly1]
> 2019-06-20 10:55:07,022 INFO  [org.infinispan.CLUSTER] (thread-84,ejb,wildfly1) ISPN100001: Node wildfly2 left the cluster
> 2019-06-20 10:55:07,109 ERROR [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,wildfly1) JGRP000029: wildfly1: failed sending message to wildfly2 (133 bytes): java.net.ConnectException: Connection refused (Connection refused), headers: FORK: ejb:web, UNICAST3: DATA, seqno=44773, TP: [cluster=ejb]
> 2019-06-20 10:55:07,251 ERROR [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,wildfly1) JGRP000029: wildfly1: failed sending message to wildfly2 (133 bytes): java.net.ConnectException: Connection refused (Connection refused), headers: FORK: ejb:web, UNICAST3: DATA, seqno=44773, TP: [cluster=ejb]
> 2019-06-20 10:55:07,342 ERROR [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,wildfly1) JGRP000029: wildfly1: failed sending message to wildfly2 (133 bytes): java.net.ConnectException: Connection refused (Connection refused), headers: FORK: ejb:web, UNICAST3: DATA, seqno=44773, TP: [cluster=ejb]
> 2019-06-20 10:55:07,437 ERROR [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,wildfly1) JGRP000029: wildfly1: failed sending message to wildfly2 (133 bytes): java.net.ConnectException: Connection refused (Connection refused), headers: FORK: ejb:web, UNICAST3: DATA, seqno=44773, TP: [cluster=ejb]
> {noformat}
> Complete logs:
> * [eap-7.x-clustering-db-session-shutdown-repl-mssql-2016#2|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/EAP7/view/EAP7-Clustering/view/EAP7-Clustering-Database/job/eap-7.x-clustering-db-session-shutdown-repl-mssql-2016/2/artifact/report/wildfly/wlf_20194320-104347-wildfly-service-1-server.log/*view*/]
> * [eap-7.x-clustering-http-session-granular-dist-jvmkill-haproxy#4|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/EAP7/view/EAP7-Clustering/view/EAP7-Clustering-HTTP-Session-Granular/job/eap-7.x-clustering-http-session-granular-dist-jvmkill-haproxy/4/artifact/report/wildfly/wlf_20193120-123102-wildfly-service-1-server.log/*view*/]
> * [eap-7.x-clustering-http-session-shutdown-scattered#10|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/EAP7/view/EAP7-Clustering/view/EAP7-Clustering-HTTP-Session/job/eap-7.x-clustering-http-session-shutdown-scattered/10/artifact/report/wildfly/wlf_20193321-123338-wildfly-service-1-server.log/*view*/]
> The number of errors in the log files is elevated: about 1000-2000 per node;
> Overall fail-rate is still close to 0%.

--
This message was sent by Atlassian Jira
(v7.12.1#712002)