[JBoss JIRA] (JGRP-2461) Clustering can fail when re-adding an existing node using TCP_NIO2

Monday, 16 March 2020

    [
https://issues.redhat.com/browse/JGRP-2461?page=com.atlassian.jira.plugin...
] 

Robert Mitchell commented on JGRP-2461:
---------------------------------------

In detail, here is our channel configuration:

{code}
            channel = new JChannel(
                    // Primary transport
                    new TCP_NIO2()
                            .setBindPort(CLUSTER_PORT) // CLUSTER_PORT=7800
                            .setThreadPoolMaxThreads(20)
                            .setPortRange(0)

.setBindAddress(InetAddress.getByName(clusterManager.getLocalIpAddress())) // primary
external IP address
                            .level("TRACE")
                            .setValue("use_ip_addrs", true), // Make sure we
know each other's IP address
                    // Cluster identification and merging
                    new TCPPING()
                            .initialHosts(initialHosts()) // Provides 2 "well
known" hosts as the initial hosts
                            .portRange(0)
                            .level("TRACE")
                            .setValue("async_discovery", true),
                    new MERGE3().level("TRACE"),
                    // Failure detection
                    new FD_SOCK()
                            .setValue("start_port", 7825)
                            .setValue("port_range", 0)
                            .setValue("bind_addr",
InetAddress.getByName(clusterManager.getLocalIpAddress())),
                    new FD_ALL().timeout(30000),
                    new VERIFY_SUSPECT(),
                    // Security
                    new SYM_ENCRYPT().keystoreName("...keystore path..."),
                    new SERIALIZE(),  // Make sure the entire message is encrypted
                    // Ensure deliveries
                    new NAKACK2().setUseMcastXmit(false),
                    new UNICAST3(),
                    // Garbage collection
                    new STABLE(),
                    // Membership management
                    new
GMS().joinTimeout(2000).printLocalAddress(false).level("TRACE"),
                    // Flow control (non-blocking)
                    new UFC_NB(),
                    new MFC_NB(),
                    // Large message fragmentation
                    new FRAG2()
            );
{code}

...
 Clustering can fail when re-adding an existing node using TCP_NIO2
 ------------------------------------------------------------------

                 Key: JGRP-2461
                 URL: https://issues.redhat.com/browse/JGRP-2461
             Project: JGroups
          Issue Type: Bug
    Affects Versions: 4.1.8
            Reporter: Robert Mitchell
            Assignee: Bela Ban
            Priority: Major

 When a node leaves a cluster and then later attempts to re-enter, a race condition can
occur where the clustering fails to occur.  Here is the sequence of events that seems to
allow this to occur:
 # The rejoining node must have a "higher" IP address than the current cluster
coordinator.
 # On the rejoin attempt, the coordinator sends a message to the rejoining node before the
rejoining node sends to the coordinator using its prior address.  I have seen this happen
for two reasons:
 ## UNICAST3 is resending messages (which often happens with the final LEAVE_RSP from the
prior cluster membership because it apparently does not get acked before the connection
closes)
 ## TCPPING is sending a ping request to the cached prior address.
 # The connection gets established.  It will then be used by the rejoining node whenever
communicating with the cluster coordinator.
 #  However, the cluster coordinator has this as the connection for the prior address.  So
the following happens whenever it wants to send a message to the rejoining node:
 ## It will attempt to create a new connection.
 ## The rejoining node will reject the connection as a redundant connection with its
current connection taking precedence since it is coming from the same logical address as
the "bad" connection.
 Since the messages needed to find and join the cluster or merge the two clusters are all
unicast messages, the rejoining node will never get them and not be able to join until
something happens that causes the initial connection to get closed. 

--
This message was sent by Atlassian Jira
(v7.13.8#713008)

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006