[jboss-jira] [JBoss JIRA] (JGRP-2461) Clustering can fail when re-adding an existing node using TCP_NIO2

[Robert Mitchell (Jira)]

    [ https://issues.redhat.com/browse/JGRP-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000284#comment-14000284 ] 

Robert Mitchell commented on JGRP-2461:
---------------------------------------

In detail, here is our channel configuration:

{code}
            channel = new JChannel(
                    // Primary transport
                    new TCP_NIO2()
                            .setBindPort(CLUSTER_PORT) // CLUSTER_PORT=7800
                            .setThreadPoolMaxThreads(20)
                            .setPortRange(0)
                            .setBindAddress(InetAddress.getByName(clusterManager.getLocalIpAddress())) // primary external IP address
                            .level("TRACE")
                            .setValue("use_ip_addrs", true), // Make sure we know each other's IP address
                    // Cluster identification and merging
                    new TCPPING()
                            .initialHosts(initialHosts()) // Provides 2 "well known" hosts as the initial hosts
                            .portRange(0)
                            .level("TRACE")
                            .setValue("async_discovery", true),
                    new MERGE3().level("TRACE"),
                    // Failure detection
                    new FD_SOCK()
                            .setValue("start_port", 7825)
                            .setValue("port_range", 0)
                            .setValue("bind_addr", InetAddress.getByName(clusterManager.getLocalIpAddress())),
                    new FD_ALL().timeout(30000),
                    new VERIFY_SUSPECT(),
                    // Security
                    new SYM_ENCRYPT().keystoreName("...keystore path..."),
                    new SERIALIZE(),  // Make sure the entire message is encrypted
                    // Ensure deliveries
                    new NAKACK2().setUseMcastXmit(false),
                    new UNICAST3(),
                    // Garbage collection
                    new STABLE(),
                    // Membership management
                    new GMS().joinTimeout(2000).printLocalAddress(false).level("TRACE"),
                    // Flow control (non-blocking)
                    new UFC_NB(),
                    new MFC_NB(),
                    // Large message fragmentation
                    new FRAG2()
            );
{code}

> Clustering can fail when re-adding an existing node using TCP_NIO2
> ------------------------------------------------------------------
>
>                 Key: JGRP-2461
>                 URL: https://issues.redhat.com/browse/JGRP-2461
>             Project: JGroups
>          Issue Type: Bug
>    Affects Versions: 4.1.8
>            Reporter: Robert Mitchell
>            Assignee: Bela Ban
>            Priority: Major
>
> When a node leaves a cluster and then later attempts to re-enter, a race condition can occur where the clustering fails to occur.  Here is the sequence of events that seems to allow this to occur:
> # The rejoining node must have a "higher" IP address than the current cluster coordinator.
> # On the rejoin attempt, the coordinator sends a message to the rejoining node before the rejoining node sends to the coordinator using its prior address.  I have seen this happen for two reasons:
> ## UNICAST3 is resending messages (which often happens with the final LEAVE_RSP from the prior cluster membership because it apparently does not get acked before the connection closes)
> ## TCPPING is sending a ping request to the cached prior address.
> # The connection gets established.  It will then be used by the rejoining node whenever communicating with the cluster coordinator.
> #  However, the cluster coordinator has this as the connection for the prior address.  So the following happens whenever it wants to send a message to the rejoining node:
> ## It will attempt to create a new connection.
> ## The rejoining node will reject the connection as a redundant connection with its current connection taking precedence since it is coming from the same logical address as the "bad" connection.
> Since the messages needed to find and join the cluster or merge the two clusters are all unicast messages, the rejoining node will never get them and not be able to join until something happens that causes the initial connection to get closed.



--
This message was sent by Atlassian Jira
(v7.13.8#713008)