[ https://issues.redhat.com/browse/ISPN-11113?page=com.atlassian.jira.plugi... ] Dan Berindei updated ISPN-11113: -------------------------------- Description: Partition handling tests use {{LOCAL_PING.setClusterName()}} with a unique name to disable discovery, otherwise partitions would try to merge while they are supposed to stay separate. But {{LOCAL_PING}} uses the cluster name on stop to remove the node from the static discovery map. If the test doesn't change the cluster name back, {{LOCAL_PING}} doesn't remove the node, the next test method sees an existing coordinator, and tries to connect to it. When a test has lots of test methods, like {{ScatteredDelayedAvailabilityUpdateTest}}, each test method leaves one more coordinator in the discovery map, and each test method takes longer to start the first method. {noformat} 09:08:52,758 DEBUG (testng:[]) [GMS] address=NodeA-30899, cluster=org.infinispan.partitionhandling.ScatteredDelayedAvailabilityUpdateTest[SCATTERED_SYNC, bias=NEVER, DENY_READ_WRITES], physical address=127.0.0.1:51941 09:08:52,774 TRACE (testng:[]) [GMS] NodeA-30899: discovery took 0 ms, members: 21 rsps (5 coords) [done] 09:08:52,774 DEBUG (testng:[]) [GMS] NodeA-30899: found multiple coords: [NodeA-2608, NodeA-5606, NodeA-17288, NodeA-64297, NodeA-48475] 09:08:52,774 DEBUG (testng:[]) [GMS] NodeA-30899: sending JOIN(NodeA-30899) to NodeA-5606 09:08:54,774 WARN (testng:[]) [GMS] NodeA-30899: JOIN(NodeA-30899) sent to NodeA-5606 timed out (after 2000 ms), on try 0 09:08:54,774 DEBUG (testng:[]) [GMS] NodeA-30899: sending JOIN(NodeA-30899) to NodeA-64297 09:08:56,775 WARN (testng:[]) [GMS] NodeA-30899: JOIN(NodeA-30899) sent to NodeA-64297 timed out (after 2000 ms), on try 0 09:08:56,775 DEBUG (testng:[]) [GMS] NodeA-30899: sending JOIN(NodeA-30899) to NodeA-48475 09:08:58,775 WARN (testng:[]) [GMS] NodeA-30899: JOIN(NodeA-30899) sent to NodeA-48475 timed out (after 2000 ms), on try 0 09:08:58,775 DEBUG (testng:[]) [GMS] NodeA-30899: sending JOIN(NodeA-30899) to NodeA-17288 09:09:00,776 WARN (testng:[]) [GMS] NodeA-30899: JOIN(NodeA-30899) sent to NodeA-17288 timed out (after 2000 ms), on try 0 09:09:00,776 DEBUG (testng:[]) [GMS] NodeA-30899: sending JOIN(NodeA-30899) to NodeA-2608 09:09:02,776 WARN (testng:[]) [GMS] NodeA-30899: JOIN(NodeA-30899) sent to NodeA-2608 timed out (after 2000 ms), on try 0 09:09:02,776 TRACE (testng:[]) [GMS] NodeA-30899: discovery took 0 ms, members: 21 rsps (5 coords) [done] 09:09:02,776 DEBUG (testng:[]) [GMS] NodeA-30899: found multiple coords: [NodeA-2608, NodeA-5606, NodeA-17288, NodeA-64297, NodeA-48475] 09:09:02,776 DEBUG (testng:[]) [GMS] NodeA-30899: sending JOIN(NodeA-30899) to NodeA-5606 09:09:04,776 WARN (testng:[]) [GMS] NodeA-30899: JOIN(NodeA-30899) sent to NodeA-5606 timed out (after 2000 ms), on try 1 ... 09:09:12,777 TRACE (testng:[]) [GMS] NodeA-30899: discovery took 0 ms, members: 21 rsps (5 coords) [done] 09:09:12,778 DEBUG (testng:[]) [GMS] NodeA-30899: found multiple coords: [NodeA-2608, NodeA-5606, NodeA-17288, NodeA-64297, NodeA-48475] 09:09:12,778 DEBUG (testng:[]) [GMS] NodeA-30899: sending JOIN(NodeA-30899) to NodeA-2608 09:09:14,778 WARN (testng:[]) [GMS] NodeA-30899: JOIN(NodeA-30899) sent to NodeA-2608 timed out (after 2000 ms), on try 2 ... 09:09:22,780 WARN (testng:[]) [GMS] NodeA-30899: too many JOIN attempts (3): becoming singleton {noformat} was: Partition handling tests use {{LOCAL_PING.setClusterName()}} with a unique name to disable discovery, otherwise partitions would try to merge while they are supposed to stay separate. But {{LOCAL_PING}} uses the cluster name on stop to remove the node from the static discovery map. If the test doesn't change the cluster name back, {{LOCAL_PING}} doesn't remove the node, the next test method sees an existing coordinator, and tries to connect to it. When a test has lots of test methods, like {{ScatteredDelayedAvailabilityUpdateTest}}, each test method leaves one more coordinator in the discovery map, and each test method takes longer to start the first method. {noformat} 09:08:52,758 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] address=ScatteredDelayedAvailabilityUpdateTest-NodeA-30899, cluster=org.infinispan.partitionhandling.ScatteredDelayedAvailabilityUpdateTest[SCATTERED_SYNC, bias=NEVER, DENY_READ_WRITES], physical address=127.0.0.1:51941 09:08:52,774 TRACE (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: discovery took 0 ms, members: 21 rsps (5 coords) [done] 09:08:52,774 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: found multiple coords: [ScatteredDelayedAvailabilityUpdateTest-NodeA-2608, ScatteredDelayedAvailabilityUpdateTest-NodeA-5606, ScatteredDelayedAvailabilityUpdateTest-NodeA-17288, ScatteredDelayedAvailabilityUpdateTest-NodeA-64297, ScatteredDelayedAvailabilityUpdateTest-NodeA-48475] 09:08:52,774 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-5606 09:08:54,774 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-5606 timed out (after 2000 ms), on try 0 09:08:54,774 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-64297 09:08:56,775 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-64297 timed out (after 2000 ms), on try 0 09:08:56,775 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-48475 09:08:58,775 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-48475 timed out (after 2000 ms), on try 0 09:08:58,775 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-17288 09:09:00,776 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-17288 timed out (after 2000 ms), on try 0 09:09:00,776 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-2608 09:09:02,776 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-2608 timed out (after 2000 ms), on try 0 09:09:02,776 TRACE (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: discovery took 0 ms, members: 21 rsps (5 coords) [done] 09:09:02,776 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: found multiple coords: [ScatteredDelayedAvailabilityUpdateTest-NodeA-2608, ScatteredDelayedAvailabilityUpdateTest-NodeA-5606, ScatteredDelayedAvailabilityUpdateTest-NodeA-17288, ScatteredDelayedAvailabilityUpdateTest-NodeA-64297, ScatteredDelayedAvailabilityUpdateTest-NodeA-48475] 09:09:02,776 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-5606 09:09:04,776 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-5606 timed out (after 2000 ms), on try 1 09:09:04,776 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-17288 09:09:06,777 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-17288 timed out (after 2000 ms), on try 1 09:09:06,777 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-2608 09:09:08,777 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-2608 timed out (after 2000 ms), on try 1 09:09:08,777 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-64297 09:09:10,777 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-64297 timed out (after 2000 ms), on try 1 09:09:10,777 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-48475 09:09:12,777 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-48475 timed out (after 2000 ms), on try 1 09:09:12,777 TRACE (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: discovery took 0 ms, members: 21 rsps (5 coords) [done] 09:09:12,778 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: found multiple coords: [ScatteredDelayedAvailabilityUpdateTest-NodeA-2608, ScatteredDelayedAvailabilityUpdateTest-NodeA-5606, ScatteredDelayedAvailabilityUpdateTest-NodeA-17288, ScatteredDelayedAvailabilityUpdateTest-NodeA-64297, ScatteredDelayedAvailabilityUpdateTest-NodeA-48475] 09:09:12,778 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-2608 09:09:14,778 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-2608 timed out (after 2000 ms), on try 2 09:09:14,778 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-17288 09:09:16,778 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-17288 timed out (after 2000 ms), on try 2 09:09:16,778 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-48475 09:09:18,779 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-48475 timed out (after 2000 ms), on try 2 09:09:18,779 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-64297 09:09:20,779 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-64297 timed out (after 2000 ms), on try 2 09:09:20,779 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-5606 09:09:22,779 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-5606 timed out (after 2000 ms), on try 2 09:09:22,780 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: too many JOIN attempts (3): becoming singleton {noformat} -- This message was sent by Atlassian Jira (v7.13.8#713008)