[JBoss JIRA] (ISPN-11113) ScatteredDelayedAvailabilityUpdateTest takes too long
by Dan Berindei (Jira)
Dan Berindei created ISPN-11113:
-----------------------------------
Summary: ScatteredDelayedAvailabilityUpdateTest takes too long
Key: ISPN-11113
URL: https://issues.redhat.com/browse/ISPN-11113
Project: Infinispan
Issue Type: Bug
Components: Core, Test Suite
Affects Versions: 10.1.0.Final
Reporter: Dan Berindei
Assignee: Dan Berindei
Fix For: 11.0.0.Final, 10.1.1.Final
Partition handling tests use {{LOCAL_PING.setClusterName()}} with a unique name to disable discovery, otherwise partitions would try to merge while they are supposed to stay separate.
But {{LOCAL_PING}} uses the cluster name on stop to remove the node from the static discovery map. If the test doesn't change the cluster name back, {{LOCAL_PING}} doesn't remove the node, the next test method sees an existing coordinator, and tries to connect to it. When a test has lots of test methods, like {{ScatteredDelayedAvailabilityUpdateTest}}, each test method leaves one more coordinator in the discovery map, and each test method takes longer to start the first method.
{noformat}
09:08:52,758 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] address=ScatteredDelayedAvailabilityUpdateTest-NodeA-30899, cluster=org.infinispan.partitionhandling.ScatteredDelayedAvailabilityUpdateTest[SCATTERED_SYNC, bias=NEVER, DENY_READ_WRITES], physical address=127.0.0.1:51941
09:08:52,774 TRACE (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: discovery took 0 ms, members: 21 rsps (5 coords) [done]
09:08:52,774 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: found multiple coords: [ScatteredDelayedAvailabilityUpdateTest-NodeA-2608, ScatteredDelayedAvailabilityUpdateTest-NodeA-5606, ScatteredDelayedAvailabilityUpdateTest-NodeA-17288, ScatteredDelayedAvailabilityUpdateTest-NodeA-64297, ScatteredDelayedAvailabilityUpdateTest-NodeA-48475]
09:08:52,774 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-5606
09:08:54,774 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-5606 timed out (after 2000 ms), on try 0
09:08:54,774 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-64297
09:08:56,775 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-64297 timed out (after 2000 ms), on try 0
09:08:56,775 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-48475
09:08:58,775 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-48475 timed out (after 2000 ms), on try 0
09:08:58,775 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-17288
09:09:00,776 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-17288 timed out (after 2000 ms), on try 0
09:09:00,776 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-2608
09:09:02,776 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-2608 timed out (after 2000 ms), on try 0
09:09:02,776 TRACE (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: discovery took 0 ms, members: 21 rsps (5 coords) [done]
09:09:02,776 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: found multiple coords: [ScatteredDelayedAvailabilityUpdateTest-NodeA-2608, ScatteredDelayedAvailabilityUpdateTest-NodeA-5606, ScatteredDelayedAvailabilityUpdateTest-NodeA-17288, ScatteredDelayedAvailabilityUpdateTest-NodeA-64297, ScatteredDelayedAvailabilityUpdateTest-NodeA-48475]
09:09:02,776 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-5606
09:09:04,776 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-5606 timed out (after 2000 ms), on try 1
09:09:04,776 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-17288
09:09:06,777 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-17288 timed out (after 2000 ms), on try 1
09:09:06,777 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-2608
09:09:08,777 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-2608 timed out (after 2000 ms), on try 1
09:09:08,777 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-64297
09:09:10,777 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-64297 timed out (after 2000 ms), on try 1
09:09:10,777 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-48475
09:09:12,777 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-48475 timed out (after 2000 ms), on try 1
09:09:12,777 TRACE (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: discovery took 0 ms, members: 21 rsps (5 coords) [done]
09:09:12,778 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: found multiple coords: [ScatteredDelayedAvailabilityUpdateTest-NodeA-2608, ScatteredDelayedAvailabilityUpdateTest-NodeA-5606, ScatteredDelayedAvailabilityUpdateTest-NodeA-17288, ScatteredDelayedAvailabilityUpdateTest-NodeA-64297, ScatteredDelayedAvailabilityUpdateTest-NodeA-48475]
09:09:12,778 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-2608
09:09:14,778 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-2608 timed out (after 2000 ms), on try 2
09:09:14,778 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-17288
09:09:16,778 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-17288 timed out (after 2000 ms), on try 2
09:09:16,778 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-48475
09:09:18,779 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-48475 timed out (after 2000 ms), on try 2
09:09:18,779 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-64297
09:09:20,779 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-64297 timed out (after 2000 ms), on try 2
09:09:20,779 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-5606
09:09:22,779 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-5606 timed out (after 2000 ms), on try 2
09:09:22,780 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: too many JOIN attempts (3): becoming singleton
{noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 4 months
[JBoss JIRA] (ISPN-11113) ScatteredDelayedAvailabilityUpdateTest takes too long
by Dan Berindei (Jira)
[ https://issues.redhat.com/browse/ISPN-11113?page=com.atlassian.jira.plugi... ]
Dan Berindei updated ISPN-11113:
--------------------------------
Status: Open (was: New)
> ScatteredDelayedAvailabilityUpdateTest takes too long
> -----------------------------------------------------
>
> Key: ISPN-11113
> URL: https://issues.redhat.com/browse/ISPN-11113
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite
> Affects Versions: 10.1.0.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Fix For: 11.0.0.Final, 10.1.1.Final
>
>
> Partition handling tests use {{LOCAL_PING.setClusterName()}} with a unique name to disable discovery, otherwise partitions would try to merge while they are supposed to stay separate.
> But {{LOCAL_PING}} uses the cluster name on stop to remove the node from the static discovery map. If the test doesn't change the cluster name back, {{LOCAL_PING}} doesn't remove the node, the next test method sees an existing coordinator, and tries to connect to it. When a test has lots of test methods, like {{ScatteredDelayedAvailabilityUpdateTest}}, each test method leaves one more coordinator in the discovery map, and each test method takes longer to start the first method.
> {noformat}
> 09:08:52,758 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] address=ScatteredDelayedAvailabilityUpdateTest-NodeA-30899, cluster=org.infinispan.partitionhandling.ScatteredDelayedAvailabilityUpdateTest[SCATTERED_SYNC, bias=NEVER, DENY_READ_WRITES], physical address=127.0.0.1:51941
> 09:08:52,774 TRACE (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: discovery took 0 ms, members: 21 rsps (5 coords) [done]
> 09:08:52,774 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: found multiple coords: [ScatteredDelayedAvailabilityUpdateTest-NodeA-2608, ScatteredDelayedAvailabilityUpdateTest-NodeA-5606, ScatteredDelayedAvailabilityUpdateTest-NodeA-17288, ScatteredDelayedAvailabilityUpdateTest-NodeA-64297, ScatteredDelayedAvailabilityUpdateTest-NodeA-48475]
> 09:08:52,774 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-5606
> 09:08:54,774 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-5606 timed out (after 2000 ms), on try 0
> 09:08:54,774 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-64297
> 09:08:56,775 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-64297 timed out (after 2000 ms), on try 0
> 09:08:56,775 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-48475
> 09:08:58,775 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-48475 timed out (after 2000 ms), on try 0
> 09:08:58,775 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-17288
> 09:09:00,776 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-17288 timed out (after 2000 ms), on try 0
> 09:09:00,776 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-2608
> 09:09:02,776 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-2608 timed out (after 2000 ms), on try 0
> 09:09:02,776 TRACE (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: discovery took 0 ms, members: 21 rsps (5 coords) [done]
> 09:09:02,776 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: found multiple coords: [ScatteredDelayedAvailabilityUpdateTest-NodeA-2608, ScatteredDelayedAvailabilityUpdateTest-NodeA-5606, ScatteredDelayedAvailabilityUpdateTest-NodeA-17288, ScatteredDelayedAvailabilityUpdateTest-NodeA-64297, ScatteredDelayedAvailabilityUpdateTest-NodeA-48475]
> 09:09:02,776 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-5606
> 09:09:04,776 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-5606 timed out (after 2000 ms), on try 1
> 09:09:04,776 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-17288
> 09:09:06,777 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-17288 timed out (after 2000 ms), on try 1
> 09:09:06,777 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-2608
> 09:09:08,777 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-2608 timed out (after 2000 ms), on try 1
> 09:09:08,777 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-64297
> 09:09:10,777 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-64297 timed out (after 2000 ms), on try 1
> 09:09:10,777 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-48475
> 09:09:12,777 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-48475 timed out (after 2000 ms), on try 1
> 09:09:12,777 TRACE (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: discovery took 0 ms, members: 21 rsps (5 coords) [done]
> 09:09:12,778 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: found multiple coords: [ScatteredDelayedAvailabilityUpdateTest-NodeA-2608, ScatteredDelayedAvailabilityUpdateTest-NodeA-5606, ScatteredDelayedAvailabilityUpdateTest-NodeA-17288, ScatteredDelayedAvailabilityUpdateTest-NodeA-64297, ScatteredDelayedAvailabilityUpdateTest-NodeA-48475]
> 09:09:12,778 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-2608
> 09:09:14,778 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-2608 timed out (after 2000 ms), on try 2
> 09:09:14,778 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-17288
> 09:09:16,778 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-17288 timed out (after 2000 ms), on try 2
> 09:09:16,778 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-48475
> 09:09:18,779 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-48475 timed out (after 2000 ms), on try 2
> 09:09:18,779 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-64297
> 09:09:20,779 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-64297 timed out (after 2000 ms), on try 2
> 09:09:20,779 DEBUG (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: sending JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) to ScatteredDelayedAvailabilityUpdateTest-NodeA-5606
> 09:09:22,779 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: JOIN(ScatteredDelayedAvailabilityUpdateTest-NodeA-30899) sent to ScatteredDelayedAvailabilityUpdateTest-NodeA-5606 timed out (after 2000 ms), on try 2
> 09:09:22,780 WARN (testng-ScatteredDelayedAvailabilityUpdateTest:[]) [GMS] ScatteredDelayedAvailabilityUpdateTest-NodeA-30899: too many JOIN attempts (3): becoming singleton
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 4 months
[JBoss JIRA] (ISPN-11112) Make expiration non blocking
by Will Burns (Jira)
Will Burns created ISPN-11112:
---------------------------------
Summary: Make expiration non blocking
Key: ISPN-11112
URL: https://issues.redhat.com/browse/ISPN-11112
Project: Infinispan
Issue Type: Enhancement
Components: Expiration
Reporter: Will Burns
Assignee: Will Burns
Fix For: 11.0.0.Final
Expiration today is inherently blocking due to living in the DataContainer. We need to move this elsewhere or refactor in such a way to let expiration be non blocking. Moving this to be non blocking will most likely highlight many other bugs in expiration code as our tests had no easy to do such things when everything was blocking (aka lots of random failures).
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 4 months
[JBoss JIRA] (ISPN-11000) LocalTopologyManager should not wait for view if the local node is not a member
by Dan Berindei (Jira)
[ https://issues.redhat.com/browse/ISPN-11000?page=com.atlassian.jira.plugi... ]
Dan Berindei resolved ISPN-11000.
---------------------------------
Fix Version/s: 10.1.0.Final
Resolution: Done
Fixed with ISPN-10310
> LocalTopologyManager should not wait for view if the local node is not a member
> -------------------------------------------------------------------------------
>
> Key: ISPN-11000
> URL: https://issues.redhat.com/browse/ISPN-11000
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.4.16.Final, 10.1.0.Beta1
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Fix For: 10.1.0.Final
>
>
> Sometimes a node is excluded from the cluster view but it can still receive multicast messages like FD_ALL heartbeats and topology updates from the coordinator.
> Because it is still receiving heartbeats, the excluded is not becoming coordinator itself and installing a new view. If MERGE3 doesn't merge the partitions, it could keep the outdated view for a long time, and {{LocalTopologyManagerImpl}} will block many transport threads waiting for the right view to process the topology updates that keep coming from the coordinator:
> {noformat}
> 11:31:01,052 INFO [org.radargun.service.InfinispanRestAPI] (pool-2-thread-1) CacheManagerInfo{clusterMembers=[edg-perf01-21541, edg-perf02-54831, edg-perf05-28640, edg-perf03-47882, edg-perf06-47720, edg-perf04-19840, edg-perf07-34498, edg-perf08-52975], clusterSize=8}
> [33m11:31:05,281 WARN [org.jgroups.protocols.pbcast.GMS] (jgroups-79,edg-perf03-47882) edg-perf03-47882: not member of view [edg-perf01-21541|6]; discarding it
> 11:31:11,041 INFO [org.radargun.service.InfinispanRestAPI] (pool-2-thread-1) CacheManagerInfo{clusterMembers=[edg-perf01-21541, edg-perf02-54831, edg-perf05-28640, edg-perf03-47882, edg-perf06-47720, edg-perf04-19840, edg-perf07-34498, edg-perf08-52975], clusterSize=8}
> [0m[33m11:31:16,267 WARN [org.jgroups.protocols.pbcast.GMS] (jgroups-80,edg-perf03-47882) edg-perf03-47882: failed to create view from delta-view; dropping view: java.lang.IllegalStateException: the view-id of the delta view ([edg-perf01-21541|6]) doesn't match the current view-id ([edg-perf01-21541|5]); discarding delta view [edg-perf01-21541|7], ref-view=[edg-perf01-21541|6], left=[edg-perf06-47720]
> [0m[33m11:31:16,274 WARN [org.jgroups.protocols.pbcast.GMS] (jgroups-80,edg-perf03-47882) edg-perf03-47882: not member of view [edg-perf01-21541|7]; discarding it
> 11:31:21,035 INFO [org.radargun.service.InfinispanRestAPI] (pool-2-thread-1) CacheManagerInfo{clusterMembers=[edg-perf01-21541, edg-perf02-54831, edg-perf05-28640, edg-perf03-47882, edg-perf06-47720, edg-perf04-19840, edg-perf07-34498, edg-perf08-52975], clusterSize=8}
> 11:31:31,040 INFO [org.radargun.service.InfinispanRestAPI] (pool-2-thread-1) CacheManagerInfo{clusterMembers=[edg-perf01-21541, edg-perf02-54831, edg-perf05-28640, edg-perf03-47882, edg-perf06-47720, edg-perf04-19840, edg-perf07-34498, edg-perf08-52975], clusterSize=8}
> 11:31:41,047 INFO [org.radargun.service.InfinispanRestAPI] (pool-2-thread-1) CacheManagerInfo{clusterMembers=[edg-perf01-21541, edg-perf02-54831, edg-perf05-28640, edg-perf03-47882, edg-perf06-47720, edg-perf04-19840, edg-perf07-34498, edg-perf08-52975], clusterSize=8}
> 11:31:51,033 INFO [org.radargun.service.InfinispanRestAPI] (pool-2-thread-1) CacheManagerInfo{clusterMembers=[edg-perf01-21541, edg-perf02-54831, edg-perf05-28640, edg-perf03-47882, edg-perf06-47720, edg-perf04-19840, edg-perf07-34498, edg-perf08-52975], clusterSize=8}
> 11:32:01,035 INFO [org.radargun.service.InfinispanRestAPI] (pool-2-thread-1) CacheManagerInfo{clusterMembers=[edg-perf01-21541, edg-perf02-54831, edg-perf05-28640, edg-perf03-47882, edg-perf06-47720, edg-perf04-19840, edg-perf07-34498, edg-perf08-52975], clusterSize=8}
> [0m[33m11:32:03,051 WARN [org.jgroups.protocols.pbcast.GMS] (jgroups-80,edg-perf03-47882) edg-perf03-47882: failed to create view from delta-view; dropping view: java.lang.IllegalStateException: the view-id of the delta view ([edg-perf01-21541|7]) doesn't match the current view-id ([edg-perf01-21541|5]); discarding delta view [edg-perf01-21541|8], ref-view=[edg-perf01-21541|7], left=[edg-perf04-19840]
> [0m[33m11:32:03,063 WARN [org.jgroups.protocols.pbcast.GMS] (jgroups-80,edg-perf03-47882) edg-perf03-47882: not member of view [edg-perf01-21541|8]; discarding it
> [0m[31m11:32:05,321 ERROR [org.infinispan.topology.LocalTopologyManagerImpl] (transport-thread--p5-t5) ISPN000452: Failed to update topology for cache memcachedCache: org.infinispan.util.concurrent.TimeoutException: ISPN000451: Timed out waiting for view 6, current view is 5
> at org.infinispan.topology.LocalTopologyManagerImpl.waitForView(LocalTopologyManagerImpl.java:571)
> at org.infinispan.topology.LocalTopologyManagerImpl.doHandleTopologyUpdate(LocalTopologyManagerImpl.java:302)
> at org.infinispan.topology.LocalTopologyManagerImpl.lambda$handleTopologyUpdate$1(LocalTopologyManagerImpl.java:286)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:37)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:227)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {noformat}
> {{LocalTopologyManagerImpl.doHandleTopologyUpdate()}} could check if the local node is a member of the new topology first, avoid blocking, and avoid logging an error message.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 4 months