[ https://issues.jboss.org/browse/JGRP-1876?page=com.atlassian.jira.plugin.... ] Bela Ban edited comment on JGRP-1876 at 1/13/15 2:07 AM: --------------------------------------------------------- Another (perhaps different) issue brought up by Dan: I might have brought this up before, but I'm seeing some weird (and quite rare) stuff with MERGE3 and multiple partitions merging at the same time. I have a cluster with three nodes: S, T, and U. I split it in 3 partitions with DISCARD, then I disable DISCARD on a single node (U, which will be the merge coordinator when the partitions merge back). Then I create another node V and make it join U's cluster. After this, I disable DISCARD on S and T, and I wait for the partitions to join. Sometimes, however, the merge doesn't go as planned, and U is excluded from the merge: {noformat} 11:13:06,230 DEBUG (Incoming-1,U:) [GMS] U: installing view [U|4] (2) [U, V] 11:13:06,251 DEBUG (testng-ClusterTopologyManagerTest:) [GMS] V: installing view [U|4] (2) [U, V] 11:13:06,264 DEBUG (testng-ClusterTopologyManagerTest:) [ClusterTopologyManagerTest] Merging the cluster partitions 11:13:06,271 TRACE (ViewHandler,U:) [GMS] U: got all ACKs (1) from joiners for view [U|4] 11:13:12,721 TRACE (Timer-2,T:) [MERGE3] discovery protocol returned 3 responses: 3 rsps (2 coords) [done] 11:13:13,059 DEBUG (Timer-3,T:) [MERGE3] I (T) will be the merge leader 11:13:13,059 TRACE (Timer-3,T:) [MERGE3] merge participants are [T, V, S] 11:13:13,101 DEBUG (ViewHandler,T:) [Merger] T: I will be the leader. Starting the merge task for 4 coords 11:13:13,102 DEBUG (MergeTask,T:) [Merger] T: merge task T::4 started with 3 coords 11:13:13,102 TRACE (INT-2,T:) [GMS] T: got merge response from T, merge_id=T::4, merge data is sender=T, view=[T|3] (1) [T], digest=T: [1 (1)] 11:13:13,143 TRACE (INT-1,T:) [GMS] T: got merge response from S, merge_id=T::4, merge data is sender=S, view=[S|4] (1) [S], digest=S: [15 (15)] 11:13:13,143 TRACE (INT-2,T:) [GMS] T: got merge response from V, merge_id=T::4, merge data is sender=V, view=[U|4] (1) [V], digest=V: [0 (0)] 11:13:13,143 DEBUG (MergeTask,T:) [Merger] T: installing merge view [T|5] (3 members) in 3 coords 11:13:13,143 DEBUG (MergeTask,T:) [Merger] T: merge T::4 took 41 ms 11:13:13,143 TRACE (Incoming-1,T:) [GMS] T: mcasting view MergeView::[T|5] (3) [T, S, V], 2 subgroups: [S|4] (1) [S], [T|3] (1) [T] (3 mbrs) {noformat} I'm not sure why, but V suddenly reports being the only one in view U|4, and the number of coords is 2, then 4, and then 3. Can you take a look if maybe there's something wrong with the logging? Ideally I'd like U to always be part of the merge, but I'm not sure if it's missing because Discovery.findMembers is called with async = true or because there's a problem with our TEST_PING protocol implementation. The next best thing would be to exclude V from the merge, since it's not really a coordinator. I haven't been able to reproduce this even when I re-enabled DISCARD on U before disabling it on S and T, so I've given up on obtaining a reliable reproducer for now. I have attached the full log, in case there's something else there that could help. was (Author: belaban): Another (perhaps different) issue brought up by Dan: I might have brought this up before, but I'm seeing some weird (and quite rare) stuff with MERGE3 and multiple partitions merging at the same time. I have a cluster with three nodes: S, T, and U. I split it in 3 partitions with DISCARD, then I disable DISCARD on a single node (U, which will be the merge coordinator when the partitions merge back). Then I create another node V and make it join U's cluster. After this, I disable DISCARD on S and T, and I wait for the partitions to join. Sometimes, however, the merge doesn't go as planned, and U is excluded from the merge: 11:13:06,230 DEBUG (Incoming-1,U:) [GMS] U: installing view [U|4] (2) [U, V] 11:13:06,251 DEBUG (testng-ClusterTopologyManagerTest:) [GMS] V: installing view [U|4] (2) [U, V] 11:13:06,264 DEBUG (testng-ClusterTopologyManagerTest:) [ClusterTopologyManagerTest] Merging the cluster partitions 11:13:06,271 TRACE (ViewHandler,U:) [GMS] U: got all ACKs (1) from joiners for view [U|4] 11:13:12,721 TRACE (Timer-2,T:) [MERGE3] discovery protocol returned 3 responses: 3 rsps (2 coords) [done] 11:13:13,059 DEBUG (Timer-3,T:) [MERGE3] I (T) will be the merge leader 11:13:13,059 TRACE (Timer-3,T:) [MERGE3] merge participants are [T, V, S] 11:13:13,101 DEBUG (ViewHandler,T:) [Merger] T: I will be the leader. Starting the merge task for 4 coords 11:13:13,102 DEBUG (MergeTask,T:) [Merger] T: merge task T::4 started with 3 coords 11:13:13,102 TRACE (INT-2,T:) [GMS] T: got merge response from T, merge_id=T::4, merge data is sender=T, view=[T|3] (1) [T], digest=T: [1 (1)] 11:13:13,143 TRACE (INT-1,T:) [GMS] T: got merge response from S, merge_id=T::4, merge data is sender=S, view=[S|4] (1) [S], digest=S: [15 (15)] 11:13:13,143 TRACE (INT-2,T:) [GMS] T: got merge response from V, merge_id=T::4, merge data is sender=V, view=[U|4] (1) [V], digest=V: [0 (0)] 11:13:13,143 DEBUG (MergeTask,T:) [Merger] T: installing merge view [T|5] (3 members) in 3 coords 11:13:13,143 DEBUG (MergeTask,T:) [Merger] T: merge T::4 took 41 ms 11:13:13,143 TRACE (Incoming-1,T:) [GMS] T: mcasting view MergeView::[T|5] (3) [T, S, V], 2 subgroups: [S|4] (1) [S], [T|3] (1) [T] (3 mbrs) I'm not sure why, but V suddenly reports being the only one in view U|4, and the number of coords is 2, then 4, and then 3. Can you take a look if maybe there's something wrong with the logging? Ideally I'd like U to always be part of the merge, but I'm not sure if it's missing because Discovery.findMembers is called with async = true or because there's a problem with our TEST_PING protocol implementation. The next best thing would be to exclude V from the merge, since it's not really a coordinator. I haven't been able to reproduce this even when I re-enabled DISCARD on U before disabling it on S and T, so I've given up on obtaining a reliable reproducer for now. I have attached the full log, in case there's something else there that could help. -- This message was sent by Atlassian JIRA (v6.3.11#6341)