[JBoss JIRA] (JGRP-1910) MERGE3: Do not lose any members from view during a series of merges
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1910?page=com.atlassian.jira.plugin.... ]
Bela Ban edited comment on JGRP-1910 at 3/11/15 9:49 AM:
---------------------------------------------------------
OK, so the problem is this: if we have a split like the one below:
{noformat}
0: [0|9] (3) [0, 2, 1]
1: [0|9] (3) [0, 2, 1]
2: [0|9] (3) [0, 2, 1]
3: [3|9] (2) [3, 4]
4: [3|9] (2) [3, 4]
5: [5|10] (3) [5, 6, 7]
6: [5|10] (3) [5, 6, 7]
7: [5|10] (3) [5, 6, 7]
8: [9|9] (2) [9, 8]
9: [9|9] (2) [9, 8]
{noformat}
, members send {{INFO}} messages (in {{MERGE3}}) with the view-id, e.g. 7 sends {{\[5|10\]}}.
Every member multicasts an {{INFO}} message at a random interval \[{{MERGE3.min_interval..MERGE3_max_interval}}\].
The coordinators of the subclusters (0, 3, 5 and 9) check every {{MERGE3.check_interval}} ms if they've received different view-ids and initiate a merge if so.
The problem with your test is that it drops {{INFO}} messages (not retransmitted as {{MERGE3}} is below {{UNICASTX}} and {{NAKACKX}}, so when a coordinator starts the view-id check, it may not have received all view-ids.
Example:
{noformat}
5: I will be the merge leader. Starting the merge task. Coords: [5], views: {5=[5|10] (3) [5, 6, 7], 1=[0|9] (3) [0, 2, 1]}
9: I will be the merge leader. Starting the merge task. Coords: [9, 3], views: {4=[3|9] (2) [3, 4], 9=[9|9] (2) [9, 8], 3=[3|9] (2) [3, 4], 7=[5|11] (6) [5, 0, 2, 1, 6, 7]}
{noformat}
Here, we can see that 5 and 9 were merge leaders and started merges at almost the same time. Depending on which merge leader is able to contact more members, there will be more than 1 merge needed to end up with a fully merged view.
E.g if 5 succeeds in contacting itself, 3 and 0 *before 9 contacts them* , then a merge will ensue including 0, 3 and 5. 9 will not merge with anyone else, as it started a merge on its own and therefore rejects 5's merge request.
The problem can be mitigated (but not eliminated altogether) by reducing {{min_interval}} and {{max_interval}} and increasing {{check_interval}} in {{MERGE3}}: this way, coordinators are more likely to get more {{INFO}} messages from everyone (despite the message drops) and this reduces the chances of multiple merge leaders being chosen.
was (Author: belaban):
OK, so the problem is this: if we have a split like the one below:
{noformat}
0: [0|9] (3) [0, 2, 1]
1: [0|9] (3) [0, 2, 1]
2: [0|9] (3) [0, 2, 1]
3: [3|9] (2) [3, 4]
4: [3|9] (2) [3, 4]
5: [5|10] (3) [5, 6, 7]
6: [5|10] (3) [5, 6, 7]
7: [5|10] (3) [5, 6, 7]
8: [9|9] (2) [9, 8]
9: [9|9] (2) [9, 8]
{noformat}
, members send {{INFO}} messages (in {{MERGE3}}) with the view-id, e.g. 7 sends {{\[5|10\]}}.
Every member multicasts an {{INFO}} message at a random interval [{{MERGE3.min_interval..MERGE3_max_interval}}].
The coordinators of the subclusters (0, 3, 5 and 9) check every {{MERGE3.check_interval}} ms if they've received different view-ids and initiate a merge if so.
The problem with your test is that it drops {{INFO}} messages (not retransmitted as {{MERGE3}} is below {{UNICASTX}} and {{NAKACKX}}, so when a coordinator starts the view-id check, it may not have received all view-ids.
Example:
{noformat}
5: I will be the merge leader. Starting the merge task. Coords: [5], views: {5=[5|10] (3) [5, 6, 7], 1=[0|9] (3) [0, 2, 1]}
9: I will be the merge leader. Starting the merge task. Coords: [9, 3], views: {4=[3|9] (2) [3, 4], 9=[9|9] (2) [9, 8], 3=[3|9] (2) [3, 4], 7=[5|11] (6) [5, 0, 2, 1, 6, 7]}
{noformat}
Here, we can see that 5 and 9 were merge leaders and started merges at almost the same time. Depending on which merge leader is able to contact more members, there will be more than 1 merge needed to end up with a fully merged view.
E.g if 5 succeeds in contacting itself, 3 and 0 *before 9 contacts them* , then a merge will ensue including 0, 3 and 5. 9 will not merge with anyone else, as it started a merge on its own and therefore rejects 5's merge request.
The problem can be mitigated (but not eliminated altogether) by reducing {{min_interval}} and {{max_interval}} and increasing {{check_interval}} in {{MERGE3}}: this way, coordinators are more likely to get more {{INFO}} messages from everyone (despite the message drops) and this reduces the chances of multiple merge leaders being chosen.
> MERGE3: Do not lose any members from view during a series of merges
> -------------------------------------------------------------------
>
> Key: JGRP-1910
> URL: https://issues.jboss.org/browse/JGRP-1910
> Project: JGroups
> Issue Type: Bug
> Reporter: Radim Vansa
> Assignee: Bela Ban
> Fix For: 3.6.3
>
> Attachments: SplitMergeFailFastTest.java, SplitMergeTest.java
>
>
> When connection between nodes is re-established, MERGE3 should merge the cluster together. This often does not involve a single MergeView but a series of such events. The problematic property of this protocol is that some of those views can lack certain members, though these are reachable.
> This causes problem in Infinispan since the cache cannot be fully rebalanced before another merge arrives, and all owners of certain segment can be gradually removed (and added again) to the view, while this is not detected as partition but crashed nodes -> losing all owners means data loss.
> Removing members from view should be the role of FDx protocols, not MERGEx.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years, 4 months
[JBoss JIRA] (JGRP-1910) MERGE3: Do not lose any members from view during a series of merges
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1910?page=com.atlassian.jira.plugin.... ]
Bela Ban edited comment on JGRP-1910 at 3/11/15 9:49 AM:
---------------------------------------------------------
OK, so the problem is this: if we have a split like the one below:
{noformat}
0: [0|9] (3) [0, 2, 1]
1: [0|9] (3) [0, 2, 1]
2: [0|9] (3) [0, 2, 1]
3: [3|9] (2) [3, 4]
4: [3|9] (2) [3, 4]
5: [5|10] (3) [5, 6, 7]
6: [5|10] (3) [5, 6, 7]
7: [5|10] (3) [5, 6, 7]
8: [9|9] (2) [9, 8]
9: [9|9] (2) [9, 8]
{noformat}
, members send {{INFO}} messages (in {{MERGE3}}) with the view-id, e.g. 7 sends {{\[5|10\]}}.
Every member multicasts an {{INFO}} message at a random interval [{{MERGE3.min_interval..MERGE3_max_interval}}].
The coordinators of the subclusters (0, 3, 5 and 9) check every {{MERGE3.check_interval}} ms if they've received different view-ids and initiate a merge if so.
The problem with your test is that it drops {{INFO}} messages (not retransmitted as {{MERGE3}} is below {{UNICASTX}} and {{NAKACKX}}, so when a coordinator starts the view-id check, it may not have received all view-ids.
Example:
{noformat}
5: I will be the merge leader. Starting the merge task. Coords: [5], views: {5=[5|10] (3) [5, 6, 7], 1=[0|9] (3) [0, 2, 1]}
9: I will be the merge leader. Starting the merge task. Coords: [9, 3], views: {4=[3|9] (2) [3, 4], 9=[9|9] (2) [9, 8], 3=[3|9] (2) [3, 4], 7=[5|11] (6) [5, 0, 2, 1, 6, 7]}
{noformat}
Here, we can see that 5 and 9 were merge leaders and started merges at almost the same time. Depending on which merge leader is able to contact more members, there will be more than 1 merge needed to end up with a fully merged view.
E.g if 5 succeeds in contacting itself, 3 and 0 *before 9 contacts them* , then a merge will ensue including 0, 3 and 5. 9 will not merge with anyone else, as it started a merge on its own and therefore rejects 5's merge request.
The problem can be mitigated (but not eliminated altogether) by reducing {{min_interval}} and {{max_interval}} and increasing {{check_interval}} in {{MERGE3}}: this way, coordinators are more likely to get more {{INFO}} messages from everyone (despite the message drops) and this reduces the chances of multiple merge leaders being chosen.
was (Author: belaban):
OK, so the problem is this: if we have a split like the one below:
{noformat}
0: [0|9] (3) [0, 2, 1]
1: [0|9] (3) [0, 2, 1]
2: [0|9] (3) [0, 2, 1]
3: [3|9] (2) [3, 4]
4: [3|9] (2) [3, 4]
5: [5|10] (3) [5, 6, 7]
6: [5|10] (3) [5, 6, 7]
7: [5|10] (3) [5, 6, 7]
8: [9|9] (2) [9, 8]
9: [9|9] (2) [9, 8]
{noformat}
, members send {{INFO}} messages (in {{MERGE3}}) with the view-id, e.g. 7 sends {{\[5|10\]}}.
Every member multicasts an {{INFO}} message at a random interval {{\[MERGE3.min_interval..MERGE3_max_interval\]}}.
The coordinators of the subclusters (0, 3, 5 and 9) check every {MERGE3.check_interval}} ms if they've received different view-ids and initiate a merge if so.
The problem with your test is that it drops {{INFO}} messages (not retransmitted as {{MERGE3}} is below {{UNICASTX}} and {{NAKACKX}}, so when a coordinator starts the view-id check, it may not have received all view-ids.
Example:
{noformat}
5: I will be the merge leader. Starting the merge task. Coords: [5], views: {5=[5|10] (3) [5, 6, 7], 1=[0|9] (3) [0, 2, 1]}
9: I will be the merge leader. Starting the merge task. Coords: [9, 3], views: {4=[3|9] (2) [3, 4], 9=[9|9] (2) [9, 8], 3=[3|9] (2) [3, 4], 7=[5|11] (6) [5, 0, 2, 1, 6, 7]}
{noformat}
Here, we can see that 5 and 9 were merge leaders and started merges at almost the same time. Depending on which merge leader is able to contact more members, there will be more than 1 merge needed to end up with a fully merged view.
E.g if 5 succeeds in contacting itself, 3 and 0 *before 9 contacts them* , then a merge will ensue including 0, 3 and 5. 9 will not merge with anyone else, as it started a merge on its own and therefore rejects 5's merge request.
The problem can be mitigated (but not eliminated altogether) by reducing {{min_interval}} and {{max_interval}} and increasing {{check_interval}} in {{MERGE3}}: this way, coordinators are more likely to get more {{INFO}} messages from everyone (despite the message drops) and this reduces the chances of multiple merge leaders being chosen.
> MERGE3: Do not lose any members from view during a series of merges
> -------------------------------------------------------------------
>
> Key: JGRP-1910
> URL: https://issues.jboss.org/browse/JGRP-1910
> Project: JGroups
> Issue Type: Bug
> Reporter: Radim Vansa
> Assignee: Bela Ban
> Fix For: 3.6.3
>
> Attachments: SplitMergeFailFastTest.java, SplitMergeTest.java
>
>
> When connection between nodes is re-established, MERGE3 should merge the cluster together. This often does not involve a single MergeView but a series of such events. The problematic property of this protocol is that some of those views can lack certain members, though these are reachable.
> This causes problem in Infinispan since the cache cannot be fully rebalanced before another merge arrives, and all owners of certain segment can be gradually removed (and added again) to the view, while this is not detected as partition but crashed nodes -> losing all owners means data loss.
> Removing members from view should be the role of FDx protocols, not MERGEx.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years, 4 months
[JBoss JIRA] (JGRP-1910) MERGE3: Do not lose any members from view during a series of merges
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1910?page=com.atlassian.jira.plugin.... ]
Bela Ban edited comment on JGRP-1910 at 3/11/15 9:34 AM:
---------------------------------------------------------
No, I haven't yet had a chance to complete the PlusCal manual.
What do you mean ? If you're C having view {{\[A|5] \[A,B,C\]}} and then get view {{\[B|6\] \[B,C\]}}, on what grounds would you make C reject it ?
was (Author: belaban):
No, I haven't yet had a chance to complete the PlusCal manual.
What do you mean ? If you're C have view {{\[A|5] \[A,B,C\]}} and then get view {{\[B|6\] \[B,C\]}}, on what grounds would you make C reject it ?
> MERGE3: Do not lose any members from view during a series of merges
> -------------------------------------------------------------------
>
> Key: JGRP-1910
> URL: https://issues.jboss.org/browse/JGRP-1910
> Project: JGroups
> Issue Type: Bug
> Reporter: Radim Vansa
> Assignee: Bela Ban
> Fix For: 3.6.3
>
> Attachments: SplitMergeFailFastTest.java, SplitMergeTest.java
>
>
> When connection between nodes is re-established, MERGE3 should merge the cluster together. This often does not involve a single MergeView but a series of such events. The problematic property of this protocol is that some of those views can lack certain members, though these are reachable.
> This causes problem in Infinispan since the cache cannot be fully rebalanced before another merge arrives, and all owners of certain segment can be gradually removed (and added again) to the view, while this is not detected as partition but crashed nodes -> losing all owners means data loss.
> Removing members from view should be the role of FDx protocols, not MERGEx.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years, 4 months
[JBoss JIRA] (JGRP-1910) MERGE3: Do not lose any members from view during a series of merges
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1910?page=com.atlassian.jira.plugin.... ]
Bela Ban edited comment on JGRP-1910 at 3/11/15 9:34 AM:
---------------------------------------------------------
No, I haven't yet had a chance to complete the PlusCal manual.
What do you mean ? If you're C have view {{\[A|5] \[A,B,C\]}} and then get view {{\[B|6\] \[B,C\]}}, on what grounds would you make C reject it ?
was (Author: belaban):
No, I haven't yet had a chance to complete the PlusCal manual.
What do you mean ? If you're C have view {{\[A|5] \[A,B,C\}}} and the get view {{\[B|6\] \[B,C\}}}, on what grounds would you make C reject it ?
> MERGE3: Do not lose any members from view during a series of merges
> -------------------------------------------------------------------
>
> Key: JGRP-1910
> URL: https://issues.jboss.org/browse/JGRP-1910
> Project: JGroups
> Issue Type: Bug
> Reporter: Radim Vansa
> Assignee: Bela Ban
> Fix For: 3.6.3
>
> Attachments: SplitMergeFailFastTest.java, SplitMergeTest.java
>
>
> When connection between nodes is re-established, MERGE3 should merge the cluster together. This often does not involve a single MergeView but a series of such events. The problematic property of this protocol is that some of those views can lack certain members, though these are reachable.
> This causes problem in Infinispan since the cache cannot be fully rebalanced before another merge arrives, and all owners of certain segment can be gradually removed (and added again) to the view, while this is not detected as partition but crashed nodes -> losing all owners means data loss.
> Removing members from view should be the role of FDx protocols, not MERGEx.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years, 4 months
[JBoss JIRA] (JGRP-1910) MERGE3: Do not lose any members from view during a series of merges
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1910?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on JGRP-1910:
--------------------------------
No, I haven't yet had a chance to complete the PlusCal manual.
What do you mean ? If you're C have view {{\[A|5] \[A,B,C\}}} and the get view {{\[B|6\] \[B,C\}}}, on what grounds would you make C reject it ?
> MERGE3: Do not lose any members from view during a series of merges
> -------------------------------------------------------------------
>
> Key: JGRP-1910
> URL: https://issues.jboss.org/browse/JGRP-1910
> Project: JGroups
> Issue Type: Bug
> Reporter: Radim Vansa
> Assignee: Bela Ban
> Fix For: 3.6.3
>
> Attachments: SplitMergeFailFastTest.java, SplitMergeTest.java
>
>
> When connection between nodes is re-established, MERGE3 should merge the cluster together. This often does not involve a single MergeView but a series of such events. The problematic property of this protocol is that some of those views can lack certain members, though these are reachable.
> This causes problem in Infinispan since the cache cannot be fully rebalanced before another merge arrives, and all owners of certain segment can be gradually removed (and added again) to the view, while this is not detected as partition but crashed nodes -> losing all owners means data loss.
> Removing members from view should be the role of FDx protocols, not MERGEx.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years, 4 months
[JBoss JIRA] (WFLY-4416) Cannot obtain DOMImplementationRegistry instance
by Thomas Diesler (JIRA)
[ https://issues.jboss.org/browse/WFLY-4416?page=com.atlassian.jira.plugin.... ]
Thomas Diesler commented on WFLY-4416:
--------------------------------------
Yes, same on 9.x
https://github.com/wildfly/wildfly/pull/7249
> Cannot obtain DOMImplementationRegistry instance
> ------------------------------------------------
>
> Key: WFLY-4416
> URL: https://issues.jboss.org/browse/WFLY-4416
> Project: WildFly
> Issue Type: Bug
> Components: XML Frameworks
> Affects Versions: 8.2.0.Final
> Reporter: Thomas Diesler
> Assignee: Jason Greene
>
> {code}
> testDOMImplementationRegistry(org.jboss.as.test.smoke.xml.DOMImplementationRegistryTestCase) Time elapsed: 0.09 sec <<< ERROR!
> java.lang.ClassNotFoundException: com.sun.org.apache.xerces.internal.dom.DOMXSImplementationSourceImpl from [Module "deployment.dom-registry-test:main" from Service Module Loader]
> at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:213)
> at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:459)
> at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:408)
> at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:389)
> at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:134)
> at org.w3c.dom.bootstrap.DOMImplementationRegistry.newInstance(DOMImplementationRegistry.java:182)
> at org.jboss.as.test.smoke.xml.DOMImplementationRegistryTestCase.testDOMImplementationRegistry(DOMImplementationRegistryTestCase.java:52)
> {code}
> CrossRef: https://github.com/wildfly-extras/wildfly-camel/issues/391
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years, 4 months
[JBoss JIRA] (JGRP-1910) MERGE3: Do not lose any members from view during a series of merges
by Radim Vansa (JIRA)
[ https://issues.jboss.org/browse/JGRP-1910?page=com.atlassian.jira.plugin.... ]
Radim Vansa commented on JGRP-1910:
-----------------------------------
[~belaban] Any chance you have managed to model MERGE3 in +Cal?
So the problem is probably that a node accepts a merge view from merge leader X after accepting another view from merge leader Y. We can't eliminate two leaders being chosen concurrently, but a view that's not based on current view should be rejected.
> MERGE3: Do not lose any members from view during a series of merges
> -------------------------------------------------------------------
>
> Key: JGRP-1910
> URL: https://issues.jboss.org/browse/JGRP-1910
> Project: JGroups
> Issue Type: Bug
> Reporter: Radim Vansa
> Assignee: Bela Ban
> Fix For: 3.6.3
>
> Attachments: SplitMergeFailFastTest.java, SplitMergeTest.java
>
>
> When connection between nodes is re-established, MERGE3 should merge the cluster together. This often does not involve a single MergeView but a series of such events. The problematic property of this protocol is that some of those views can lack certain members, though these are reachable.
> This causes problem in Infinispan since the cache cannot be fully rebalanced before another merge arrives, and all owners of certain segment can be gradually removed (and added again) to the view, while this is not detected as partition but crashed nodes -> losing all owners means data loss.
> Removing members from view should be the role of FDx protocols, not MERGEx.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years, 4 months
[JBoss JIRA] (JGRP-1910) MERGE3: Do not lose any members from view during a series of merges
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1910?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on JGRP-1910:
--------------------------------
To reduces the chances of multiple parallel merges, we could say that the _merge leader_ are collected from all views, e.g.
{noformat}
0 -> [0|5] (4) [0,1,2,3]
5 -> [5|5] (2) [5,6]
9 -> [8|5] (2) [8,9]
{noformat}
Here, the current algorithm to pick merge leaders only collects view creators that actually *sent* the {{INFO}} message: 0 and 5, but *not* 8. The reason is that we want to pick responses from live members, from which we received {{INFO}} messages, and not from members that were proposed by other members and might be incorrect.
For example, 8 might have crashed, but 9 still has it in its view. We don't want to wait until 8 has been exluded by failure detection.
If 8 was indeed dead and we collected 0, 5 and 8 as merge leaders and 8 became merge leader (by sorting 0, 5 and 8 (UUID-wise) and taking the first one), then we'd waste merge rounds until 8 was actualy declared dead and removed from the views.
> MERGE3: Do not lose any members from view during a series of merges
> -------------------------------------------------------------------
>
> Key: JGRP-1910
> URL: https://issues.jboss.org/browse/JGRP-1910
> Project: JGroups
> Issue Type: Bug
> Reporter: Radim Vansa
> Assignee: Bela Ban
> Fix For: 3.6.3
>
> Attachments: SplitMergeFailFastTest.java, SplitMergeTest.java
>
>
> When connection between nodes is re-established, MERGE3 should merge the cluster together. This often does not involve a single MergeView but a series of such events. The problematic property of this protocol is that some of those views can lack certain members, though these are reachable.
> This causes problem in Infinispan since the cache cannot be fully rebalanced before another merge arrives, and all owners of certain segment can be gradually removed (and added again) to the view, while this is not detected as partition but crashed nodes -> losing all owners means data loss.
> Removing members from view should be the role of FDx protocols, not MERGEx.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years, 4 months
[JBoss JIRA] (WFLY-4194) Deprecate default-stack JGroups subsystem attribute
by Paul Ferraro (JIRA)
[ https://issues.jboss.org/browse/WFLY-4194?page=com.atlassian.jira.plugin.... ]
Paul Ferraro closed WFLY-4194.
------------------------------
Fix Version/s: 9.0.0.Beta1
Resolution: Done
Fixed already.
> Deprecate default-stack JGroups subsystem attribute
> ---------------------------------------------------
>
> Key: WFLY-4194
> URL: https://issues.jboss.org/browse/WFLY-4194
> Project: WildFly
> Issue Type: Enhancement
> Components: Clustering
> Affects Versions: 9.0.0.Alpha1
> Reporter: Paul Ferraro
> Assignee: Paul Ferraro
> Fix For: 9.0.0.Beta1
>
>
> The original purpose of the default-stack attribute was to provide a default ChannelFactory implementation. However, with the introduction of fork channels, there are now 2 candidate for a default ChannelFactory:
> 1. The JChannelFactory of the default stack.
> 2. The ForkChannelFactory of the default channel.
> #2 is the better option - and is what Infinispan transports use by default.
> This would require that the stack attribute of a channel is required (currently, it is optional and defaults to the default-stack).
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years, 4 months
[JBoss JIRA] (JGRP-1910) MERGE3: Do not lose any members from view during a series of merges
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1910?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on JGRP-1910:
--------------------------------
OK, so the problem is this: if we have a split like the one below:
{noformat}
0: [0|9] (3) [0, 2, 1]
1: [0|9] (3) [0, 2, 1]
2: [0|9] (3) [0, 2, 1]
3: [3|9] (2) [3, 4]
4: [3|9] (2) [3, 4]
5: [5|10] (3) [5, 6, 7]
6: [5|10] (3) [5, 6, 7]
7: [5|10] (3) [5, 6, 7]
8: [9|9] (2) [9, 8]
9: [9|9] (2) [9, 8]
{noformat}
, members send {{INFO}} messages (in {{MERGE3}}) with the view-id, e.g. 7 sends {{\[5|10\]}}.
Every member multicasts an {{INFO}} message at a random interval {{\[MERGE3.min_interval..MERGE3_max_interval\]}}.
The coordinators of the subclusters (0, 3, 5 and 9) check every {MERGE3.check_interval}} ms if they've received different view-ids and initiate a merge if so.
The problem with your test is that it drops {{INFO}} messages (not retransmitted as {{MERGE3}} is below {{UNICASTX}} and {{NAKACKX}}, so when a coordinator starts the view-id check, it may not have received all view-ids.
Example:
{noformat}
5: I will be the merge leader. Starting the merge task. Coords: [5], views: {5=[5|10] (3) [5, 6, 7], 1=[0|9] (3) [0, 2, 1]}
9: I will be the merge leader. Starting the merge task. Coords: [9, 3], views: {4=[3|9] (2) [3, 4], 9=[9|9] (2) [9, 8], 3=[3|9] (2) [3, 4], 7=[5|11] (6) [5, 0, 2, 1, 6, 7]}
{noformat}
Here, we can see that 5 and 9 were merge leaders and started merges at almost the same time. Depending on which merge leader is able to contact more members, there will be more than 1 merge needed to end up with a fully merged view.
E.g if 5 succeeds in contacting itself, 3 and 0 *before 9 contacts them* , then a merge will ensue including 0, 3 and 5. 9 will not merge with anyone else, as it started a merge on its own and therefore rejects 5's merge request.
The problem can be mitigated (but not eliminated altogether) by reducing {{min_interval}} and {{max_interval}} and increasing {{check_interval}} in {{MERGE3}}: this way, coordinators are more likely to get more {{INFO}} messages from everyone (despite the message drops) and this reduces the chances of multiple merge leaders being chosen.
> MERGE3: Do not lose any members from view during a series of merges
> -------------------------------------------------------------------
>
> Key: JGRP-1910
> URL: https://issues.jboss.org/browse/JGRP-1910
> Project: JGroups
> Issue Type: Bug
> Reporter: Radim Vansa
> Assignee: Bela Ban
> Fix For: 3.6.3
>
> Attachments: SplitMergeFailFastTest.java, SplitMergeTest.java
>
>
> When connection between nodes is re-established, MERGE3 should merge the cluster together. This often does not involve a single MergeView but a series of such events. The problematic property of this protocol is that some of those views can lack certain members, though these are reachable.
> This causes problem in Infinispan since the cache cannot be fully rebalanced before another merge arrives, and all owners of certain segment can be gradually removed (and added again) to the view, while this is not detected as partition but crashed nodes -> losing all owners means data loss.
> Removing members from view should be the role of FDx protocols, not MERGEx.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years, 4 months