[jboss-jira] [JBoss JIRA] (JGRP-1805) OverlappingMergeTest testMergeWithDifferentPartitions fails to create correct merged view
Richard Achmatowicz (JIRA)
issues at jboss.org
Mon Mar 24 13:25:13 EDT 2014
[ https://issues.jboss.org/browse/JGRP-1805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12955775#comment-12955775 ]
Richard Achmatowicz edited comment on JGRP-1805 at 3/24/14 1:25 PM:
--------------------------------------------------------------------
An example of this failure of testMergeWithDifferentPartitions with the additional logging: (one failing copy run in the QA lab, one passing copy run on my local machine)
passing test:
{noformat}
------------- testMergeWithDifferentPartitions -----------
-------------------------------------------------------------------
GMS: address=D, cluster=OverlappingMergeTest, physical address=127.0.0.1:48135
-------------------------------------------------------------------
69904 [TRACE] GMS: - D: initial_mbrs are A
69904 [DEBUG] GMS: - election results: {A=1}
69904 [DEBUG] GMS: - sending JOIN(D) to A
69982 [TRACE] GMS: - D: JOIN-RSP=[A|3] [A, B, C, D] [size=4]
69982 [DEBUG] GMS: - D: installing view [A|3] [A, B, C, D]
==== Injecting view [A|4] [A, C, B] into A, B and C ====
69983 [DEBUG] GMS: - A: installing view [A|4] [A, C, B]
69984 [DEBUG] GMS: - B: installing view [A|4] [A, C, B]
69984 [DEBUG] GMS: - C: installing view [A|4] [A, C, B]
==== Injecting view [B|4] [B, A, C, D] into D ====
69985 [DEBUG] GMS: - D: installing view [B|4] [B, A, C, D]
A: [A|4] [A, C, B]
B: [A|4] [A, C, B]
C: [A|4] [A, C, B]
D: [B|4] [B, A, C, D]
==== Injecting a merge event into A, B, C and D====
Enabling TRACE debugging for GMS, MERGE2 and Discovery
70012 [DEBUG] GMS: - D: installing view MergeView::[A|5] [A, D, C, B], subgroups=[A|4] [A, C, B], [A|4] [A, C, B], [B|4] [D]
70013 [TRACE] GMS: - D: mcasting view MergeView::[A|5] [A, D, C, B], subgroups=[A|4] [A, C, B], [A|4] [A, C, B], [B|4] [D] (4 mbrs)
70043 [TRACE] GMS: - D: received all 4 ACKs from members for view [A|5]
Disabling TRACE debugging for GMS, MERGE2 and Discovery
A: [A|5] [A, D, C, B] (coord=true)
B: [A|5] [A, D, C, B] (coord=false)
C: [A|5] [A, D, C, B] (coord=false)
D: [A|5] [A, D, C, B] (coord=false)
{noformat}
failing test:
{noformat}
------------- testMergeWithDifferentPartitions -----------
-------------------------------------------------------------------
GMS: address=D, cluster=OverlappingMergeTest, physical address=10.16.92.187:27202
-------------------------------------------------------------------
182881 [TRACE] GMS: - D: initial_mbrs are C A
182882 [DEBUG] GMS: - election results: {A=2}
182882 [DEBUG] GMS: - sending JOIN(D) to A
182986 [TRACE] GMS: - D: JOIN-RSP=[A|3] [A, B, C, D] [size=4]
182986 [DEBUG] GMS: - D: installing view [A|3] [A, B, C, D]
==== Injecting view [A|4] [A, C, B] into A, B and C ====
182986 [DEBUG] GMS: - A: installing view [A|4] [A, C, B]
182987 [DEBUG] GMS: - B: installing view [A|4] [A, C, B]
182988 [DEBUG] GMS: - C: installing view [A|4] [A, C, B]
==== Injecting view [B|4] [B, A, C, D] into D ====
182989 [DEBUG] GMS: - D: installing view [B|4] [B, A, C, D]
A: [A|4] [A, C, B]
B: [A|4] [A, C, B]
C: [A|4] [A, C, B]
D: [B|4] [B, A, C, D]
==== Injecting a merge event into A, B, C and D====
Enabling TRACE debugging for GMS, MERGE2 and Discovery
Disabling TRACE debugging for GMS, MERGE2 and Discovery
A: [A|5] [A, B, C] (coord=true)
B: [A|5] [A, B, C] (coord=false)
C: [A|5] [A, B, C] (coord=false)
D: [B|4] [B, A, C, D] (coord=false)
{noformat}
It appears that there is no activity after the merge event is injected in the failing case, as there are no TRACE messages appearing.
was (Author: rachmato):
An example of this failure of testMergeWithDifferentPartitions with the additional logging: (one failing copy run in the QA lab, one passing copy run on my local machine)
passing test:
{noformat}
------------- testMergeWithDifferentPartitions -----------
-------------------------------------------------------------------
GMS: address=D, cluster=OverlappingMergeTest, physical address=127.0.0.1:48135
-------------------------------------------------------------------
69904 [TRACE] GMS: - D: initial_mbrs are A
69904 [DEBUG] GMS: - election results: {A=1}
69904 [DEBUG] GMS: - sending JOIN(D) to A
69982 [TRACE] GMS: - D: JOIN-RSP=[A|3] [A, B, C, D] [size=4]
69982 [DEBUG] GMS: - D: installing view [A|3] [A, B, C, D]
==== Injecting view [A|4] [A, C, B] into A, B and C ====
69983 [DEBUG] GMS: - A: installing view [A|4] [A, C, B]
69984 [DEBUG] GMS: - B: installing view [A|4] [A, C, B]
69984 [DEBUG] GMS: - C: installing view [A|4] [A, C, B]
==== Injecting view [B|4] [B, A, C, D] into D ====
69985 [DEBUG] GMS: - D: installing view [B|4] [B, A, C, D]
A: [A|4] [A, C, B]
B: [A|4] [A, C, B]
C: [A|4] [A, C, B]
D: [B|4] [B, A, C, D]
==== Injecting a merge event into A, B, C and D====
Enabling TRACE debugging for GMS, MERGE2 and Discovery
70012 [DEBUG] GMS: - D: installing view MergeView::[A|5] [A, D, C, B], subgroups=[A|4] [A, C, B], [A|4] [A, C, B], [B|4] [D]
70013 [TRACE] GMS: - D: mcasting view MergeView::[A|5] [A, D, C, B], subgroups=[A|4] [A, C, B], [A|4] [A, C, B], [B|4] [D] (4 mbrs)
70043 [TRACE] GMS: - D: received all 4 ACKs from members for view [A|5]
Disabling TRACE debugging for GMS, MERGE2 and Discovery
A: [A|5] [A, D, C, B] (coord=true)
B: [A|5] [A, D, C, B] (coord=false)
C: [A|5] [A, D, C, B] (coord=false)
D: [A|5] [A, D, C, B] (coord=false)
{noformat}
failing test:
{noformat}
------------- testMergeWithDifferentPartitions -----------
-------------------------------------------------------------------
GMS: address=D, cluster=OverlappingMergeTest, physical address=10.16.92.187:27202
-------------------------------------------------------------------
182881 [TRACE] GMS: - D: initial_mbrs are C A
182882 [DEBUG] GMS: - election results: {A=2}
182882 [DEBUG] GMS: - sending JOIN(D) to A
182986 [TRACE] GMS: - D: JOIN-RSP=[A|3] [A, B, C, D] [size=4]
182986 [DEBUG] GMS: - D: installing view [A|3] [A, B, C, D]
==== Injecting view [A|4] [A, C, B] into A, B and C ====
182986 [DEBUG] GMS: - A: installing view [A|4] [A, C, B]
182987 [DEBUG] GMS: - B: installing view [A|4] [A, C, B]
182988 [DEBUG] GMS: - C: installing view [A|4] [A, C, B]
==== Injecting view [B|4] [B, A, C, D] into D ====
182989 [DEBUG] GMS: - D: installing view [B|4] [B, A, C, D]
A: [A|4] [A, C, B]
B: [A|4] [A, C, B]
C: [A|4] [A, C, B]
D: [B|4] [B, A, C, D]
==== Injecting a merge event into A, B, C and D====
Enabling TRACE debugging for GMS, MERGE2 and Discovery
Disabling TRACE debugging for GMS, MERGE2 and Discovery
A: [A|5] [A, B, C] (coord=true)
B: [A|5] [A, B, C] (coord=false)
C: [A|5] [A, B, C] (coord=false)
D: [B|4] [B, A, C, D] (coord=false)
{noformat}
It appears that there is no activity after the merge even is injected in the failing case, as there are no TRACE messages appearing.
> OverlappingMergeTest testMergeWithDifferentPartitions fails to create correct merged view
> -----------------------------------------------------------------------------------------
>
> Key: JGRP-1805
> URL: https://issues.jboss.org/browse/JGRP-1805
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 3.2.13
> Environment: RHEL
> Reporter: Richard Achmatowicz
> Assignee: Bela Ban
> Fix For: 3.2.13
>
>
> This test does the following:
> - creates four channels a,b,c,d
> - injects views A: {A,C,B}, B:{A,C,B}, C:{A,C,B} and D: {B,A,C,D}
> - injects a merge event in each of channels A,B,C,D representing these four views
> - checks that all channels have the final view of size 4
> The test fails intermittently on RHEL, with the same failure each time:
> {noformat}
> 181595 [DEBUG] GMS: - A: installing view [A|2] [A]
> [testng] 181596 [TRACE] GMS: - A: received all 1 ACKs from members for view [A|2]
> [testng] 181866 [TRACE] GMS: - view [A|3] [] is empty: will not multicast it (last view)
> [testng]
> [testng] -------------------------------------------------------------------
> [testng] GMS: address=A, cluster=OverlappingMergeTest, physical address=10.16.94.42:27199
> [testng] -------------------------------------------------------------------
> [testng] 184954 [TRACE] GMS: - A: no initial members discovered: creating group as first member
> [testng] 184954 [DEBUG] GMS: - A: installing view [A|0] [A]
> [testng] 184955 [DEBUG] GMS: - created group (first member). My view is [A|0], impl is org.jgroups.protocols.pbcast.CoordGmsImpl
> [testng]
> [testng] -------------------------------------------------------------------
> [testng] GMS: address=B, cluster=OverlappingMergeTest, physical address=10.16.94.42:27200
> [testng] -------------------------------------------------------------------
> [testng] 184961 [TRACE] GMS: - B: initial_mbrs are A
> [testng] 184961 [DEBUG] GMS: - election results: {A=1}
> [testng] 184961 [DEBUG] GMS: - sending JOIN(B) to A
> [testng] 185013 [TRACE] GMS: - A: new members=[B], suspected=[], leaving=[], new view: [A|1] [A, B]
> [testng] 185014 [TRACE] GMS: - A: mcasting view [A|1] [A, B] (2 mbrs)
> [testng]
> [testng] 185025 [DEBUG] GMS: - A: installing view [A|1] [A, B]
> [testng] 185026 [TRACE] GMS: - A: received all 1 ACKs from members for view [A|1]
> [testng] 185055 [TRACE] GMS: - B: JOIN-RSP=[A|1] [A, B] [size=2]
> [testng]
> [testng]
> [testng] 185055 [DEBUG] GMS: - B: installing view [A|1] [A, B]
> [testng] 185057 [TRACE] GMS: - A: received all ACKs (1) from joiners for view [A|1]
> [testng]
> [testng] -------------------------------------------------------------------
> [testng] GMS: address=C, cluster=OverlappingMergeTest, physical address=10.16.94.42:27201
> [testng] -------------------------------------------------------------------
> [testng] 185064 [TRACE] GMS: - C: initial_mbrs are B A
> [testng] 185064 [DEBUG] GMS: - election results: {A=2}
> [testng] 185064 [DEBUG] GMS: - sending JOIN(C) to A
> [testng] 185108 [TRACE] GMS: - A: new members=[C], suspected=[], leaving=[], new view: [A|2] [A, B, C]
> [testng] 185108 [TRACE] GMS: - A: mcasting view [A|2] [A, B, C] (3 mbrs)
> [testng]
> [testng] 185117 [DEBUG] GMS: - A: installing view [A|2] [A, B, C]
> [testng] 185118 [DEBUG] GMS: - B: installing view [A|2] [A, B, C]
> [testng] 185119 [TRACE] GMS: - A: received all 2 ACKs from members for view [A|2]
> [testng] 185148 [TRACE] GMS: - C: JOIN-RSP=[A|2] [A, B, C] [size=3]
> [testng]
> [testng]
> [testng] 185149 [DEBUG] GMS: - C: installing view [A|2] [A, B, C]
> [testng] 185151 [TRACE] GMS: - A: received all ACKs (1) from joiners for view [A|2]
> [testng]
> [testng] -------------------------------------------------------------------
> [testng] GMS: address=D, cluster=OverlappingMergeTest, physical address=10.16.94.42:27202
> [testng] -------------------------------------------------------------------
> [testng] 185164 [TRACE] GMS: - D: initial_mbrs are B C A
> [testng] 185164 [DEBUG] GMS: - election results: {A=3}
> [testng] 185164 [DEBUG] GMS: - sending JOIN(D) to A
> [testng] 185203 [TRACE] GMS: - A: new members=[D], suspected=[], leaving=[], new view: [A|3] [A, B, C, D]
> [testng] 185203 [TRACE] GMS: - A: mcasting view [A|3] [A, B, C, D] (4 mbrs)
> [testng]
> [testng] 185210 [DEBUG] GMS: - A: installing view [A|3] [A, B, C, D]
> [testng] 185211 [DEBUG] GMS: - B: installing view [A|3] [A, B, C, D]
> [testng] 185211 [DEBUG] GMS: - C: installing view [A|3] [A, B, C, D]
> [testng] 185213 [TRACE] GMS: - A: received all 3 ACKs from members for view [A|3]
> [testng] 185242 [TRACE] GMS: - D: JOIN-RSP=[A|3] [A, B, C, D] [size=4]
> [testng]
> [testng]
> [testng] 185242 [DEBUG] GMS: - D: installing view [A|3] [A, B, C, D]
> [testng] 185242 [TRACE] GMS: - A: received all ACKs (1) from joiners for view [A|3]
> [testng]
> [testng] ==== Injecting view [A|4] [A, C, B] into A, B and C ====
> [testng] 185243 [DEBUG] GMS: - A: installing view [A|4] [A, C, B]
> [testng] 185243 [DEBUG] GMS: - B: installing view [A|4] [A, C, B]
> [testng] 185244 [DEBUG] GMS: - C: installing view [A|4] [A, C, B]
> [testng]
> [testng] ==== Injecting view [B|4] [B, A, C, D] into D ====
> [testng]
> [testng] 185245 [DEBUG] GMS: - D: installing view [B|4] [B, A, C, D]
> [testng] A: [A|4] [A, C, B]
> [testng] B: [A|4] [A, C, B]
> [testng] C: [A|4] [A, C, B]
> [testng] D: [B|4] [B, A, C, D]
> [testng]
> [testng] ==== Injecting a merge event into A, B, C and D====
> [testng] 185251 [TRACE] GMS: - A: got merge response from A, merge_id=A::3, merge data is sender=A, view=[A|4] [A, C, B], digest=C: [0 (0)], B: [0 (0)], A: [4 (4)]
> [testng] 185253 [TRACE] GMS: - B: queue is suspended; request MERGE(4 views) is discarded
> [testng] 185255 [TRACE] GMS: - C: queue is suspended; request MERGE(4 views) is discarded
> [testng] 185255 [TRACE] GMS: - A: got merge response from B, merge_id=A::3, merge data is sender=B, view=[A|4] [A, C, B], digest=C: [0 (0)], B: [0 (1)], A: [4 (4)]
> [testng] 190286 [TRACE] GMS: - A: mcasting view MergeView::[A|5] [A, B, C], subgroups=[A|4] [A, C, B], [A|4] [A, C, B] (3 mbrs)
> [testng]
> [testng] 190286 [TRACE] GMS: - B: mcasting view MergeView::[A|5] [A, B, C], subgroups=[A|4] [A, C, B], [A|4] [A, C, B] (3 mbrs)
> [testng]
> [testng] 190317 [DEBUG] GMS: - A: installing view MergeView::[A|5] [A, B, C], subgroups=[A|4] [A, C, B], [A|4] [A, C, B]
> [testng] 190318 [DEBUG] GMS: - B: installing view MergeView::[A|5] [A, B, C], subgroups=[A|4] [A, C, B], [A|4] [A, C, B]
> [testng] 190318 [DEBUG] GMS: - C: installing view MergeView::[A|5] [A, B, C], subgroups=[A|4] [A, C, B], [A|4] [A, C, B]
> [testng] 190320 [TRACE] GMS: - A: received all 3 ACKs from members for view [A|5]
> [testng] 190320 [TRACE] GMS: - B: received all 3 ACKs from members for view [A|5]
> [testng] A: [A|5] [A, B, C] (coord=true)
> [testng] B: [A|5] [A, B, C] (coord=false)
> [testng] C: [A|5] [A, B, C] (coord=false)
> [testng] D: [B|4] [B, A, C, D] (coord=false)
> [testng] 195277 [DEBUG] GMS: - D: sending LEAVE request to B
> [testng] FAIL: [1] org.jgroups.tests.OverlappingMergeTest.testMergeWithDifferentPartitions()
> {noformat}
> Whenever this test fails, I see that the queues are suspended on the initial merge attempt.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
More information about the jboss-jira
mailing list