5 a.m.
Asymmetric link down causes endless retransmission after merge
--------------------------------------------------------------
Key: JGRP-1031
URL: https://jira.jboss.org/jira/browse/JGRP-1031
Project: JGroups
Issue Type: Bug
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.8
Attachments: udp.xml
This is with FD only (don't use FD_ALL and remove FD_SOCK). The config is attached to
this case. We need to add <DISCARD use_gui="true"/> above UDP.
How to reproduce:
- Start 3 Draws:
draw -props ./udp.xml -name 202
draw -props ./udp.xml -name 201
draw -props ./udp.xml -name 203
- On 201: discard all traffic from 202
- Wait until the following views have been installed:
202: 202,203
201: 202,201,203
203: 202,203
Then enable reception of traffic on 201 from 202 again.
Then draw in all 3 instances. Looking at NAKACK, we'll see that the retransmission
table's size increases. We'll also see retransmission requests from 201 to 202
(for messages which seem to be in 202's sent-table !)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-1031) Asymmetric link down causes endless retransmission after merge
[
https://jira.jboss.org/jira/browse/JGRP-1031?page=com.atlassian.jira.plug...
]
Bela Ban updated JGRP-1031:
---------------------------
Attachment: udp.xml
Asymmetric link down causes endless retransmission after merge
--------------------------------------------------------------
Key: JGRP-1031
URL: https://jira.jboss.org/jira/browse/JGRP-1031
Project: JGroups
Issue Type: Bug
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.8
Attachments: udp.xml
This is with FD only (don't use FD_ALL and remove FD_SOCK). The config is attached to
this case. We need to add <DISCARD use_gui="true"/> above UDP.
How to reproduce:
- Start 3 Draws:
draw -props ./udp.xml -name 202
draw -props ./udp.xml -name 201
draw -props ./udp.xml -name 203
- On 201: discard all traffic from 202
- Wait until the following views have been installed:
202: 202,203
201: 202,201,203
203: 202,203
Then enable reception of traffic on 201 from 202 again.
Then draw in all 3 instances. Looking at NAKACK, we'll see that the retransmission
table's size increases. We'll also see retransmission requests from 201 to 202
(for messages which seem to be in 202's sent-table !)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-1031) Asymmetric link down causes endless retransmission after merge
[
https://jira.jboss.org/jira/browse/JGRP-1031?page=com.atlassian.jira.plug...
]
Bela Ban updated JGRP-1031:
---------------------------
Attachment: (was: udp.xml)
Asymmetric link down causes endless retransmission after merge
--------------------------------------------------------------
Key: JGRP-1031
URL: https://jira.jboss.org/jira/browse/JGRP-1031
Project: JGroups
Issue Type: Bug
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.8
Attachments: udp.xml
This is with FD only (don't use FD_ALL and remove FD_SOCK). The config is attached to
this case. We need to add <DISCARD use_gui="true"/> above UDP.
How to reproduce:
- Start 3 Draws:
draw -props ./udp.xml -name 202
draw -props ./udp.xml -name 201
draw -props ./udp.xml -name 203
- On 201: discard all traffic from 202
- Wait until the following views have been installed:
202: 202,203
201: 202,201,203
203: 202,203
Then enable reception of traffic on 201 from 202 again.
Then draw in all 3 instances. Looking at NAKACK, we'll see that the retransmission
table's size increases. We'll also see retransmission requests from 201 to 202
(for messages which seem to be in 202's sent-table !)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-1031) Asymmetric link down causes endless retransmission after merge
[
https://jira.jboss.org/jira/browse/JGRP-1031?page=com.atlassian.jira.plug...
]
Bela Ban updated JGRP-1031:
---------------------------
Attachment: udp.xml
Asymmetric link down causes endless retransmission after merge
--------------------------------------------------------------
Key: JGRP-1031
URL: https://jira.jboss.org/jira/browse/JGRP-1031
Project: JGroups
Issue Type: Bug
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.8
Attachments: udp.xml
This is with FD only (don't use FD_ALL and remove FD_SOCK). The config is attached to
this case. We need to add <DISCARD use_gui="true"/> above UDP.
How to reproduce:
- Start 3 Draws:
draw -props ./udp.xml -name 202
draw -props ./udp.xml -name 201
draw -props ./udp.xml -name 203
- On 201: discard all traffic from 202
- Wait until the following views have been installed:
202: 202,203
201: 202,201,203
203: 202,203
Then enable reception of traffic on 201 from 202 again.
Then draw in all 3 instances. Looking at NAKACK, we'll see that the retransmission
table's size increases. We'll also see retransmission requests from 201 to 202
(for messages which seem to be in 202's sent-table !)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:24 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1031) Asymmetric link down causes endless retransmission after merge
[
https://jira.jboss.org/jira/browse/JGRP-1031?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1031:
--------------------------------
To reproduce, we need to wait (ca 60 secs) for a stability message to purge messages,
before we re-enable 201 to receive traffic from 202. Also send some messages when 201 is
disabled
Asymmetric link down causes endless retransmission after merge
--------------------------------------------------------------
Key: JGRP-1031
URL: https://jira.jboss.org/jira/browse/JGRP-1031
Project: JGroups
Issue Type: Bug
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.8
Attachments: udp.xml
This is with FD only (don't use FD_ALL and remove FD_SOCK). The config is attached to
this case. We need to add <DISCARD use_gui="true"/> above UDP.
How to reproduce:
- Start 3 Draws:
draw -props ./udp.xml -name 202
draw -props ./udp.xml -name 201
draw -props ./udp.xml -name 203
- On 201: discard all traffic from 202
- Wait until the following views have been installed:
202: 202,203
201: 202,201,203
203: 202,203
Then enable reception of traffic on 201 from 202 again.
Then draw in all 3 instances. Looking at NAKACK, we'll see that the retransmission
table's size increases. We'll also see retransmission requests from 201 to 202
(for messages which seem to be in 202's sent-table !)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:32 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1031) Asymmetric link down causes endless retransmission after merge
[
https://jira.jboss.org/jira/browse/JGRP-1031?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1031:
--------------------------------
The issue is the following:
- The view after disabling 201 is:
201: 202, 201, 203
202: 202, 203
203: 202, 203
- Stability (STABLE) goes on between 202 and 203 (because they don't need to wait for
consensus from 201 as it's not part of their view)
- However, 201 does receive the stability messages sent by 203, and therefore removes
seqnos from 202, for example, although it never received them !
- When the merge happens, 201 now is missing messages from 202 and asks for retransmission
!!
Asymmetric link down causes endless retransmission after merge
--------------------------------------------------------------
Key: JGRP-1031
URL: https://jira.jboss.org/jira/browse/JGRP-1031
Project: JGroups
Issue Type: Bug
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.8
Attachments: udp.xml
This is with FD only (don't use FD_ALL and remove FD_SOCK). The config is attached to
this case. We need to add <DISCARD use_gui="true"/> above UDP.
How to reproduce:
- Start 3 Draws:
draw -props ./udp.xml -name 202
draw -props ./udp.xml -name 201
draw -props ./udp.xml -name 203
- On 201: discard all traffic from 202
- Wait until the following views have been installed:
202: 202,203
201: 202,201,203
203: 202,203
Then enable reception of traffic on 201 from 202 again.
Then draw in all 3 instances. Looking at NAKACK, we'll see that the retransmission
table's size increases. We'll also see retransmission requests from 201 to 202
(for messages which seem to be in 202's sent-table !)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:47 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1031) Asymmetric link down causes endless retransmission after merge
[
https://jira.jboss.org/jira/browse/JGRP-1031?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1031:
--------------------------------
Hmm, okay, 202 and 203 purged some messages from their tables. However, when 201 merges,
the digests should be such that 201 gets the digests from 202 and 203, and therefore
should NOT ask for retransmission of missing messages from 202 or 203 !
Asymmetric link down causes endless retransmission after merge
--------------------------------------------------------------
Key: JGRP-1031
URL: https://jira.jboss.org/jira/browse/JGRP-1031
Project: JGroups
Issue Type: Bug
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.8
Attachments: udp.xml
This is with FD only (don't use FD_ALL and remove FD_SOCK). The config is attached to
this case. We need to add <DISCARD use_gui="true"/> above UDP.
How to reproduce:
- Start 3 Draws:
draw -props ./udp.xml -name 202
draw -props ./udp.xml -name 201
draw -props ./udp.xml -name 203
- On 201: discard all traffic from 202
- Wait until the following views have been installed:
202: 202,203
201: 202,201,203
203: 202,203
Then enable reception of traffic on 201 from 202 again.
Then draw in all 3 instances. Looking at NAKACK, we'll see that the retransmission
table's size increases. We'll also see retransmission requests from 201 to 202
(for messages which seem to be in 202's sent-table !)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:09 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1031) Asymmetric link down causes endless retransmission after merge
[
https://jira.jboss.org/jira/browse/JGRP-1031?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1031:
--------------------------------
Looks like the merge view is never *received* by 201 (even though it is re-enabled to
receive traffic from 202) !
Asymmetric link down causes endless retransmission after merge
--------------------------------------------------------------
Key: JGRP-1031
URL: https://jira.jboss.org/jira/browse/JGRP-1031
Project: JGroups
Issue Type: Bug
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.8
Attachments: udp.xml
This is with FD only (don't use FD_ALL and remove FD_SOCK). The config is attached to
this case. We need to add <DISCARD use_gui="true"/> above UDP.
How to reproduce:
- Start 3 Draws:
draw -props ./udp.xml -name 202
draw -props ./udp.xml -name 201
draw -props ./udp.xml -name 203
- On 201: discard all traffic from 202
- Wait until the following views have been installed:
202: 202,203
201: 202,201,203
203: 202,203
Then enable reception of traffic on 201 from 202 again.
Then draw in all 3 instances. Looking at NAKACK, we'll see that the retransmission
table's size increases. We'll also see retransmission requests from 201 to 202
(for messages which seem to be in 202's sent-table !)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:25 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1031) Asymmetric link down causes endless retransmission after merge
[
https://jira.jboss.org/jira/browse/JGRP-1031?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1031:
--------------------------------
Investigate the following:
- 202 has #100, but 201 has only #50 from 202
- The new digest then shows #100 as highest seqno for 202
- However, will the MergeView sent by the merge leader (202) be queued by 201 because it
expects #51 from 202 ?
In other words, should we send the digests via unicasts to all merge participants ?
Is this the reason 201 doesn't receive the MergeView ?
Asymmetric link down causes endless retransmission after merge
--------------------------------------------------------------
Key: JGRP-1031
URL: https://jira.jboss.org/jira/browse/JGRP-1031
Project: JGroups
Issue Type: Bug
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.8
Attachments: udp.xml
This is with FD only (don't use FD_ALL and remove FD_SOCK). The config is attached to
this case. We need to add <DISCARD use_gui="true"/> above UDP.
How to reproduce:
- Start 3 Draws:
draw -props ./udp.xml -name 202
draw -props ./udp.xml -name 201
draw -props ./udp.xml -name 203
- On 201: discard all traffic from 202
- Wait until the following views have been installed:
202: 202,203
201: 202,201,203
203: 202,203
Then enable reception of traffic on 201 from 202 again.
Then draw in all 3 instances. Looking at NAKACK, we'll see that the retransmission
table's size increases. We'll also see retransmission requests from 201 to 202
(for messages which seem to be in 202's sent-table !)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:57 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1031) Asymmetric link down causes endless retransmission after merge
[
https://jira.jboss.org/jira/browse/JGRP-1031?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1031:
--------------------------------
There's a merge between
201: 202,201,203
and
202: 202,203
Now, because 201 is *not* in the coordinator role, the INSTALL_MERGE_VIEW (sent to 201 and
202 by 202 via unicasts) will call handleMergeView(), which is a no-op on 201 !
Asymmetric link down causes endless retransmission after merge
--------------------------------------------------------------
Key: JGRP-1031
URL: https://jira.jboss.org/jira/browse/JGRP-1031
Project: JGroups
Issue Type: Bug
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.8
Attachments: udp.xml
This is with FD only (don't use FD_ALL and remove FD_SOCK). The config is attached to
this case. We need to add <DISCARD use_gui="true"/> above UDP.
How to reproduce:
- Start 3 Draws:
draw -props ./udp.xml -name 202
draw -props ./udp.xml -name 201
draw -props ./udp.xml -name 203
- On 201: discard all traffic from 202
- Wait until the following views have been installed:
202: 202,203
201: 202,201,203
203: 202,203
Then enable reception of traffic on 201 from 202 again.
Then draw in all 3 instances. Looking at NAKACK, we'll see that the retransmission
table's size increases. We'll also see retransmission requests from 201 to 202
(for messages which seem to be in 202's sent-table !)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:19 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1031) Asymmetric link down causes endless retransmission after merge
[
https://jira.jboss.org/jira/browse/JGRP-1031?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1031:
--------------------------------
We have the following situation (members 201, 202 and 203) and their views:
[201]: [202|2] [202, 201, 201]
[202]: [202|3] [202, 201]
[201]: [202|3] [202, 201]
The different views are [202|2] and [202|3} and these are passed up the stack to GMS (by
MERGE2).
The coordinator is 202 and it will multicast a MergeView to all members of the
consolidated view. However, for the reasons outlined above, the multicast will not be
received by 201 as its next seqno expected from 202 is lower than the actual seqno sent by
202. Therefore, 201 will queue the MergeView message up, but it will never receive the
older messages from 202 because 202 and 203 had a stability event which purged older
messages !
The MergView contains a digest, which would correct the situation, but 201 will never
deliver the message, therefore not apply the digest !
Possible solution: in merges with a single coordinator, *unicast* the MergeViews ?
Asymmetric link down causes endless retransmission after merge
--------------------------------------------------------------
Key: JGRP-1031
URL: https://jira.jboss.org/jira/browse/JGRP-1031
Project: JGroups
Issue Type: Bug
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.8
Attachments: udp.xml
This is with FD only (don't use FD_ALL and remove FD_SOCK). The config is attached to
this case. We need to add <DISCARD use_gui="true"/> above UDP.
How to reproduce:
- Start 3 Draws:
draw -props ./udp.xml -name 202
draw -props ./udp.xml -name 201
draw -props ./udp.xml -name 203
- On 201: discard all traffic from 202
- Wait until the following views have been installed:
202: 202,203
201: 202,201,203
203: 202,203
Then enable reception of traffic on 201 from 202 again.
Then draw in all 3 instances. Looking at NAKACK, we'll see that the retransmission
table's size increases. We'll also see retransmission requests from 201 to 202
(for messages which seem to be in 202's sent-table !)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:17 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1031) Asymmetric link down causes endless retransmission after merge
[
https://jira.jboss.org/jira/browse/JGRP-1031?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1031:
--------------------------------
2 questions:
#1 Shouldn't GMS.fixDigests() solve the above issue (once a system is stuck) ?
(No, it doesn't because it only fetches digests from the current partition ! So,
not from 201)
#2 Should we maybe include 202 *and* 201 as merge coordinators ?
- This would entail making certain methods like handleMergeView() or
handleMergeRequest() executable also by participants
and not only by coordinators (CoordGmsImpl)
- What are the criteria for including members in the merge coords set ?
Asymmetric link down causes endless retransmission after merge
--------------------------------------------------------------
Key: JGRP-1031
URL: https://jira.jboss.org/jira/browse/JGRP-1031
Project: JGroups
Issue Type: Bug
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.8
Attachments: udp.xml
This is with FD only (don't use FD_ALL and remove FD_SOCK). The config is attached to
this case. We need to add <DISCARD use_gui="true"/> above UDP.
How to reproduce:
- Start 3 Draws:
draw -props ./udp.xml -name 202
draw -props ./udp.xml -name 201
draw -props ./udp.xml -name 203
- On 201: discard all traffic from 202
- Wait until the following views have been installed:
202: 202,203
201: 202,201,203
203: 202,203
Then enable reception of traffic on 201 from 202 again.
Then draw in all 3 instances. Looking at NAKACK, we'll see that the retransmission
table's size increases. We'll also see retransmission requests from 201 to 202
(for messages which seem to be in 202's sent-table !)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:06 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1031) Asymmetric link down causes endless retransmission after merge
[
https://jira.jboss.org/jira/browse/JGRP-1031?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1031:
--------------------------------
The fix was #2: to include both 202 *and* 201 as merge participants.
Asymmetric link down causes endless retransmission after merge
--------------------------------------------------------------
Key: JGRP-1031
URL: https://jira.jboss.org/jira/browse/JGRP-1031
Project: JGroups
Issue Type: Bug
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.8
Attachments: udp.xml
This is with FD only (don't use FD_ALL and remove FD_SOCK). The config is attached to
this case. We need to add <DISCARD use_gui="true"/> above UDP.
How to reproduce:
- Start 3 Draws:
draw -props ./udp.xml -name 202
draw -props ./udp.xml -name 201
draw -props ./udp.xml -name 203
- On 201: discard all traffic from 202
- Wait until the following views have been installed:
202: 202,203
201: 202,201,203
203: 202,203
Then enable reception of traffic on 201 from 202 again.
Then draw in all 3 instances. Looking at NAKACK, we'll see that the retransmission
table's size increases. We'll also see retransmission requests from 201 to 202
(for messages which seem to be in 202's sent-table !)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:08 a.m.
New subject: [JBoss JIRA] Resolved: (JGRP-1031) Asymmetric link down causes endless retransmission after merge
[
https://jira.jboss.org/jira/browse/JGRP-1031?page=com.atlassian.jira.plug...
]
Bela Ban resolved JGRP-1031.
----------------------------
Resolution: Done
The unit test is GMS_MergeTest.testMergeAsymmetricPartitions()
Asymmetric link down causes endless retransmission after merge
--------------------------------------------------------------
Key: JGRP-1031
URL: https://jira.jboss.org/jira/browse/JGRP-1031
Project: JGroups
Issue Type: Bug
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.8
Attachments: udp.xml
This is with FD only (don't use FD_ALL and remove FD_SOCK). The config is attached to
this case. We need to add <DISCARD use_gui="true"/> above UDP.
How to reproduce:
- Start 3 Draws:
draw -props ./udp.xml -name 202
draw -props ./udp.xml -name 201
draw -props ./udp.xml -name 203
- On 201: discard all traffic from 202
- Wait until the following views have been installed:
202: 202,203
201: 202,201,203
203: 202,203
Then enable reception of traffic on 201 from 202 again.
Then draw in all 3 instances. Looking at NAKACK, we'll see that the retransmission
table's size increases. We'll also see retransmission requests from 201 to 202
(for messages which seem to be in 202's sent-table !)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6155
days inactive
6188
days old
13 comments
1 participants
participants (1)
-
Bela Ban (JIRA)