12:10 p.m.
David Hotham created JGRP-1455:
----------------------------------
Summary: Message lost in NAKACK due to digest error
Key: JGRP-1455
URL: https://issues.jboss.org/browse/JGRP-1455
Project: JGroups
Issue Type: Bug
Affects Versions: 3.0.9
Reporter: David Hotham
Assignee: Bela Ban
Hello,
In this issue an application-level message broadcast to the cluster is being discarded by
NAKACK2, on a new joiner.
I think I understand roughly what's going on - skip to the end for a suggested fix!
I'll keep all my trace so that I can investigate further details if needed.
So, let's start with trace from the new joiner (CFS-B-chucklebrothers), showing that:
- it sets a digest claiming that the sequence numbers for CFS-A-tinkywinky are 26 (26)
- CFS-A-tinkywinky then sends messages with sequence numbers 26 and 27
- Only message 27 is passed upwards
{noformat}
2012-04-18 19:26:49.133 [ForkJoinPool-1-worker-3] DEBUG
org.jgroups.protocols.pbcast.NAKACK2 -
[CFS-B-chucklebrothers setDigest()]
existing digest: []
new digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
resulting digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=39 bytes], headers are SEQUENCER: WRAPPED_BCAST
(tag=[CFS-B-chucklebrothers|0]), NAKACK2: [MSG, seqno=26], TCP: [channel_name=Clumpy Test
Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=51 bytes], headers are SEQUENCER: BCAST (tag=[CFS-A-tinkywinky|11]),
NAKACK2: [MSG, seqno=27], TCP: [channel_name=Clumpy Test Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.pbcast.NAKACK2 - CFS-B-chucklebrothers: received
CFS-A-tinkywinky#27
{noformat}
And here's the trace from CFS-A-tinkywinky showing that:
- the digest that it sent only claimed sequence numbers 25 (25)
{noformat}
2012-04-18 19:26:49.132 [OOB-1,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.TCP - sending msg to CFS-B-chucklebrothers, src=CFS-A-tinkywinky,
headers are GMS: GmsHeader[JOIN_RSP]: join_rsp=view: [CFS-A-tinkywinky|3]
[CFS-A-tinkywinky, CFS-A-chucklebrothers, CFS-B-tinkywinky, CFS-B-chucklebrothers],
digest: CFS-A-tinkywinky: [25 (25)], CFS-A-chucklebrothers: [0 (0)], CFS-B-tinkywinky: [0
(0)], CFS-B-chucklebrothers: [0 (0)], UNICAST2: DATA, seqno=2, conn_id=3, TCP:
[channel_name=Clumpy Test Cluster]
{noformat}
By looking at trace from the other members receiving message 26, I can see that this is an
application level message.
I think that the incrementing of the received sequence number is deliberate, per
ClientGmsImpl ("see doc/design/varia2.txt for details"). If I understand
correctly, it's intended to compensate for the fact that the the digest doesn't
include the broadcast VIEW message.
However, CFS-A-tinkywinky shows this:
{noformat}
2012-04-18 19:26:49.125 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] WARN
org.jgroups.protocols.pbcast.GMS - CFS-B-chucklebrothers already present; returning
existing view [CFS-A-tinkywinky|3] [CFS-A-tinkywinky, CFS-A-chucklebrothers,
CFS-B-tinkywinky, CFS-B-chucklebrothers]
2012-04-18 19:26:49.126 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.pbcast.GMS - found no members to add or remove, will not create new
view
{noformat}
My thinking is that since the coordinator does not broadcast a VIEW message, it's a
mistake for CFS-B-chucklebrothers to have fixed up the digest.
Possibly the fix is simply to remove the block of code in
CoordGmsImpl.handleMembershipChange() for "found no members to add or remove",
and send out a view anyway?
Thanks!
David
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:10 p.m.
New subject: [JBoss JIRA] (JGRP-1455) Message lost in NAKACK2 due to digest error
[
https://issues.jboss.org/browse/JGRP-1455?page=com.atlassian.jira.plugin....
]
David Hotham updated JGRP-1455:
-------------------------------
Summary: Message lost in NAKACK2 due to digest error (was: Message lost in NAKACK due
to digest error)
Message lost in NAKACK2 due to digest error
-------------------------------------------
Key: JGRP-1455
URL: https://issues.jboss.org/browse/JGRP-1455
Project: JGroups
Issue Type: Bug
Affects Versions: 3.0.9
Reporter: David Hotham
Assignee: Bela Ban
Hello,
In this issue an application-level message broadcast to the cluster is being discarded by
NAKACK2, on a new joiner.
I think I understand roughly what's going on - skip to the end for a suggested fix!
I'll keep all my trace so that I can investigate further details if needed.
So, let's start with trace from the new joiner (CFS-B-chucklebrothers), showing
that:
- it sets a digest claiming that the sequence numbers for CFS-A-tinkywinky are 26 (26)
- CFS-A-tinkywinky then sends messages with sequence numbers 26 and 27
- Only message 27 is passed upwards
{noformat}
2012-04-18 19:26:49.133 [ForkJoinPool-1-worker-3] DEBUG
org.jgroups.protocols.pbcast.NAKACK2 -
[CFS-B-chucklebrothers setDigest()]
existing digest: []
new digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
resulting digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=39 bytes], headers are SEQUENCER: WRAPPED_BCAST
(tag=[CFS-B-chucklebrothers|0]), NAKACK2: [MSG, seqno=26], TCP: [channel_name=Clumpy Test
Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=51 bytes], headers are SEQUENCER: BCAST (tag=[CFS-A-tinkywinky|11]),
NAKACK2: [MSG, seqno=27], TCP: [channel_name=Clumpy Test Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.pbcast.NAKACK2 - CFS-B-chucklebrothers: received
CFS-A-tinkywinky#27
{noformat}
And here's the trace from CFS-A-tinkywinky showing that:
- the digest that it sent only claimed sequence numbers 25 (25)
{noformat}
2012-04-18 19:26:49.132 [OOB-1,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.TCP - sending msg to CFS-B-chucklebrothers, src=CFS-A-tinkywinky,
headers are GMS: GmsHeader[JOIN_RSP]: join_rsp=view: [CFS-A-tinkywinky|3]
[CFS-A-tinkywinky, CFS-A-chucklebrothers, CFS-B-tinkywinky, CFS-B-chucklebrothers],
digest: CFS-A-tinkywinky: [25 (25)], CFS-A-chucklebrothers: [0 (0)], CFS-B-tinkywinky: [0
(0)], CFS-B-chucklebrothers: [0 (0)], UNICAST2: DATA, seqno=2, conn_id=3, TCP:
[channel_name=Clumpy Test Cluster]
{noformat}
By looking at trace from the other members receiving message 26, I can see that this is
an application level message.
I think that the incrementing of the received sequence number is deliberate, per
ClientGmsImpl ("see doc/design/varia2.txt for details"). If I understand
correctly, it's intended to compensate for the fact that the the digest doesn't
include the broadcast VIEW message.
However, CFS-A-tinkywinky shows this:
{noformat}
2012-04-18 19:26:49.125 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] WARN
org.jgroups.protocols.pbcast.GMS - CFS-B-chucklebrothers already present; returning
existing view [CFS-A-tinkywinky|3] [CFS-A-tinkywinky, CFS-A-chucklebrothers,
CFS-B-tinkywinky, CFS-B-chucklebrothers]
2012-04-18 19:26:49.126 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.pbcast.GMS - found no members to add or remove, will not create new
view
{noformat}
My thinking is that since the coordinator does not broadcast a VIEW message, it's a
mistake for CFS-B-chucklebrothers to have fixed up the digest.
Possibly the fix is simply to remove the block of code in
CoordGmsImpl.handleMembershipChange() for "found no members to add or remove",
and send out a view anyway?
Thanks!
David
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:43 p.m.
New subject: [JBoss JIRA] (JGRP-1455) Message lost in NAKACK2 due to digest error
[
https://issues.jboss.org/browse/JGRP-1455?page=com.atlassian.jira.plugin....
]
Bela Ban updated JGRP-1455:
---------------------------
Fix Version/s: 3.0.10
3.1
Message lost in NAKACK2 due to digest error
-------------------------------------------
Key: JGRP-1455
URL: https://issues.jboss.org/browse/JGRP-1455
Project: JGroups
Issue Type: Bug
Affects Versions: 3.0.9
Reporter: David Hotham
Assignee: Bela Ban
Fix For: 3.0.10, 3.1
Hello,
In this issue an application-level message broadcast to the cluster is being discarded by
NAKACK2, on a new joiner.
I think I understand roughly what's going on - skip to the end for a suggested fix!
I'll keep all my trace so that I can investigate further details if needed.
So, let's start with trace from the new joiner (CFS-B-chucklebrothers), showing
that:
- it sets a digest claiming that the sequence numbers for CFS-A-tinkywinky are 26 (26)
- CFS-A-tinkywinky then sends messages with sequence numbers 26 and 27
- Only message 27 is passed upwards
{noformat}
2012-04-18 19:26:49.133 [ForkJoinPool-1-worker-3] DEBUG
org.jgroups.protocols.pbcast.NAKACK2 -
[CFS-B-chucklebrothers setDigest()]
existing digest: []
new digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
resulting digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=39 bytes], headers are SEQUENCER: WRAPPED_BCAST
(tag=[CFS-B-chucklebrothers|0]), NAKACK2: [MSG, seqno=26], TCP: [channel_name=Clumpy Test
Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=51 bytes], headers are SEQUENCER: BCAST (tag=[CFS-A-tinkywinky|11]),
NAKACK2: [MSG, seqno=27], TCP: [channel_name=Clumpy Test Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.pbcast.NAKACK2 - CFS-B-chucklebrothers: received
CFS-A-tinkywinky#27
{noformat}
And here's the trace from CFS-A-tinkywinky showing that:
- the digest that it sent only claimed sequence numbers 25 (25)
{noformat}
2012-04-18 19:26:49.132 [OOB-1,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.TCP - sending msg to CFS-B-chucklebrothers, src=CFS-A-tinkywinky,
headers are GMS: GmsHeader[JOIN_RSP]: join_rsp=view: [CFS-A-tinkywinky|3]
[CFS-A-tinkywinky, CFS-A-chucklebrothers, CFS-B-tinkywinky, CFS-B-chucklebrothers],
digest: CFS-A-tinkywinky: [25 (25)], CFS-A-chucklebrothers: [0 (0)], CFS-B-tinkywinky: [0
(0)], CFS-B-chucklebrothers: [0 (0)], UNICAST2: DATA, seqno=2, conn_id=3, TCP:
[channel_name=Clumpy Test Cluster]
{noformat}
By looking at trace from the other members receiving message 26, I can see that this is
an application level message.
I think that the incrementing of the received sequence number is deliberate, per
ClientGmsImpl ("see doc/design/varia2.txt for details"). If I understand
correctly, it's intended to compensate for the fact that the the digest doesn't
include the broadcast VIEW message.
However, CFS-A-tinkywinky shows this:
{noformat}
2012-04-18 19:26:49.125 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] WARN
org.jgroups.protocols.pbcast.GMS - CFS-B-chucklebrothers already present; returning
existing view [CFS-A-tinkywinky|3] [CFS-A-tinkywinky, CFS-A-chucklebrothers,
CFS-B-tinkywinky, CFS-B-chucklebrothers]
2012-04-18 19:26:49.126 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.pbcast.GMS - found no members to add or remove, will not create new
view
{noformat}
My thinking is that since the coordinator does not broadcast a VIEW message, it's a
mistake for CFS-B-chucklebrothers to have fixed up the digest.
Possibly the fix is simply to remove the block of code in
CoordGmsImpl.handleMembershipChange() for "found no members to add or remove",
and send out a view anyway?
Thanks!
David
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:08 a.m.
New subject: [JBoss JIRA] (JGRP-1455) Message lost in NAKACK2 due to digest error
[
https://issues.jboss.org/browse/JGRP-1455?page=com.atlassian.jira.plugin....
]
Bela Ban commented on JGRP-1455:
--------------------------------
Hmm, I'm wondering if simply not incrementing the digest wouldn't be the better
and simpler solution: if a joiner gets #25 and then gets #25 again, it will simply discard
it. Perhaps changing the log level from WARN to TRACE on duplicate messages would be
needed...
Message lost in NAKACK2 due to digest error
-------------------------------------------
Key: JGRP-1455
URL: https://issues.jboss.org/browse/JGRP-1455
Project: JGroups
Issue Type: Bug
Affects Versions: 3.0.9
Reporter: David Hotham
Assignee: Bela Ban
Fix For: 3.0.10, 3.1
Hello,
In this issue an application-level message broadcast to the cluster is being discarded by
NAKACK2, on a new joiner.
I think I understand roughly what's going on - skip to the end for a suggested fix!
I'll keep all my trace so that I can investigate further details if needed.
So, let's start with trace from the new joiner (CFS-B-chucklebrothers), showing
that:
- it sets a digest claiming that the sequence numbers for CFS-A-tinkywinky are 26 (26)
- CFS-A-tinkywinky then sends messages with sequence numbers 26 and 27
- Only message 27 is passed upwards
{noformat}
2012-04-18 19:26:49.133 [ForkJoinPool-1-worker-3] DEBUG
org.jgroups.protocols.pbcast.NAKACK2 -
[CFS-B-chucklebrothers setDigest()]
existing digest: []
new digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
resulting digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=39 bytes], headers are SEQUENCER: WRAPPED_BCAST
(tag=[CFS-B-chucklebrothers|0]), NAKACK2: [MSG, seqno=26], TCP: [channel_name=Clumpy Test
Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=51 bytes], headers are SEQUENCER: BCAST (tag=[CFS-A-tinkywinky|11]),
NAKACK2: [MSG, seqno=27], TCP: [channel_name=Clumpy Test Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.pbcast.NAKACK2 - CFS-B-chucklebrothers: received
CFS-A-tinkywinky#27
{noformat}
And here's the trace from CFS-A-tinkywinky showing that:
- the digest that it sent only claimed sequence numbers 25 (25)
{noformat}
2012-04-18 19:26:49.132 [OOB-1,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.TCP - sending msg to CFS-B-chucklebrothers, src=CFS-A-tinkywinky,
headers are GMS: GmsHeader[JOIN_RSP]: join_rsp=view: [CFS-A-tinkywinky|3]
[CFS-A-tinkywinky, CFS-A-chucklebrothers, CFS-B-tinkywinky, CFS-B-chucklebrothers],
digest: CFS-A-tinkywinky: [25 (25)], CFS-A-chucklebrothers: [0 (0)], CFS-B-tinkywinky: [0
(0)], CFS-B-chucklebrothers: [0 (0)], UNICAST2: DATA, seqno=2, conn_id=3, TCP:
[channel_name=Clumpy Test Cluster]
{noformat}
By looking at trace from the other members receiving message 26, I can see that this is
an application level message.
I think that the incrementing of the received sequence number is deliberate, per
ClientGmsImpl ("see doc/design/varia2.txt for details"). If I understand
correctly, it's intended to compensate for the fact that the the digest doesn't
include the broadcast VIEW message.
However, CFS-A-tinkywinky shows this:
{noformat}
2012-04-18 19:26:49.125 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] WARN
org.jgroups.protocols.pbcast.GMS - CFS-B-chucklebrothers already present; returning
existing view [CFS-A-tinkywinky|3] [CFS-A-tinkywinky, CFS-A-chucklebrothers,
CFS-B-tinkywinky, CFS-B-chucklebrothers]
2012-04-18 19:26:49.126 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.pbcast.GMS - found no members to add or remove, will not create new
view
{noformat}
My thinking is that since the coordinator does not broadcast a VIEW message, it's a
mistake for CFS-B-chucklebrothers to have fixed up the digest.
Possibly the fix is simply to remove the block of code in
CoordGmsImpl.handleMembershipChange() for "found no members to add or remove",
and send out a view anyway?
Thanks!
David
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:38 a.m.
New subject: [JBoss JIRA] (JGRP-1455) Message lost in NAKACK2 due to digest error
[
https://issues.jboss.org/browse/JGRP-1455?page=com.atlassian.jira.plugin....
]
Bela Ban commented on JGRP-1455:
--------------------------------
The kludge with incrementing the coord's seqno probably doesn't work anyway:
- A (the coord) gets its digest (A:5); A's next seqno to be expected is #6
- A multicasts application messages A:6 and A:7
- A multicasts the new view for D (message A:8)
- A unicasts the join response to D with digest A:5)
- D increments A:5 to A:6
--> A:6 was an application message, and not the view, so this is clearly wrong !
--> Perhaps returning A:5 in the digest, *not* incrementing it at the joiner and
silently discarding the duplicate view (as suggested above) is the solution to this.
Message lost in NAKACK2 due to digest error
-------------------------------------------
Key: JGRP-1455
URL: https://issues.jboss.org/browse/JGRP-1455
Project: JGroups
Issue Type: Bug
Affects Versions: 3.0.9
Reporter: David Hotham
Assignee: Bela Ban
Fix For: 3.0.10, 3.1
Hello,
In this issue an application-level message broadcast to the cluster is being discarded by
NAKACK2, on a new joiner.
I think I understand roughly what's going on - skip to the end for a suggested fix!
I'll keep all my trace so that I can investigate further details if needed.
So, let's start with trace from the new joiner (CFS-B-chucklebrothers), showing
that:
- it sets a digest claiming that the sequence numbers for CFS-A-tinkywinky are 26 (26)
- CFS-A-tinkywinky then sends messages with sequence numbers 26 and 27
- Only message 27 is passed upwards
{noformat}
2012-04-18 19:26:49.133 [ForkJoinPool-1-worker-3] DEBUG
org.jgroups.protocols.pbcast.NAKACK2 -
[CFS-B-chucklebrothers setDigest()]
existing digest: []
new digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
resulting digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=39 bytes], headers are SEQUENCER: WRAPPED_BCAST
(tag=[CFS-B-chucklebrothers|0]), NAKACK2: [MSG, seqno=26], TCP: [channel_name=Clumpy Test
Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=51 bytes], headers are SEQUENCER: BCAST (tag=[CFS-A-tinkywinky|11]),
NAKACK2: [MSG, seqno=27], TCP: [channel_name=Clumpy Test Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.pbcast.NAKACK2 - CFS-B-chucklebrothers: received
CFS-A-tinkywinky#27
{noformat}
And here's the trace from CFS-A-tinkywinky showing that:
- the digest that it sent only claimed sequence numbers 25 (25)
{noformat}
2012-04-18 19:26:49.132 [OOB-1,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.TCP - sending msg to CFS-B-chucklebrothers, src=CFS-A-tinkywinky,
headers are GMS: GmsHeader[JOIN_RSP]: join_rsp=view: [CFS-A-tinkywinky|3]
[CFS-A-tinkywinky, CFS-A-chucklebrothers, CFS-B-tinkywinky, CFS-B-chucklebrothers],
digest: CFS-A-tinkywinky: [25 (25)], CFS-A-chucklebrothers: [0 (0)], CFS-B-tinkywinky: [0
(0)], CFS-B-chucklebrothers: [0 (0)], UNICAST2: DATA, seqno=2, conn_id=3, TCP:
[channel_name=Clumpy Test Cluster]
{noformat}
By looking at trace from the other members receiving message 26, I can see that this is
an application level message.
I think that the incrementing of the received sequence number is deliberate, per
ClientGmsImpl ("see doc/design/varia2.txt for details"). If I understand
correctly, it's intended to compensate for the fact that the the digest doesn't
include the broadcast VIEW message.
However, CFS-A-tinkywinky shows this:
{noformat}
2012-04-18 19:26:49.125 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] WARN
org.jgroups.protocols.pbcast.GMS - CFS-B-chucklebrothers already present; returning
existing view [CFS-A-tinkywinky|3] [CFS-A-tinkywinky, CFS-A-chucklebrothers,
CFS-B-tinkywinky, CFS-B-chucklebrothers]
2012-04-18 19:26:49.126 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.pbcast.GMS - found no members to add or remove, will not create new
view
{noformat}
My thinking is that since the coordinator does not broadcast a VIEW message, it's a
mistake for CFS-B-chucklebrothers to have fixed up the digest.
Possibly the fix is simply to remove the block of code in
CoordGmsImpl.handleMembershipChange() for "found no members to add or remove",
and send out a view anyway?
Thanks!
David
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:15 a.m.
New subject: [JBoss JIRA] (JGRP-1455) Message lost in NAKACK2 due to digest error
[
https://issues.jboss.org/browse/JGRP-1455?page=com.atlassian.jira.plugin....
]
David Hotham commented on JGRP-1455:
------------------------------------
I think you're right. Certainly if the kludge can safely be removed, then that leaves
things much cleaner.
I see now that the [design
notes|https://github.com/belaban/JGroups/blob/master/doc/design/varia2.txt] are addressing
a problem 'in the absence of a retransmission protocol'. So perhaps there simply
isn't any need for this, since NAKACK / NAKACK2 include a retransmission protocol.
Message lost in NAKACK2 due to digest error
-------------------------------------------
Key: JGRP-1455
URL: https://issues.jboss.org/browse/JGRP-1455
Project: JGroups
Issue Type: Bug
Affects Versions: 3.0.9
Reporter: David Hotham
Assignee: Bela Ban
Fix For: 3.0.10, 3.1
Hello,
In this issue an application-level message broadcast to the cluster is being discarded by
NAKACK2, on a new joiner.
I think I understand roughly what's going on - skip to the end for a suggested fix!
I'll keep all my trace so that I can investigate further details if needed.
So, let's start with trace from the new joiner (CFS-B-chucklebrothers), showing
that:
- it sets a digest claiming that the sequence numbers for CFS-A-tinkywinky are 26 (26)
- CFS-A-tinkywinky then sends messages with sequence numbers 26 and 27
- Only message 27 is passed upwards
{noformat}
2012-04-18 19:26:49.133 [ForkJoinPool-1-worker-3] DEBUG
org.jgroups.protocols.pbcast.NAKACK2 -
[CFS-B-chucklebrothers setDigest()]
existing digest: []
new digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
resulting digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=39 bytes], headers are SEQUENCER: WRAPPED_BCAST
(tag=[CFS-B-chucklebrothers|0]), NAKACK2: [MSG, seqno=26], TCP: [channel_name=Clumpy Test
Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=51 bytes], headers are SEQUENCER: BCAST (tag=[CFS-A-tinkywinky|11]),
NAKACK2: [MSG, seqno=27], TCP: [channel_name=Clumpy Test Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.pbcast.NAKACK2 - CFS-B-chucklebrothers: received
CFS-A-tinkywinky#27
{noformat}
And here's the trace from CFS-A-tinkywinky showing that:
- the digest that it sent only claimed sequence numbers 25 (25)
{noformat}
2012-04-18 19:26:49.132 [OOB-1,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.TCP - sending msg to CFS-B-chucklebrothers, src=CFS-A-tinkywinky,
headers are GMS: GmsHeader[JOIN_RSP]: join_rsp=view: [CFS-A-tinkywinky|3]
[CFS-A-tinkywinky, CFS-A-chucklebrothers, CFS-B-tinkywinky, CFS-B-chucklebrothers],
digest: CFS-A-tinkywinky: [25 (25)], CFS-A-chucklebrothers: [0 (0)], CFS-B-tinkywinky: [0
(0)], CFS-B-chucklebrothers: [0 (0)], UNICAST2: DATA, seqno=2, conn_id=3, TCP:
[channel_name=Clumpy Test Cluster]
{noformat}
By looking at trace from the other members receiving message 26, I can see that this is
an application level message.
I think that the incrementing of the received sequence number is deliberate, per
ClientGmsImpl ("see doc/design/varia2.txt for details"). If I understand
correctly, it's intended to compensate for the fact that the the digest doesn't
include the broadcast VIEW message.
However, CFS-A-tinkywinky shows this:
{noformat}
2012-04-18 19:26:49.125 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] WARN
org.jgroups.protocols.pbcast.GMS - CFS-B-chucklebrothers already present; returning
existing view [CFS-A-tinkywinky|3] [CFS-A-tinkywinky, CFS-A-chucklebrothers,
CFS-B-tinkywinky, CFS-B-chucklebrothers]
2012-04-18 19:26:49.126 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.pbcast.GMS - found no members to add or remove, will not create new
view
{noformat}
My thinking is that since the coordinator does not broadcast a VIEW message, it's a
mistake for CFS-B-chucklebrothers to have fixed up the digest.
Possibly the fix is simply to remove the block of code in
CoordGmsImpl.handleMembershipChange() for "found no members to add or remove",
and send out a view anyway?
Thanks!
David
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:39 a.m.
New subject: [JBoss JIRA] (JGRP-1455) Message lost in NAKACK2 due to digest error
[
https://issues.jboss.org/browse/JGRP-1455?page=com.atlassian.jira.plugin....
]
David Hotham commented on JGRP-1455:
------------------------------------
Hi,
Have you been able to convince yourself that your proposed fix is good yet? It has been
holding up well in my testing.
While I have local fixes for nearly all of the issues that I've opened (per the series
of pull requests), I would very much prefer to be running with a version of JGroups that
has been formally released. Can you say anything about if / when you expect to take a
look at JGRP-1449, JGRP-1451, JGRP-1452, this issue, and JGRP-1458?
I do understand that a man who buys free software has no special claim on your time - and
I'm very happy with the help you've been able to give me so far! It would just be
helpful to my own planning to know what your intentions are.
Thanks!
Message lost in NAKACK2 due to digest error
-------------------------------------------
Key: JGRP-1455
URL: https://issues.jboss.org/browse/JGRP-1455
Project: JGroups
Issue Type: Bug
Affects Versions: 3.0.9
Reporter: David Hotham
Assignee: Bela Ban
Fix For: 3.0.10, 3.1
Hello,
In this issue an application-level message broadcast to the cluster is being discarded by
NAKACK2, on a new joiner.
I think I understand roughly what's going on - skip to the end for a suggested fix!
I'll keep all my trace so that I can investigate further details if needed.
So, let's start with trace from the new joiner (CFS-B-chucklebrothers), showing
that:
- it sets a digest claiming that the sequence numbers for CFS-A-tinkywinky are 26 (26)
- CFS-A-tinkywinky then sends messages with sequence numbers 26 and 27
- Only message 27 is passed upwards
{noformat}
2012-04-18 19:26:49.133 [ForkJoinPool-1-worker-3] DEBUG
org.jgroups.protocols.pbcast.NAKACK2 -
[CFS-B-chucklebrothers setDigest()]
existing digest: []
new digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
resulting digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=39 bytes], headers are SEQUENCER: WRAPPED_BCAST
(tag=[CFS-B-chucklebrothers|0]), NAKACK2: [MSG, seqno=26], TCP: [channel_name=Clumpy Test
Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=51 bytes], headers are SEQUENCER: BCAST (tag=[CFS-A-tinkywinky|11]),
NAKACK2: [MSG, seqno=27], TCP: [channel_name=Clumpy Test Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.pbcast.NAKACK2 - CFS-B-chucklebrothers: received
CFS-A-tinkywinky#27
{noformat}
And here's the trace from CFS-A-tinkywinky showing that:
- the digest that it sent only claimed sequence numbers 25 (25)
{noformat}
2012-04-18 19:26:49.132 [OOB-1,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.TCP - sending msg to CFS-B-chucklebrothers, src=CFS-A-tinkywinky,
headers are GMS: GmsHeader[JOIN_RSP]: join_rsp=view: [CFS-A-tinkywinky|3]
[CFS-A-tinkywinky, CFS-A-chucklebrothers, CFS-B-tinkywinky, CFS-B-chucklebrothers],
digest: CFS-A-tinkywinky: [25 (25)], CFS-A-chucklebrothers: [0 (0)], CFS-B-tinkywinky: [0
(0)], CFS-B-chucklebrothers: [0 (0)], UNICAST2: DATA, seqno=2, conn_id=3, TCP:
[channel_name=Clumpy Test Cluster]
{noformat}
By looking at trace from the other members receiving message 26, I can see that this is
an application level message.
I think that the incrementing of the received sequence number is deliberate, per
ClientGmsImpl ("see doc/design/varia2.txt for details"). If I understand
correctly, it's intended to compensate for the fact that the the digest doesn't
include the broadcast VIEW message.
However, CFS-A-tinkywinky shows this:
{noformat}
2012-04-18 19:26:49.125 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] WARN
org.jgroups.protocols.pbcast.GMS - CFS-B-chucklebrothers already present; returning
existing view [CFS-A-tinkywinky|3] [CFS-A-tinkywinky, CFS-A-chucklebrothers,
CFS-B-tinkywinky, CFS-B-chucklebrothers]
2012-04-18 19:26:49.126 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.pbcast.GMS - found no members to add or remove, will not create new
view
{noformat}
My thinking is that since the coordinator does not broadcast a VIEW message, it's a
mistake for CFS-B-chucklebrothers to have fixed up the digest.
Possibly the fix is simply to remove the block of code in
CoordGmsImpl.handleMembershipChange() for "found no members to add or remove",
and send out a view anyway?
Thanks!
David
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:59 a.m.
New subject: [JBoss JIRA] (JGRP-1455) Message lost in NAKACK2 due to digest error
[
https://issues.jboss.org/browse/JGRP-1455?page=com.atlassian.jira.plugin....
]
Bela Ban commented on JGRP-1455:
--------------------------------
OK, so I committed JGRP-1455 and merged it onto master, and I also backported the change
to Branch_JGroups_3_0.
Can you try with the latest 3.0 or 3.1 (master) and see if this works ? The change is
small and in ClientGmsImpl.
If everything works without problems, can you resolve this issue ?
I hope to look into the other SEQUENCER related issues soon (I'm busy preparing for
JBossWorld)...
Message lost in NAKACK2 due to digest error
-------------------------------------------
Key: JGRP-1455
URL: https://issues.jboss.org/browse/JGRP-1455
Project: JGroups
Issue Type: Bug
Affects Versions: 3.0.9
Reporter: David Hotham
Assignee: Bela Ban
Fix For: 3.0.10, 3.1
Hello,
In this issue an application-level message broadcast to the cluster is being discarded by
NAKACK2, on a new joiner.
I think I understand roughly what's going on - skip to the end for a suggested fix!
I'll keep all my trace so that I can investigate further details if needed.
So, let's start with trace from the new joiner (CFS-B-chucklebrothers), showing
that:
- it sets a digest claiming that the sequence numbers for CFS-A-tinkywinky are 26 (26)
- CFS-A-tinkywinky then sends messages with sequence numbers 26 and 27
- Only message 27 is passed upwards
{noformat}
2012-04-18 19:26:49.133 [ForkJoinPool-1-worker-3] DEBUG
org.jgroups.protocols.pbcast.NAKACK2 -
[CFS-B-chucklebrothers setDigest()]
existing digest: []
new digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
resulting digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=39 bytes], headers are SEQUENCER: WRAPPED_BCAST
(tag=[CFS-B-chucklebrothers|0]), NAKACK2: [MSG, seqno=26], TCP: [channel_name=Clumpy Test
Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=51 bytes], headers are SEQUENCER: BCAST (tag=[CFS-A-tinkywinky|11]),
NAKACK2: [MSG, seqno=27], TCP: [channel_name=Clumpy Test Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.pbcast.NAKACK2 - CFS-B-chucklebrothers: received
CFS-A-tinkywinky#27
{noformat}
And here's the trace from CFS-A-tinkywinky showing that:
- the digest that it sent only claimed sequence numbers 25 (25)
{noformat}
2012-04-18 19:26:49.132 [OOB-1,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.TCP - sending msg to CFS-B-chucklebrothers, src=CFS-A-tinkywinky,
headers are GMS: GmsHeader[JOIN_RSP]: join_rsp=view: [CFS-A-tinkywinky|3]
[CFS-A-tinkywinky, CFS-A-chucklebrothers, CFS-B-tinkywinky, CFS-B-chucklebrothers],
digest: CFS-A-tinkywinky: [25 (25)], CFS-A-chucklebrothers: [0 (0)], CFS-B-tinkywinky: [0
(0)], CFS-B-chucklebrothers: [0 (0)], UNICAST2: DATA, seqno=2, conn_id=3, TCP:
[channel_name=Clumpy Test Cluster]
{noformat}
By looking at trace from the other members receiving message 26, I can see that this is
an application level message.
I think that the incrementing of the received sequence number is deliberate, per
ClientGmsImpl ("see doc/design/varia2.txt for details"). If I understand
correctly, it's intended to compensate for the fact that the the digest doesn't
include the broadcast VIEW message.
However, CFS-A-tinkywinky shows this:
{noformat}
2012-04-18 19:26:49.125 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] WARN
org.jgroups.protocols.pbcast.GMS - CFS-B-chucklebrothers already present; returning
existing view [CFS-A-tinkywinky|3] [CFS-A-tinkywinky, CFS-A-chucklebrothers,
CFS-B-tinkywinky, CFS-B-chucklebrothers]
2012-04-18 19:26:49.126 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.pbcast.GMS - found no members to add or remove, will not create new
view
{noformat}
My thinking is that since the coordinator does not broadcast a VIEW message, it's a
mistake for CFS-B-chucklebrothers to have fixed up the digest.
Possibly the fix is simply to remove the block of code in
CoordGmsImpl.handleMembershipChange() for "found no members to add or remove",
and send out a view anyway?
Thanks!
David
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:23 a.m.
New subject: [JBoss JIRA] (JGRP-1455) Message lost in NAKACK2 due to digest error
[
https://issues.jboss.org/browse/JGRP-1455?page=com.atlassian.jira.plugin....
]
David Hotham resolved JGRP-1455.
--------------------------------
Resolution: Done
Thanks - looks good!
(Really I should leave a stress test running for a few hours before closing the issue, but
this fix is near enough to the change I've been running with that I'm comfortable
that basic verification is sufficient.)
I hope that the various SEQUENCER issues are relatively uncontroversial. I have been
running successfully with these fixes for some time, and I can confirm that the system
holds up much better with them than without them.
Message lost in NAKACK2 due to digest error
-------------------------------------------
Key: JGRP-1455
URL: https://issues.jboss.org/browse/JGRP-1455
Project: JGroups
Issue Type: Bug
Affects Versions: 3.0.9
Reporter: David Hotham
Assignee: Bela Ban
Fix For: 3.0.10, 3.1
Hello,
In this issue an application-level message broadcast to the cluster is being discarded by
NAKACK2, on a new joiner.
I think I understand roughly what's going on - skip to the end for a suggested fix!
I'll keep all my trace so that I can investigate further details if needed.
So, let's start with trace from the new joiner (CFS-B-chucklebrothers), showing
that:
- it sets a digest claiming that the sequence numbers for CFS-A-tinkywinky are 26 (26)
- CFS-A-tinkywinky then sends messages with sequence numbers 26 and 27
- Only message 27 is passed upwards
{noformat}
2012-04-18 19:26:49.133 [ForkJoinPool-1-worker-3] DEBUG
org.jgroups.protocols.pbcast.NAKACK2 -
[CFS-B-chucklebrothers setDigest()]
existing digest: []
new digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
resulting digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=39 bytes], headers are SEQUENCER: WRAPPED_BCAST
(tag=[CFS-B-chucklebrothers|0]), NAKACK2: [MSG, seqno=26], TCP: [channel_name=Clumpy Test
Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=51 bytes], headers are SEQUENCER: BCAST (tag=[CFS-A-tinkywinky|11]),
NAKACK2: [MSG, seqno=27], TCP: [channel_name=Clumpy Test Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.pbcast.NAKACK2 - CFS-B-chucklebrothers: received
CFS-A-tinkywinky#27
{noformat}
And here's the trace from CFS-A-tinkywinky showing that:
- the digest that it sent only claimed sequence numbers 25 (25)
{noformat}
2012-04-18 19:26:49.132 [OOB-1,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.TCP - sending msg to CFS-B-chucklebrothers, src=CFS-A-tinkywinky,
headers are GMS: GmsHeader[JOIN_RSP]: join_rsp=view: [CFS-A-tinkywinky|3]
[CFS-A-tinkywinky, CFS-A-chucklebrothers, CFS-B-tinkywinky, CFS-B-chucklebrothers],
digest: CFS-A-tinkywinky: [25 (25)], CFS-A-chucklebrothers: [0 (0)], CFS-B-tinkywinky: [0
(0)], CFS-B-chucklebrothers: [0 (0)], UNICAST2: DATA, seqno=2, conn_id=3, TCP:
[channel_name=Clumpy Test Cluster]
{noformat}
By looking at trace from the other members receiving message 26, I can see that this is
an application level message.
I think that the incrementing of the received sequence number is deliberate, per
ClientGmsImpl ("see doc/design/varia2.txt for details"). If I understand
correctly, it's intended to compensate for the fact that the the digest doesn't
include the broadcast VIEW message.
However, CFS-A-tinkywinky shows this:
{noformat}
2012-04-18 19:26:49.125 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] WARN
org.jgroups.protocols.pbcast.GMS - CFS-B-chucklebrothers already present; returning
existing view [CFS-A-tinkywinky|3] [CFS-A-tinkywinky, CFS-A-chucklebrothers,
CFS-B-tinkywinky, CFS-B-chucklebrothers]
2012-04-18 19:26:49.126 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.pbcast.GMS - found no members to add or remove, will not create new
view
{noformat}
My thinking is that since the coordinator does not broadcast a VIEW message, it's a
mistake for CFS-B-chucklebrothers to have fixed up the digest.
Possibly the fix is simply to remove the block of code in
CoordGmsImpl.handleMembershipChange() for "found no members to add or remove",
and send out a view anyway?
Thanks!
David
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:59 a.m.
New subject: [JBoss JIRA] (JGRP-1455) Message lost in NAKACK2 due to digest error
[
https://issues.jboss.org/browse/JGRP-1455?page=com.atlassian.jira.plugin....
]
Bela Ban commented on JGRP-1455:
--------------------------------
Hmm, this leads to at least a failing test !
Use case is (SharedTransportTest.testSendingOfMessagesAfterChannelClose()):
- Member C joins
- A broadcasts a message M
- After 500ms, we check that A, B and C have received M
- However, M was added to C but not yet delivered because view V={A,B,C} is missing in
C's table
- After (up to) 1000ms, NAKACK2 sends a retransmit request for V to A
- C now gets V and delivers V and M
- This is ca. 1000ms later, so the test fails !
Possible SOLUTION: when C joins, have the GMS layer trigger (e.g. via an event) the
retransmit request immediately.
Message lost in NAKACK2 due to digest error
-------------------------------------------
Key: JGRP-1455
URL: https://issues.jboss.org/browse/JGRP-1455
Project: JGroups
Issue Type: Bug
Affects Versions: 3.0.9
Reporter: David Hotham
Assignee: Bela Ban
Fix For: 3.0.10, 3.1
Hello,
In this issue an application-level message broadcast to the cluster is being discarded by
NAKACK2, on a new joiner.
I think I understand roughly what's going on - skip to the end for a suggested fix!
I'll keep all my trace so that I can investigate further details if needed.
So, let's start with trace from the new joiner (CFS-B-chucklebrothers), showing
that:
- it sets a digest claiming that the sequence numbers for CFS-A-tinkywinky are 26 (26)
- CFS-A-tinkywinky then sends messages with sequence numbers 26 and 27
- Only message 27 is passed upwards
{noformat}
2012-04-18 19:26:49.133 [ForkJoinPool-1-worker-3] DEBUG
org.jgroups.protocols.pbcast.NAKACK2 -
[CFS-B-chucklebrothers setDigest()]
existing digest: []
new digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
resulting digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=39 bytes], headers are SEQUENCER: WRAPPED_BCAST
(tag=[CFS-B-chucklebrothers|0]), NAKACK2: [MSG, seqno=26], TCP: [channel_name=Clumpy Test
Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=51 bytes], headers are SEQUENCER: BCAST (tag=[CFS-A-tinkywinky|11]),
NAKACK2: [MSG, seqno=27], TCP: [channel_name=Clumpy Test Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.pbcast.NAKACK2 - CFS-B-chucklebrothers: received
CFS-A-tinkywinky#27
{noformat}
And here's the trace from CFS-A-tinkywinky showing that:
- the digest that it sent only claimed sequence numbers 25 (25)
{noformat}
2012-04-18 19:26:49.132 [OOB-1,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.TCP - sending msg to CFS-B-chucklebrothers, src=CFS-A-tinkywinky,
headers are GMS: GmsHeader[JOIN_RSP]: join_rsp=view: [CFS-A-tinkywinky|3]
[CFS-A-tinkywinky, CFS-A-chucklebrothers, CFS-B-tinkywinky, CFS-B-chucklebrothers],
digest: CFS-A-tinkywinky: [25 (25)], CFS-A-chucklebrothers: [0 (0)], CFS-B-tinkywinky: [0
(0)], CFS-B-chucklebrothers: [0 (0)], UNICAST2: DATA, seqno=2, conn_id=3, TCP:
[channel_name=Clumpy Test Cluster]
{noformat}
By looking at trace from the other members receiving message 26, I can see that this is
an application level message.
I think that the incrementing of the received sequence number is deliberate, per
ClientGmsImpl ("see doc/design/varia2.txt for details"). If I understand
correctly, it's intended to compensate for the fact that the the digest doesn't
include the broadcast VIEW message.
However, CFS-A-tinkywinky shows this:
{noformat}
2012-04-18 19:26:49.125 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] WARN
org.jgroups.protocols.pbcast.GMS - CFS-B-chucklebrothers already present; returning
existing view [CFS-A-tinkywinky|3] [CFS-A-tinkywinky, CFS-A-chucklebrothers,
CFS-B-tinkywinky, CFS-B-chucklebrothers]
2012-04-18 19:26:49.126 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.pbcast.GMS - found no members to add or remove, will not create new
view
{noformat}
My thinking is that since the coordinator does not broadcast a VIEW message, it's a
mistake for CFS-B-chucklebrothers to have fixed up the digest.
Possibly the fix is simply to remove the block of code in
CoordGmsImpl.handleMembershipChange() for "found no members to add or remove",
and send out a view anyway?
Thanks!
David
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:01 p.m.
New subject: [JBoss JIRA] (JGRP-1455) Message lost in NAKACK2 due to digest error
[
https://issues.jboss.org/browse/JGRP-1455?page=com.atlassian.jira.plugin....
]
Bela Ban reopened JGRP-1455:
----------------------------
We cannot wait for the retransmission timeout to kick in
Message lost in NAKACK2 due to digest error
-------------------------------------------
Key: JGRP-1455
URL: https://issues.jboss.org/browse/JGRP-1455
Project: JGroups
Issue Type: Bug
Affects Versions: 3.0.9
Reporter: David Hotham
Assignee: Bela Ban
Fix For: 3.0.10, 3.1
Hello,
In this issue an application-level message broadcast to the cluster is being discarded by
NAKACK2, on a new joiner.
I think I understand roughly what's going on - skip to the end for a suggested fix!
I'll keep all my trace so that I can investigate further details if needed.
So, let's start with trace from the new joiner (CFS-B-chucklebrothers), showing
that:
- it sets a digest claiming that the sequence numbers for CFS-A-tinkywinky are 26 (26)
- CFS-A-tinkywinky then sends messages with sequence numbers 26 and 27
- Only message 27 is passed upwards
{noformat}
2012-04-18 19:26:49.133 [ForkJoinPool-1-worker-3] DEBUG
org.jgroups.protocols.pbcast.NAKACK2 -
[CFS-B-chucklebrothers setDigest()]
existing digest: []
new digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
resulting digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=39 bytes], headers are SEQUENCER: WRAPPED_BCAST
(tag=[CFS-B-chucklebrothers|0]), NAKACK2: [MSG, seqno=26], TCP: [channel_name=Clumpy Test
Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=51 bytes], headers are SEQUENCER: BCAST (tag=[CFS-A-tinkywinky|11]),
NAKACK2: [MSG, seqno=27], TCP: [channel_name=Clumpy Test Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.pbcast.NAKACK2 - CFS-B-chucklebrothers: received
CFS-A-tinkywinky#27
{noformat}
And here's the trace from CFS-A-tinkywinky showing that:
- the digest that it sent only claimed sequence numbers 25 (25)
{noformat}
2012-04-18 19:26:49.132 [OOB-1,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.TCP - sending msg to CFS-B-chucklebrothers, src=CFS-A-tinkywinky,
headers are GMS: GmsHeader[JOIN_RSP]: join_rsp=view: [CFS-A-tinkywinky|3]
[CFS-A-tinkywinky, CFS-A-chucklebrothers, CFS-B-tinkywinky, CFS-B-chucklebrothers],
digest: CFS-A-tinkywinky: [25 (25)], CFS-A-chucklebrothers: [0 (0)], CFS-B-tinkywinky: [0
(0)], CFS-B-chucklebrothers: [0 (0)], UNICAST2: DATA, seqno=2, conn_id=3, TCP:
[channel_name=Clumpy Test Cluster]
{noformat}
By looking at trace from the other members receiving message 26, I can see that this is
an application level message.
I think that the incrementing of the received sequence number is deliberate, per
ClientGmsImpl ("see doc/design/varia2.txt for details"). If I understand
correctly, it's intended to compensate for the fact that the the digest doesn't
include the broadcast VIEW message.
However, CFS-A-tinkywinky shows this:
{noformat}
2012-04-18 19:26:49.125 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] WARN
org.jgroups.protocols.pbcast.GMS - CFS-B-chucklebrothers already present; returning
existing view [CFS-A-tinkywinky|3] [CFS-A-tinkywinky, CFS-A-chucklebrothers,
CFS-B-tinkywinky, CFS-B-chucklebrothers]
2012-04-18 19:26:49.126 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.pbcast.GMS - found no members to add or remove, will not create new
view
{noformat}
My thinking is that since the coordinator does not broadcast a VIEW message, it's a
mistake for CFS-B-chucklebrothers to have fixed up the digest.
Possibly the fix is simply to remove the block of code in
CoordGmsImpl.handleMembershipChange() for "found no members to add or remove",
and send out a view anyway?
Thanks!
David
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:53 p.m.
New subject: [JBoss JIRA] (JGRP-1455) Message lost in NAKACK2 due to digest error
[
https://issues.jboss.org/browse/JGRP-1455?page=com.atlassian.jira.plugin....
]
Bela Ban commented on JGRP-1455:
--------------------------------
I guess the better (because cleaner) approach is to send the JoinRsp to the joiners
*before* broadcasting the view, because then the joiners *will* receive the view and
install it, so there is no retransmission needed.
I recall though that there was a reason why I broadcast the view *before* returning the
JoinRsp...
Message lost in NAKACK2 due to digest error
-------------------------------------------
Key: JGRP-1455
URL: https://issues.jboss.org/browse/JGRP-1455
Project: JGroups
Issue Type: Bug
Affects Versions: 3.0.9
Reporter: David Hotham
Assignee: Bela Ban
Fix For: 3.0.10, 3.1
Hello,
In this issue an application-level message broadcast to the cluster is being discarded by
NAKACK2, on a new joiner.
I think I understand roughly what's going on - skip to the end for a suggested fix!
I'll keep all my trace so that I can investigate further details if needed.
So, let's start with trace from the new joiner (CFS-B-chucklebrothers), showing
that:
- it sets a digest claiming that the sequence numbers for CFS-A-tinkywinky are 26 (26)
- CFS-A-tinkywinky then sends messages with sequence numbers 26 and 27
- Only message 27 is passed upwards
{noformat}
2012-04-18 19:26:49.133 [ForkJoinPool-1-worker-3] DEBUG
org.jgroups.protocols.pbcast.NAKACK2 -
[CFS-B-chucklebrothers setDigest()]
existing digest: []
new digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
resulting digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=39 bytes], headers are SEQUENCER: WRAPPED_BCAST
(tag=[CFS-B-chucklebrothers|0]), NAKACK2: [MSG, seqno=26], TCP: [channel_name=Clumpy Test
Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=51 bytes], headers are SEQUENCER: BCAST (tag=[CFS-A-tinkywinky|11]),
NAKACK2: [MSG, seqno=27], TCP: [channel_name=Clumpy Test Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.pbcast.NAKACK2 - CFS-B-chucklebrothers: received
CFS-A-tinkywinky#27
{noformat}
And here's the trace from CFS-A-tinkywinky showing that:
- the digest that it sent only claimed sequence numbers 25 (25)
{noformat}
2012-04-18 19:26:49.132 [OOB-1,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.TCP - sending msg to CFS-B-chucklebrothers, src=CFS-A-tinkywinky,
headers are GMS: GmsHeader[JOIN_RSP]: join_rsp=view: [CFS-A-tinkywinky|3]
[CFS-A-tinkywinky, CFS-A-chucklebrothers, CFS-B-tinkywinky, CFS-B-chucklebrothers],
digest: CFS-A-tinkywinky: [25 (25)], CFS-A-chucklebrothers: [0 (0)], CFS-B-tinkywinky: [0
(0)], CFS-B-chucklebrothers: [0 (0)], UNICAST2: DATA, seqno=2, conn_id=3, TCP:
[channel_name=Clumpy Test Cluster]
{noformat}
By looking at trace from the other members receiving message 26, I can see that this is
an application level message.
I think that the incrementing of the received sequence number is deliberate, per
ClientGmsImpl ("see doc/design/varia2.txt for details"). If I understand
correctly, it's intended to compensate for the fact that the the digest doesn't
include the broadcast VIEW message.
However, CFS-A-tinkywinky shows this:
{noformat}
2012-04-18 19:26:49.125 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] WARN
org.jgroups.protocols.pbcast.GMS - CFS-B-chucklebrothers already present; returning
existing view [CFS-A-tinkywinky|3] [CFS-A-tinkywinky, CFS-A-chucklebrothers,
CFS-B-tinkywinky, CFS-B-chucklebrothers]
2012-04-18 19:26:49.126 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.pbcast.GMS - found no members to add or remove, will not create new
view
{noformat}
My thinking is that since the coordinator does not broadcast a VIEW message, it's a
mistake for CFS-B-chucklebrothers to have fixed up the digest.
Possibly the fix is simply to remove the block of code in
CoordGmsImpl.handleMembershipChange() for "found no members to add or remove",
and send out a view anyway?
Thanks!
David
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:30 p.m.
New subject: [JBoss JIRA] (JGRP-1455) Message lost in NAKACK2 due to digest error
[
https://issues.jboss.org/browse/JGRP-1455?page=com.atlassian.jira.plugin....
]
Bela Ban commented on JGRP-1455:
--------------------------------
No, this doesn't work (I *knew* there was a reason) !
If we have C join {A,B} then, JChannel.connect() on C returns as soon as the JoinRsp has
been received by C, and C installs view {A,B,C}.
However, if C checks the views of A and B right after returning from connect(), they might
still be {A,B}, and not {A,B,C} as the view multicast might still be in transit !
Also, if we rely on the viewChange() callback on C to be able to invoke an RPC on {A,B,C}
(which we shouldn't !), this won't work.
Aaaah, I'll trash my (nice) change, back to square one !
Message lost in NAKACK2 due to digest error
-------------------------------------------
Key: JGRP-1455
URL: https://issues.jboss.org/browse/JGRP-1455
Project: JGroups
Issue Type: Bug
Affects Versions: 3.0.9
Reporter: David Hotham
Assignee: Bela Ban
Fix For: 3.0.10, 3.1
Hello,
In this issue an application-level message broadcast to the cluster is being discarded by
NAKACK2, on a new joiner.
I think I understand roughly what's going on - skip to the end for a suggested fix!
I'll keep all my trace so that I can investigate further details if needed.
So, let's start with trace from the new joiner (CFS-B-chucklebrothers), showing
that:
- it sets a digest claiming that the sequence numbers for CFS-A-tinkywinky are 26 (26)
- CFS-A-tinkywinky then sends messages with sequence numbers 26 and 27
- Only message 27 is passed upwards
{noformat}
2012-04-18 19:26:49.133 [ForkJoinPool-1-worker-3] DEBUG
org.jgroups.protocols.pbcast.NAKACK2 -
[CFS-B-chucklebrothers setDigest()]
existing digest: []
new digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
resulting digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=39 bytes], headers are SEQUENCER: WRAPPED_BCAST
(tag=[CFS-B-chucklebrothers|0]), NAKACK2: [MSG, seqno=26], TCP: [channel_name=Clumpy Test
Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=51 bytes], headers are SEQUENCER: BCAST (tag=[CFS-A-tinkywinky|11]),
NAKACK2: [MSG, seqno=27], TCP: [channel_name=Clumpy Test Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.pbcast.NAKACK2 - CFS-B-chucklebrothers: received
CFS-A-tinkywinky#27
{noformat}
And here's the trace from CFS-A-tinkywinky showing that:
- the digest that it sent only claimed sequence numbers 25 (25)
{noformat}
2012-04-18 19:26:49.132 [OOB-1,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.TCP - sending msg to CFS-B-chucklebrothers, src=CFS-A-tinkywinky,
headers are GMS: GmsHeader[JOIN_RSP]: join_rsp=view: [CFS-A-tinkywinky|3]
[CFS-A-tinkywinky, CFS-A-chucklebrothers, CFS-B-tinkywinky, CFS-B-chucklebrothers],
digest: CFS-A-tinkywinky: [25 (25)], CFS-A-chucklebrothers: [0 (0)], CFS-B-tinkywinky: [0
(0)], CFS-B-chucklebrothers: [0 (0)], UNICAST2: DATA, seqno=2, conn_id=3, TCP:
[channel_name=Clumpy Test Cluster]
{noformat}
By looking at trace from the other members receiving message 26, I can see that this is
an application level message.
I think that the incrementing of the received sequence number is deliberate, per
ClientGmsImpl ("see doc/design/varia2.txt for details"). If I understand
correctly, it's intended to compensate for the fact that the the digest doesn't
include the broadcast VIEW message.
However, CFS-A-tinkywinky shows this:
{noformat}
2012-04-18 19:26:49.125 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] WARN
org.jgroups.protocols.pbcast.GMS - CFS-B-chucklebrothers already present; returning
existing view [CFS-A-tinkywinky|3] [CFS-A-tinkywinky, CFS-A-chucklebrothers,
CFS-B-tinkywinky, CFS-B-chucklebrothers]
2012-04-18 19:26:49.126 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.pbcast.GMS - found no members to add or remove, will not create new
view
{noformat}
My thinking is that since the coordinator does not broadcast a VIEW message, it's a
mistake for CFS-B-chucklebrothers to have fixed up the digest.
Possibly the fix is simply to remove the block of code in
CoordGmsImpl.handleMembershipChange() for "found no members to add or remove",
and send out a view anyway?
Thanks!
David
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:03 p.m.
New subject: [JBoss JIRA] (JGRP-1455) Message lost in NAKACK2 due to digest error
[
https://issues.jboss.org/browse/JGRP-1455?page=com.atlassian.jira.plugin....
]
David Hotham commented on JGRP-1455:
------------------------------------
Is the failing test that you see really so bad? Everyone ends up seeing all the messages
that they should; only the new joiner experiences a small delay. This sounds to me like
the least bad solution that we currently have.
If that's really unacceptable to you then I agree that your proposal that C should
immediately request retransmission (without waiting for the timeout) sounds sensible. But
presumably this would be quite a bit more work...
Message lost in NAKACK2 due to digest error
-------------------------------------------
Key: JGRP-1455
URL: https://issues.jboss.org/browse/JGRP-1455
Project: JGroups
Issue Type: Bug
Affects Versions: 3.0.9
Reporter: David Hotham
Assignee: Bela Ban
Fix For: 3.0.10, 3.1
Hello,
In this issue an application-level message broadcast to the cluster is being discarded by
NAKACK2, on a new joiner.
I think I understand roughly what's going on - skip to the end for a suggested fix!
I'll keep all my trace so that I can investigate further details if needed.
So, let's start with trace from the new joiner (CFS-B-chucklebrothers), showing
that:
- it sets a digest claiming that the sequence numbers for CFS-A-tinkywinky are 26 (26)
- CFS-A-tinkywinky then sends messages with sequence numbers 26 and 27
- Only message 27 is passed upwards
{noformat}
2012-04-18 19:26:49.133 [ForkJoinPool-1-worker-3] DEBUG
org.jgroups.protocols.pbcast.NAKACK2 -
[CFS-B-chucklebrothers setDigest()]
existing digest: []
new digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
resulting digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=39 bytes], headers are SEQUENCER: WRAPPED_BCAST
(tag=[CFS-B-chucklebrothers|0]), NAKACK2: [MSG, seqno=26], TCP: [channel_name=Clumpy Test
Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=51 bytes], headers are SEQUENCER: BCAST (tag=[CFS-A-tinkywinky|11]),
NAKACK2: [MSG, seqno=27], TCP: [channel_name=Clumpy Test Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.pbcast.NAKACK2 - CFS-B-chucklebrothers: received
CFS-A-tinkywinky#27
{noformat}
And here's the trace from CFS-A-tinkywinky showing that:
- the digest that it sent only claimed sequence numbers 25 (25)
{noformat}
2012-04-18 19:26:49.132 [OOB-1,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.TCP - sending msg to CFS-B-chucklebrothers, src=CFS-A-tinkywinky,
headers are GMS: GmsHeader[JOIN_RSP]: join_rsp=view: [CFS-A-tinkywinky|3]
[CFS-A-tinkywinky, CFS-A-chucklebrothers, CFS-B-tinkywinky, CFS-B-chucklebrothers],
digest: CFS-A-tinkywinky: [25 (25)], CFS-A-chucklebrothers: [0 (0)], CFS-B-tinkywinky: [0
(0)], CFS-B-chucklebrothers: [0 (0)], UNICAST2: DATA, seqno=2, conn_id=3, TCP:
[channel_name=Clumpy Test Cluster]
{noformat}
By looking at trace from the other members receiving message 26, I can see that this is
an application level message.
I think that the incrementing of the received sequence number is deliberate, per
ClientGmsImpl ("see doc/design/varia2.txt for details"). If I understand
correctly, it's intended to compensate for the fact that the the digest doesn't
include the broadcast VIEW message.
However, CFS-A-tinkywinky shows this:
{noformat}
2012-04-18 19:26:49.125 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] WARN
org.jgroups.protocols.pbcast.GMS - CFS-B-chucklebrothers already present; returning
existing view [CFS-A-tinkywinky|3] [CFS-A-tinkywinky, CFS-A-chucklebrothers,
CFS-B-tinkywinky, CFS-B-chucklebrothers]
2012-04-18 19:26:49.126 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.pbcast.GMS - found no members to add or remove, will not create new
view
{noformat}
My thinking is that since the coordinator does not broadcast a VIEW message, it's a
mistake for CFS-B-chucklebrothers to have fixed up the digest.
Possibly the fix is simply to remove the block of code in
CoordGmsImpl.handleMembershipChange() for "found no members to add or remove",
and send out a view anyway?
Thanks!
David
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:53 p.m.
New subject: [JBoss JIRA] (JGRP-1455) Message lost in NAKACK2 due to digest error
[
https://issues.jboss.org/browse/JGRP-1455?page=com.atlassian.jira.plugin....
]
Bela Ban commented on JGRP-1455:
--------------------------------
I 'fixed' this as follows:
- A has digest A:5,B:0 (B is the joiner)
- A broadcasts view V, now its seqno is A:6
- A sends the JoinRsp and digest to B
- B installs A:5
- B immediately requests retransmission of A:6 // FIX (or kludge if you want to call it
that ! :-))
- A broadcasts A:7
- B receives A:7, stores it
- B receives (the retransmitted) A:6
- B delivers A:6 and A:7
Hmm, this covers only the very basic use case, but if A had broadcast A:6, A:7 and V (A:8)
*before* sending the JoinRsp to B, B would request A:6, but would still have to wait for
the retransmission task to ask for retransmission of A:7 and A:8 !
Perhaps A should send its *actual* highest received seqno (e.g. A:8) with the Digest, and
so B could request retransmission of A:6-8. However, there is always a window (between
getting the actual seqno and B installing it and receiving messages from now on) in which
A can continue sending messages, and B can only get missed messages by retransmission...
Message lost in NAKACK2 due to digest error
-------------------------------------------
Key: JGRP-1455
URL: https://issues.jboss.org/browse/JGRP-1455
Project: JGroups
Issue Type: Bug
Affects Versions: 3.0.9
Reporter: David Hotham
Assignee: Bela Ban
Fix For: 3.0.10, 3.1
Hello,
In this issue an application-level message broadcast to the cluster is being discarded by
NAKACK2, on a new joiner.
I think I understand roughly what's going on - skip to the end for a suggested fix!
I'll keep all my trace so that I can investigate further details if needed.
So, let's start with trace from the new joiner (CFS-B-chucklebrothers), showing
that:
- it sets a digest claiming that the sequence numbers for CFS-A-tinkywinky are 26 (26)
- CFS-A-tinkywinky then sends messages with sequence numbers 26 and 27
- Only message 27 is passed upwards
{noformat}
2012-04-18 19:26:49.133 [ForkJoinPool-1-worker-3] DEBUG
org.jgroups.protocols.pbcast.NAKACK2 -
[CFS-B-chucklebrothers setDigest()]
existing digest: []
new digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
resulting digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=39 bytes], headers are SEQUENCER: WRAPPED_BCAST
(tag=[CFS-B-chucklebrothers|0]), NAKACK2: [MSG, seqno=26], TCP: [channel_name=Clumpy Test
Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=51 bytes], headers are SEQUENCER: BCAST (tag=[CFS-A-tinkywinky|11]),
NAKACK2: [MSG, seqno=27], TCP: [channel_name=Clumpy Test Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.pbcast.NAKACK2 - CFS-B-chucklebrothers: received
CFS-A-tinkywinky#27
{noformat}
And here's the trace from CFS-A-tinkywinky showing that:
- the digest that it sent only claimed sequence numbers 25 (25)
{noformat}
2012-04-18 19:26:49.132 [OOB-1,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.TCP - sending msg to CFS-B-chucklebrothers, src=CFS-A-tinkywinky,
headers are GMS: GmsHeader[JOIN_RSP]: join_rsp=view: [CFS-A-tinkywinky|3]
[CFS-A-tinkywinky, CFS-A-chucklebrothers, CFS-B-tinkywinky, CFS-B-chucklebrothers],
digest: CFS-A-tinkywinky: [25 (25)], CFS-A-chucklebrothers: [0 (0)], CFS-B-tinkywinky: [0
(0)], CFS-B-chucklebrothers: [0 (0)], UNICAST2: DATA, seqno=2, conn_id=3, TCP:
[channel_name=Clumpy Test Cluster]
{noformat}
By looking at trace from the other members receiving message 26, I can see that this is
an application level message.
I think that the incrementing of the received sequence number is deliberate, per
ClientGmsImpl ("see doc/design/varia2.txt for details"). If I understand
correctly, it's intended to compensate for the fact that the the digest doesn't
include the broadcast VIEW message.
However, CFS-A-tinkywinky shows this:
{noformat}
2012-04-18 19:26:49.125 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] WARN
org.jgroups.protocols.pbcast.GMS - CFS-B-chucklebrothers already present; returning
existing view [CFS-A-tinkywinky|3] [CFS-A-tinkywinky, CFS-A-chucklebrothers,
CFS-B-tinkywinky, CFS-B-chucklebrothers]
2012-04-18 19:26:49.126 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.pbcast.GMS - found no members to add or remove, will not create new
view
{noformat}
My thinking is that since the coordinator does not broadcast a VIEW message, it's a
mistake for CFS-B-chucklebrothers to have fixed up the digest.
Possibly the fix is simply to remove the block of code in
CoordGmsImpl.handleMembershipChange() for "found no members to add or remove",
and send out a view anyway?
Thanks!
David
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:07 p.m.
New subject: [JBoss JIRA] (JGRP-1455) Message lost in NAKACK2 due to digest error
[
https://issues.jboss.org/browse/JGRP-1455?page=com.atlassian.jira.plugin....
]
Bela Ban commented on JGRP-1455:
--------------------------------
OK, I'll leave this as it is, as the quickfix will fix all of the issues in the
testsuite.
This was not done on the 3.0 branch; here we rely on the retransmission task to ask for
retransmission.
Message lost in NAKACK2 due to digest error
-------------------------------------------
Key: JGRP-1455
URL: https://issues.jboss.org/browse/JGRP-1455
Project: JGroups
Issue Type: Bug
Affects Versions: 3.0.9
Reporter: David Hotham
Assignee: Bela Ban
Fix For: 3.0.10, 3.1
Hello,
In this issue an application-level message broadcast to the cluster is being discarded by
NAKACK2, on a new joiner.
I think I understand roughly what's going on - skip to the end for a suggested fix!
I'll keep all my trace so that I can investigate further details if needed.
So, let's start with trace from the new joiner (CFS-B-chucklebrothers), showing
that:
- it sets a digest claiming that the sequence numbers for CFS-A-tinkywinky are 26 (26)
- CFS-A-tinkywinky then sends messages with sequence numbers 26 and 27
- Only message 27 is passed upwards
{noformat}
2012-04-18 19:26:49.133 [ForkJoinPool-1-worker-3] DEBUG
org.jgroups.protocols.pbcast.NAKACK2 -
[CFS-B-chucklebrothers setDigest()]
existing digest: []
new digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
resulting digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=39 bytes], headers are SEQUENCER: WRAPPED_BCAST
(tag=[CFS-B-chucklebrothers|0]), NAKACK2: [MSG, seqno=26], TCP: [channel_name=Clumpy Test
Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=51 bytes], headers are SEQUENCER: BCAST (tag=[CFS-A-tinkywinky|11]),
NAKACK2: [MSG, seqno=27], TCP: [channel_name=Clumpy Test Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.pbcast.NAKACK2 - CFS-B-chucklebrothers: received
CFS-A-tinkywinky#27
{noformat}
And here's the trace from CFS-A-tinkywinky showing that:
- the digest that it sent only claimed sequence numbers 25 (25)
{noformat}
2012-04-18 19:26:49.132 [OOB-1,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.TCP - sending msg to CFS-B-chucklebrothers, src=CFS-A-tinkywinky,
headers are GMS: GmsHeader[JOIN_RSP]: join_rsp=view: [CFS-A-tinkywinky|3]
[CFS-A-tinkywinky, CFS-A-chucklebrothers, CFS-B-tinkywinky, CFS-B-chucklebrothers],
digest: CFS-A-tinkywinky: [25 (25)], CFS-A-chucklebrothers: [0 (0)], CFS-B-tinkywinky: [0
(0)], CFS-B-chucklebrothers: [0 (0)], UNICAST2: DATA, seqno=2, conn_id=3, TCP:
[channel_name=Clumpy Test Cluster]
{noformat}
By looking at trace from the other members receiving message 26, I can see that this is
an application level message.
I think that the incrementing of the received sequence number is deliberate, per
ClientGmsImpl ("see doc/design/varia2.txt for details"). If I understand
correctly, it's intended to compensate for the fact that the the digest doesn't
include the broadcast VIEW message.
However, CFS-A-tinkywinky shows this:
{noformat}
2012-04-18 19:26:49.125 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] WARN
org.jgroups.protocols.pbcast.GMS - CFS-B-chucklebrothers already present; returning
existing view [CFS-A-tinkywinky|3] [CFS-A-tinkywinky, CFS-A-chucklebrothers,
CFS-B-tinkywinky, CFS-B-chucklebrothers]
2012-04-18 19:26:49.126 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.pbcast.GMS - found no members to add or remove, will not create new
view
{noformat}
My thinking is that since the coordinator does not broadcast a VIEW message, it's a
mistake for CFS-B-chucklebrothers to have fixed up the digest.
Possibly the fix is simply to remove the block of code in
CoordGmsImpl.handleMembershipChange() for "found no members to add or remove",
and send out a view anyway?
Thanks!
David
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:07 p.m.
New subject: [JBoss JIRA] (JGRP-1455) Message lost in NAKACK2 due to digest error
[
https://issues.jboss.org/browse/JGRP-1455?page=com.atlassian.jira.plugin....
]
Bela Ban resolved JGRP-1455.
----------------------------
Resolution: Done
Message lost in NAKACK2 due to digest error
-------------------------------------------
Key: JGRP-1455
URL: https://issues.jboss.org/browse/JGRP-1455
Project: JGroups
Issue Type: Bug
Affects Versions: 3.0.9
Reporter: David Hotham
Assignee: Bela Ban
Fix For: 3.0.10, 3.1
Hello,
In this issue an application-level message broadcast to the cluster is being discarded by
NAKACK2, on a new joiner.
I think I understand roughly what's going on - skip to the end for a suggested fix!
I'll keep all my trace so that I can investigate further details if needed.
So, let's start with trace from the new joiner (CFS-B-chucklebrothers), showing
that:
- it sets a digest claiming that the sequence numbers for CFS-A-tinkywinky are 26 (26)
- CFS-A-tinkywinky then sends messages with sequence numbers 26 and 27
- Only message 27 is passed upwards
{noformat}
2012-04-18 19:26:49.133 [ForkJoinPool-1-worker-3] DEBUG
org.jgroups.protocols.pbcast.NAKACK2 -
[CFS-B-chucklebrothers setDigest()]
existing digest: []
new digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
resulting digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=39 bytes], headers are SEQUENCER: WRAPPED_BCAST
(tag=[CFS-B-chucklebrothers|0]), NAKACK2: [MSG, seqno=26], TCP: [channel_name=Clumpy Test
Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=51 bytes], headers are SEQUENCER: BCAST (tag=[CFS-A-tinkywinky|11]),
NAKACK2: [MSG, seqno=27], TCP: [channel_name=Clumpy Test Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.pbcast.NAKACK2 - CFS-B-chucklebrothers: received
CFS-A-tinkywinky#27
{noformat}
And here's the trace from CFS-A-tinkywinky showing that:
- the digest that it sent only claimed sequence numbers 25 (25)
{noformat}
2012-04-18 19:26:49.132 [OOB-1,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.TCP - sending msg to CFS-B-chucklebrothers, src=CFS-A-tinkywinky,
headers are GMS: GmsHeader[JOIN_RSP]: join_rsp=view: [CFS-A-tinkywinky|3]
[CFS-A-tinkywinky, CFS-A-chucklebrothers, CFS-B-tinkywinky, CFS-B-chucklebrothers],
digest: CFS-A-tinkywinky: [25 (25)], CFS-A-chucklebrothers: [0 (0)], CFS-B-tinkywinky: [0
(0)], CFS-B-chucklebrothers: [0 (0)], UNICAST2: DATA, seqno=2, conn_id=3, TCP:
[channel_name=Clumpy Test Cluster]
{noformat}
By looking at trace from the other members receiving message 26, I can see that this is
an application level message.
I think that the incrementing of the received sequence number is deliberate, per
ClientGmsImpl ("see doc/design/varia2.txt for details"). If I understand
correctly, it's intended to compensate for the fact that the the digest doesn't
include the broadcast VIEW message.
However, CFS-A-tinkywinky shows this:
{noformat}
2012-04-18 19:26:49.125 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] WARN
org.jgroups.protocols.pbcast.GMS - CFS-B-chucklebrothers already present; returning
existing view [CFS-A-tinkywinky|3] [CFS-A-tinkywinky, CFS-A-chucklebrothers,
CFS-B-tinkywinky, CFS-B-chucklebrothers]
2012-04-18 19:26:49.126 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.pbcast.GMS - found no members to add or remove, will not create new
view
{noformat}
My thinking is that since the coordinator does not broadcast a VIEW message, it's a
mistake for CFS-B-chucklebrothers to have fixed up the digest.
Possibly the fix is simply to remove the block of code in
CoordGmsImpl.handleMembershipChange() for "found no members to add or remove",
and send out a view anyway?
Thanks!
David
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:02 p.m.
New subject: [JBoss JIRA] (JGRP-1455) Message lost in NAKACK2 due to digest error
[
https://issues.jboss.org/browse/JGRP-1455?page=com.atlassian.jira.plugin....
]
Bela Ban commented on JGRP-1455:
--------------------------------
The solution suggested here was changed in https://issues.jboss.org/browse/JGRP-1509.
Message lost in NAKACK2 due to digest error
-------------------------------------------
Key: JGRP-1455
URL: https://issues.jboss.org/browse/JGRP-1455
Project: JGroups
Issue Type: Bug
Affects Versions: 3.0.9
Reporter: David Hotham
Assignee: Bela Ban
Fix For: 3.0.10, 3.1
Hello,
In this issue an application-level message broadcast to the cluster is being discarded by
NAKACK2, on a new joiner.
I think I understand roughly what's going on - skip to the end for a suggested fix!
I'll keep all my trace so that I can investigate further details if needed.
So, let's start with trace from the new joiner (CFS-B-chucklebrothers), showing
that:
- it sets a digest claiming that the sequence numbers for CFS-A-tinkywinky are 26 (26)
- CFS-A-tinkywinky then sends messages with sequence numbers 26 and 27
- Only message 27 is passed upwards
{noformat}
2012-04-18 19:26:49.133 [ForkJoinPool-1-worker-3] DEBUG
org.jgroups.protocols.pbcast.NAKACK2 -
[CFS-B-chucklebrothers setDigest()]
existing digest: []
new digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
resulting digest: CFS-A-tinkywinky: [26 (26)], CFS-A-chucklebrothers: [0 (0)],
CFS-B-tinkywinky: [0 (0)], CFS-B-chucklebrothers: [0 (0)]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=39 bytes], headers are SEQUENCER: WRAPPED_BCAST
(tag=[CFS-B-chucklebrothers|0]), NAKACK2: [MSG, seqno=26], TCP: [channel_name=Clumpy Test
Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.TCP - received [dst: <null>, src: CFS-A-tinkywinky (3
headers), size=51 bytes], headers are SEQUENCER: BCAST (tag=[CFS-A-tinkywinky|11]),
NAKACK2: [MSG, seqno=27], TCP: [channel_name=Clumpy Test Cluster]
2012-04-18 19:26:49.200 [Incoming-2,Clumpy Test Cluster,CFS-B-chucklebrothers] TRACE
org.jgroups.protocols.pbcast.NAKACK2 - CFS-B-chucklebrothers: received
CFS-A-tinkywinky#27
{noformat}
And here's the trace from CFS-A-tinkywinky showing that:
- the digest that it sent only claimed sequence numbers 25 (25)
{noformat}
2012-04-18 19:26:49.132 [OOB-1,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.TCP - sending msg to CFS-B-chucklebrothers, src=CFS-A-tinkywinky,
headers are GMS: GmsHeader[JOIN_RSP]: join_rsp=view: [CFS-A-tinkywinky|3]
[CFS-A-tinkywinky, CFS-A-chucklebrothers, CFS-B-tinkywinky, CFS-B-chucklebrothers],
digest: CFS-A-tinkywinky: [25 (25)], CFS-A-chucklebrothers: [0 (0)], CFS-B-tinkywinky: [0
(0)], CFS-B-chucklebrothers: [0 (0)], UNICAST2: DATA, seqno=2, conn_id=3, TCP:
[channel_name=Clumpy Test Cluster]
{noformat}
By looking at trace from the other members receiving message 26, I can see that this is
an application level message.
I think that the incrementing of the received sequence number is deliberate, per
ClientGmsImpl ("see doc/design/varia2.txt for details"). If I understand
correctly, it's intended to compensate for the fact that the the digest doesn't
include the broadcast VIEW message.
However, CFS-A-tinkywinky shows this:
{noformat}
2012-04-18 19:26:49.125 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] WARN
org.jgroups.protocols.pbcast.GMS - CFS-B-chucklebrothers already present; returning
existing view [CFS-A-tinkywinky|3] [CFS-A-tinkywinky, CFS-A-chucklebrothers,
CFS-B-tinkywinky, CFS-B-chucklebrothers]
2012-04-18 19:26:49.126 [ViewHandler,Clumpy Test Cluster,CFS-A-tinkywinky] TRACE
org.jgroups.protocols.pbcast.GMS - found no members to add or remove, will not create new
view
{noformat}
My thinking is that since the coordinator does not broadcast a VIEW message, it's a
mistake for CFS-B-chucklebrothers to have fixed up the digest.
Possibly the fix is simply to remove the block of code in
CoordGmsImpl.handleMembershipChange() for "found no members to add or remove",
and send out a view anyway?
Thanks!
David
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
4310
days inactive
4455
days old
17 comments
2 participants
participants (2)
-
Bela Ban (JIRA) -
David Hotham (JIRA)