3:12 a.m.
Problem with retransmission: message not found
----------------------------------------------
Key: JGRP-455
URL: http://jira.jboss.com/jira/browse/JGRP-455
Project: JGroups
Issue Type: Bug
Affects Versions: 2.4.1 SP1
Reporter: Bela Ban
Assigned To: Bela Ban
Fix For: 2.5
Sometimes a sender cannot find a message with a seqno as requested by a receiver. The
seqno requested (47176) was already purged, sent_msgs: [48059 - 48090] (32)
This could be either an issue in STABLE, which prematurely purges messages although not
all members agreed on the highest seqnos seen, or NAKACK, which might incorrectly
'change' seqnos. Very hard to reproduce !
Trace:
42032 [ERROR] [OOB Thread,perf,10.68.32.33:32968] NAKACK.handleXmitReq(): -
(requester=10.68.28.33:33763, local_addr=10.68.32.33:32968) message
10.68.32.33:32968::47176 not found in sent msgs [48059 - 48090] (32)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:52 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-455) Problem with retransmission: message not found
[ http://jira.jboss.com/jira/browse/JGRP-455?page=all ]
Bela Ban updated JGRP-455:
--------------------------
Seems that - when this happens - the low and high_delivered seqnos are reversed:
626622 [ERROR] [OOB Thread,perf,10.68.36.33:32803] NAKACK.handleXmitReq(): -
(requester=10.68.28.33:33780, local_addr=10.68.36.33:32803)
message 10.68.36.33:32803::268648 not found in sent msgs [270516 - 271471] (956)
(stability history:
10.68.20.33:33781: [1001935 : 1001935 (1001937)], 10.68.24.33:33715: [1001935 : 1001935
(1001937)], 10.68.32.33:32986: [925014 : 925014 (931005)], 10.68.36.33:32803: [271451 :
268605 (271453)], 10.68.0.33:33776: [1001981 : 1001981 (1001983)],
10.68.12.33:33820: [353429 : 351738 (353431)], 10.68.16.33:33795: [1001964 : 1001964
(1001965)], 10.68.28.33:33780: [1001986 : 1001986 (1001987)]
10.68.20.33:33781: [1001937 : 1001937 (1001938)], 10.68.24.33:33715: [1001937 : 1001937
(1001939)], 10.68.32.33:32986: [931005 : 931005 (936845)], 10.68.36.33:32803: [271453 :
268605 (271454)], 10.68.0.33:33776: [1001983 : 1001983 (1001984)],
10.68.12.33:33820: [351458 : 351738 (353432)], 10.68.16.33:33795: [1001965 : 1001965
(1001967)], 10.68.28.33:33780: [1001987 : 1001987 (1001988)]
10.68.20.33:33781: [1001938 : 1001938 (1001940)], 10.68.24.33:33715: [1001939 : 1001939
(1001941)], 10.68.32.33:32986: [940013 : 940013 (946844)], 10.68.36.33:32803: [271454 :
268605 (271456)], 10.68.0.33:33776: [1001984 : 1001984 (1001986)],
10.68.12.33:33820: [353432 : 351738 (353434)], 10.68.16.33:33795: [1001967 : 1001967
(1001970)], 10.68.28.33:33780: [1001989 : 1001989 (1001990)]
10.68.20.33:33781: [1001940 : 1001940 (1001941)], 10.68.24.33:33715: [1001941 : 1001941
(1001942)], 10.68.32.33:32986: [946004 : 946004 (951844)], 10.68.36.33:32803: [271456 :
268605 (271458)], 10.68.0.33:33776: [1001986 : 1001986 (1001987)],
10.68.12.33:33820: [351458 : 351738 (353435)], 10.68.16.33:33795: [1001969 : 1001969
(1001971)], 10.68.28.33:33780: [1001990 : 1001990 (1001991)]
10.68.20.33:33781: [1001942 : 1001942 (1001943)], 10.68.24.33:33715: [1001943 : 1001943
(1001944)], 10.68.32.33:32986: [956163 : 957451 (961843)], 10.68.36.33:32803: [268605 :
268605 (271460)], 10.68.0.33:33776: [1001988 : 1001988 (1001989)],
10.68.12.33:33820: [353436 : 351738 (353437)], 10.68.16.33:33795: [1001972 : 1001972
(1001974)], 10.68.28.33:33780: [1001992 : 1001992 (1001993)]
10.68.20.33:33781: [1001943 : 1001943 (1001945)], 10.68.24.33:33715: [1001944 : 1001944
(1001946)], 10.68.32.33:32986: [965012 : 965012 (971001)], 10.68.36.33:32803: [271460 :
268605 (271462)], 10.68.0.33:33776: [1001989 : 1001989 (1001991)],
10.68.12.33:33820: [353437 : 351738 (353439)], 10.68.16.33:33795: [1001974 : 1001974
(1001975)], 10.68.28.33:33780: [1001994 : 1001994 (1001996)]
10.68.20.33:33781: [1001945 : 1001945 (1001948)], 10.68.24.33:33715: [1001946 : 1001946
(1001948)], 10.68.32.33:32986: [975011 : 975011 (981841)], 10.68.36.33:32803: [271462 :
268605 (271464)], 10.68.0.33:33776: [1001991 : 1001991 (1001993)],
10.68.12.33:33820: [353439 : 351738 (353441)], 10.68.16.33:33795: [1001976 : 1001976
(1001978)], 10.68.28.33:33780: [1001997 : 1001997 (1001998)]
10.68.20.33:33781: [1001947 : 1001947 (1001950)], 10.68.24.33:33715: [1001947 : 1001947
(1001949)], 10.68.32.33:32986: [980008 : 980008 (986840)], 10.68.36.33:32803: [271463 :
268605 (271465)], 10.68.0.33:33776: [1001992 : 1001992 (1001994)],
10.68.12.33:33820: [353440 : 351738 (353442)], 10.68.16.33:33795: [1001977 : 1001977
(1001979)], 10.68.28.33:33780: [1001998 : 1001998 (1001999)]
10.68.20.33:33781: [1001952 : 1001952 (1001953)], 10.68.24.33:33715: [1001951 : 1001951
(1001952)], 10.68.32.33:32986: [995999 : 995999 (1001840)], 10.68.36.33:32803: [271467 :
268605 (271468)], 10.68.0.33:33776: [1001996 : 1001996 (1001997)],
10.68.12.33:33820: [351458 : 351738 (353445)], 10.68.16.33:33795: [1001980 : 1001980
(1001982)], 10.68.28.33:33780: [1002001 : 1002001 (1002002)]
10.68.20.33:33781: [1001952 : 1001952 (1001954)], 10.68.24.33:33715: [1001951 : 1001951
(1001954)], 10.68.32.33:32986: [1000009 : 1000009 (1002003)], 10.68.36.33:32803: [271467 :
268605 (271469)], 10.68.0.33:33776: [1001996 : 1001996 (1001999)
], 10.68.12.33:33820: [353444 : 351738 (353447)], 10.68.16.33:33795: [1001981 : 1001981
(1001985)], 10.68.28.33:33780: [1002002 : 1002002 (1002003)]
Problem with retransmission: message not found
----------------------------------------------
Key: JGRP-455
URL: http://jira.jboss.com/jira/browse/JGRP-455
Project: JGroups
Issue Type: Bug
Affects Versions: 2.4.1 SP1
Reporter: Bela Ban
Assigned To: Bela Ban
Fix For: 2.5
Sometimes a sender cannot find a message with a seqno as requested by a receiver. The
seqno requested (47176) was already purged, sent_msgs: [48059 - 48090] (32)
This could be either an issue in STABLE, which prematurely purges messages although not
all members agreed on the highest seqnos seen, or NAKACK, which might incorrectly
'change' seqnos. Very hard to reproduce !
Trace:
42032 [ERROR] [OOB Thread,perf,10.68.32.33:32968] NAKACK.handleXmitReq(): -
(requester=10.68.28.33:33763, local_addr=10.68.32.33:32968) message
10.68.32.33:32968::47176 not found in sent msgs [48059 - 48090] (32)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:42 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-455) Problem with retransmission: message not found
[ http://jira.jboss.com/jira/browse/JGRP-455?page=all ]
Bela Ban updated JGRP-455:
--------------------------
Priority: Critical (was: Major)
Problem with retransmission: message not found
----------------------------------------------
Key: JGRP-455
URL: http://jira.jboss.com/jira/browse/JGRP-455
Project: JGroups
Issue Type: Bug
Affects Versions: 2.4.1 SP1
Reporter: Bela Ban
Assigned To: Bela Ban
Priority: Critical
Fix For: 2.5
Sometimes a sender cannot find a message with a seqno as requested by a receiver. The
seqno requested (47176) was already purged, sent_msgs: [48059 - 48090] (32)
This could be either an issue in STABLE, which prematurely purges messages although not
all members agreed on the highest seqnos seen, or NAKACK, which might incorrectly
'change' seqnos. Very hard to reproduce !
Trace:
42032 [ERROR] [OOB Thread,perf,10.68.32.33:32968] NAKACK.handleXmitReq(): -
(requester=10.68.28.33:33763, local_addr=10.68.32.33:32968) message
10.68.32.33:32968::47176 not found in sent msgs [48059 - 48090] (32)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:40 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-455) Problem with retransmission: message not found
[ http://jira.jboss.com/jira/browse/JGRP-455?page=all ]
Bela Ban updated JGRP-455:
--------------------------
Attachment: tmp2.txt
tmp2.txt shows that message 10.68.0.33:33814::66521 was not found in sent msgs [66632 -
68191]. However, stability shows that the last stability message
10: 10.68.0.33:33814: [66520 : 66520 (68190)],
purged messages with seqnos less than 66520, so seqno 66521 should have been found !
The question is why were sent_msgs purged to 66632 ?
Problem with retransmission: message not found
----------------------------------------------
Key: JGRP-455
URL: http://jira.jboss.com/jira/browse/JGRP-455
Project: JGroups
Issue Type: Bug
Affects Versions: 2.4.1 SP1
Reporter: Bela Ban
Assigned To: Bela Ban
Priority: Critical
Fix For: 2.5
Attachments: tmp2.txt
Sometimes a sender cannot find a message with a seqno as requested by a receiver. The
seqno requested (47176) was already purged, sent_msgs: [48059 - 48090] (32)
This could be either an issue in STABLE, which prematurely purges messages although not
all members agreed on the highest seqnos seen, or NAKACK, which might incorrectly
'change' seqnos. Very hard to reproduce !
Trace:
42032 [ERROR] [OOB Thread,perf,10.68.32.33:32968] NAKACK.handleXmitReq(): -
(requester=10.68.28.33:33763, local_addr=10.68.32.33:32968) message
10.68.32.33:32968::47176 not found in sent msgs [48059 - 48090] (32)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:12 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-455) Problem with retransmission: message not found
[ http://jira.jboss.com/jira/browse/JGRP-455?page=all ]
Bela Ban updated JGRP-455:
--------------------------
Description:
Sometimes a sender cannot find a message with a seqno as requested by a receiver. The
seqno requested (47176) was already purged, sent_msgs: [48059 - 48090] (32)
This could be either an issue in STABLE, which prematurely purges messages although not
all members agreed on the highest seqnos seen, or NAKACK, which might incorrectly
'change' seqnos. Very hard to reproduce !
Trace:
42032 [ERROR] [OOB Thread,perf,10.68.32.33:32968] NAKACK.handleXmitReq(): -
(requester=10.68.28.33:33763, local_addr=10.68.32.33:32968) message
10.68.32.33:32968::47176 not found in sent msgs [48059 - 48090] (32)
was:
Sometimes a sender cannot find a message with a seqno as requested by a receiver. The
seqno requested (47176) was already purged, sent_msgs: [48059 - 48090] (32)
This could be either an issue in STABLE, which prematurely purges messages although not
all members agreed on the highest seqnos seen, or NAKACK, which might incorrectly
'change' seqnos. Very hard to reproduce !
Trace:
42032 [ERROR] [OOB Thread,perf,10.68.32.33:32968] NAKACK.handleXmitReq(): -
(requester=10.68.28.33:33763, local_addr=10.68.32.33:32968) message
10.68.32.33:32968::47176 not found in sent msgs [48059 - 48090] (32)
Priority: Blocker (was: Critical)
Problem with retransmission: message not found
----------------------------------------------
Key: JGRP-455
URL: http://jira.jboss.com/jira/browse/JGRP-455
Project: JGroups
Issue Type: Bug
Affects Versions: 2.4.1 SP1
Reporter: Bela Ban
Assigned To: Bela Ban
Priority: Blocker
Fix For: 2.5
Attachments: tmp2.txt, udp-atl.xml
Sometimes a sender cannot find a message with a seqno as requested by a receiver. The
seqno requested (47176) was already purged, sent_msgs: [48059 - 48090] (32)
This could be either an issue in STABLE, which prematurely purges messages although not
all members agreed on the highest seqnos seen, or NAKACK, which might incorrectly
'change' seqnos. Very hard to reproduce !
Trace:
42032 [ERROR] [OOB Thread,perf,10.68.32.33:32968] NAKACK.handleXmitReq(): -
(requester=10.68.28.33:33763, local_addr=10.68.32.33:32968) message
10.68.32.33:32968::47176 not found in sent msgs [48059 - 48090] (32)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:12 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-455) Problem with retransmission: message not found
[ http://jira.jboss.com/jira/browse/JGRP-455?page=all ]
Bela Ban updated JGRP-455:
--------------------------
Attachment: udp-atl.xml
Problem with retransmission: message not found
----------------------------------------------
Key: JGRP-455
URL: http://jira.jboss.com/jira/browse/JGRP-455
Project: JGroups
Issue Type: Bug
Affects Versions: 2.4.1 SP1
Reporter: Bela Ban
Assigned To: Bela Ban
Priority: Blocker
Fix For: 2.5
Attachments: tmp2.txt, udp-atl.xml
Sometimes a sender cannot find a message with a seqno as requested by a receiver. The
seqno requested (47176) was already purged, sent_msgs: [48059 - 48090] (32)
This could be either an issue in STABLE, which prematurely purges messages although not
all members agreed on the highest seqnos seen, or NAKACK, which might incorrectly
'change' seqnos. Very hard to reproduce !
Trace:
42032 [ERROR] [OOB Thread,perf,10.68.32.33:32968] NAKACK.handleXmitReq(): -
(requester=10.68.28.33:33763, local_addr=10.68.32.33:32968) message
10.68.32.33:32968::47176 not found in sent msgs [48059 - 48090] (32)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:12 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-455) Problem with retransmission: message not found
[ http://jira.jboss.com/jira/browse/JGRP-455?page=comments#action_12361298 ]
Bela Ban commented on JGRP-455:
-------------------------------
With udp-atl, on 8 nodes, perf.Test fails. It *does not fail* if STABLE.max_bytes is set
to 5M instead of 1M !
Problem with retransmission: message not found
----------------------------------------------
Key: JGRP-455
URL: http://jira.jboss.com/jira/browse/JGRP-455
Project: JGroups
Issue Type: Bug
Affects Versions: 2.4.1 SP1
Reporter: Bela Ban
Assigned To: Bela Ban
Priority: Blocker
Fix For: 2.5
Attachments: tmp2.txt, udp-atl.xml
Sometimes a sender cannot find a message with a seqno as requested by a receiver. The
seqno requested (47176) was already purged, sent_msgs: [48059 - 48090] (32)
This could be either an issue in STABLE, which prematurely purges messages although not
all members agreed on the highest seqnos seen, or NAKACK, which might incorrectly
'change' seqnos. Very hard to reproduce !
Trace:
42032 [ERROR] [OOB Thread,perf,10.68.32.33:32968] NAKACK.handleXmitReq(): -
(requester=10.68.28.33:33763, local_addr=10.68.32.33:32968) message
10.68.32.33:32968::47176 not found in sent msgs [48059 - 48090] (32)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:42 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-455) Problem with retransmission: message not found
[ http://jira.jboss.com/jira/browse/JGRP-455?page=comments#action_12361323 ]
Bela Ban commented on JGRP-455:
-------------------------------
This doesn't happen with 2.4.1 SP3; all 8 senders complete, with ca 14'000 1K
messages/sec
Problem with retransmission: message not found
----------------------------------------------
Key: JGRP-455
URL: http://jira.jboss.com/jira/browse/JGRP-455
Project: JGroups
Issue Type: Bug
Affects Versions: 2.4.1 SP1
Reporter: Bela Ban
Assigned To: Bela Ban
Priority: Blocker
Fix For: 2.5
Attachments: tmp2.txt, udp-atl.xml
Sometimes a sender cannot find a message with a seqno as requested by a receiver. The
seqno requested (47176) was already purged, sent_msgs: [48059 - 48090] (32)
This could be either an issue in STABLE, which prematurely purges messages although not
all members agreed on the highest seqnos seen, or NAKACK, which might incorrectly
'change' seqnos. Very hard to reproduce !
Trace:
42032 [ERROR] [OOB Thread,perf,10.68.32.33:32968] NAKACK.handleXmitReq(): -
(requester=10.68.28.33:33763, local_addr=10.68.32.33:32968) message
10.68.32.33:32968::47176 not found in sent msgs [48059 - 48090] (32)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:41 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-455) Problem with retransmission: message not found
[ http://jira.jboss.com/jira/browse/JGRP-455?page=comments#action_12361344 ]
Bela Ban commented on JGRP-455:
-------------------------------
110172 [ERROR] [OOB Thread,perf,10.68.0.33:35861] NAKACK.handleXmitReq(): -
(requester=10.68.28.33:32843, local_addr=10.68.0.33:35861) message 10.68.0.33:35861::23179
not found in retransmission table of 10.68.0.33:35861: [23011 : 39326 (39326) (size=14651,
missing=0)]
Problem with retransmission: message not found
----------------------------------------------
Key: JGRP-455
URL: http://jira.jboss.com/jira/browse/JGRP-455
Project: JGroups
Issue Type: Bug
Affects Versions: 2.4.1 SP1
Reporter: Bela Ban
Assigned To: Bela Ban
Priority: Blocker
Fix For: 2.5
Attachments: tmp2.txt, udp-atl.xml
Sometimes a sender cannot find a message with a seqno as requested by a receiver. The
seqno requested (47176) was already purged, sent_msgs: [48059 - 48090] (32)
This could be either an issue in STABLE, which prematurely purges messages although not
all members agreed on the highest seqnos seen, or NAKACK, which might incorrectly
'change' seqnos. Very hard to reproduce !
Trace:
42032 [ERROR] [OOB Thread,perf,10.68.32.33:32968] NAKACK.handleXmitReq(): -
(requester=10.68.28.33:33763, local_addr=10.68.32.33:32968) message
10.68.32.33:32968::47176 not found in sent msgs [48059 - 48090] (32)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:32 p.m.
New subject: [JBoss JIRA] Updated: (JGRP-455) Problem with retransmission: message not found
[ http://jira.jboss.com/jira/browse/JGRP-455?page=all ]
Bela Ban updated JGRP-455:
--------------------------
Attachment: tmp.txt
Problem with retransmission: message not found
----------------------------------------------
Key: JGRP-455
URL: http://jira.jboss.com/jira/browse/JGRP-455
Project: JGroups
Issue Type: Bug
Affects Versions: 2.4.1 SP1
Reporter: Bela Ban
Assigned To: Bela Ban
Priority: Blocker
Fix For: 2.5
Attachments: tmp.txt, tmp2.txt, udp-atl.xml
Sometimes a sender cannot find a message with a seqno as requested by a receiver. The
seqno requested (47176) was already purged, sent_msgs: [48059 - 48090] (32)
This could be either an issue in STABLE, which prematurely purges messages although not
all members agreed on the highest seqnos seen, or NAKACK, which might incorrectly
'change' seqnos. Very hard to reproduce !
Trace:
42032 [ERROR] [OOB Thread,perf,10.68.32.33:32968] NAKACK.handleXmitReq(): -
(requester=10.68.28.33:33763, local_addr=10.68.32.33:32968) message
10.68.32.33:32968::47176 not found in sent msgs [48059 - 48090] (32)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:39 p.m.
New subject: [JBoss JIRA] Commented: (JGRP-455) Problem with retransmission: message not found
[ http://jira.jboss.com/jira/browse/JGRP-455?page=comments#action_12361805 ]
Bela Ban commented on JGRP-455:
-------------------------------
attached tmp.txt. The issue is that STABLE message #8 has 100698 as highest delivered
seqno for 10.68.0.33, then #9 is 100695, then #10 is 100695 !
Issue: each stability message must be >= the previous one, investigate why #8 is 100698
! This causes message 100698 to get purged, therefore no retransmission. Points to a bug
in STABLE
Problem with retransmission: message not found
----------------------------------------------
Key: JGRP-455
URL: http://jira.jboss.com/jira/browse/JGRP-455
Project: JGroups
Issue Type: Bug
Affects Versions: 2.4.1 SP1
Reporter: Bela Ban
Assigned To: Bela Ban
Priority: Blocker
Fix For: 2.5
Attachments: tmp.txt, tmp2.txt, udp-atl.xml
Sometimes a sender cannot find a message with a seqno as requested by a receiver. The
seqno requested (47176) was already purged, sent_msgs: [48059 - 48090] (32)
This could be either an issue in STABLE, which prematurely purges messages although not
all members agreed on the highest seqnos seen, or NAKACK, which might incorrectly
'change' seqnos. Very hard to reproduce !
Trace:
42032 [ERROR] [OOB Thread,perf,10.68.32.33:32968] NAKACK.handleXmitReq(): -
(requester=10.68.28.33:33763, local_addr=10.68.32.33:32968) message
10.68.32.33:32968::47176 not found in sent msgs [48059 - 48090] (32)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:05 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-455) Problem with retransmission: message not found
[ http://jira.jboss.com/jira/browse/JGRP-455?page=comments#action_12363687 ]
Bela Ban commented on JGRP-455:
-------------------------------
The issue was in STABLE: the heard_from list could be incorrectly updated, so we'd
sometimes send a STABILITY message even though we had not yet received votes from all
members.
I changed the design slightly, so that now instead of heard_from we have votes, which is
empty initially and only when all members have sent their votes, we send a STABILITY
message and clear votes again.
The clearing of votes and resetting of the digest is now done with the same lock
("lock"), as there was a race condition when synchronizing separately on digest
and votes.
Tests with 1M 1K messages per sender on 8 nodes passed multiple times (before the error
occurred after 100K messages).
Problem with retransmission: message not found
----------------------------------------------
Key: JGRP-455
URL: http://jira.jboss.com/jira/browse/JGRP-455
Project: JGroups
Issue Type: Bug
Affects Versions: 2.4.1 SP1
Reporter: Bela Ban
Assigned To: Bela Ban
Priority: Blocker
Fix For: 2.5
Attachments: tmp.txt, tmp2.txt, udp-atl.xml
Sometimes a sender cannot find a message with a seqno as requested by a receiver. The
seqno requested (47176) was already purged, sent_msgs: [48059 - 48090] (32)
This could be either an issue in STABLE, which prematurely purges messages although not
all members agreed on the highest seqnos seen, or NAKACK, which might incorrectly
'change' seqnos. Very hard to reproduce !
Trace:
42032 [ERROR] [OOB Thread,perf,10.68.32.33:32968] NAKACK.handleXmitReq(): -
(requester=10.68.28.33:33763, local_addr=10.68.32.33:32968) message
10.68.32.33:32968::47176 not found in sent msgs [48059 - 48090] (32)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:11 a.m.
New subject: [JBoss JIRA] Resolved: (JGRP-455) Problem with retransmission: message not found
[ http://jira.jboss.com/jira/browse/JGRP-455?page=all ]
Bela Ban resolved JGRP-455.
---------------------------
Resolution: Done
fixed according to proposed solution. Backported to 2.4 (only the changes related to the
mutexes). Tested with 8 nodes in ATL lab, both for 2.5 and 2.4
Problem with retransmission: message not found
----------------------------------------------
Key: JGRP-455
URL: http://jira.jboss.com/jira/browse/JGRP-455
Project: JGroups
Issue Type: Bug
Affects Versions: 2.4.1 SP1
Reporter: Bela Ban
Assigned To: Bela Ban
Priority: Blocker
Fix For: 2.5
Attachments: tmp.txt, tmp2.txt, udp-atl.xml
Sometimes a sender cannot find a message with a seqno as requested by a receiver. The
seqno requested (47176) was already purged, sent_msgs: [48059 - 48090] (32)
This could be either an issue in STABLE, which prematurely purges messages although not
all members agreed on the highest seqnos seen, or NAKACK, which might incorrectly
'change' seqnos. Very hard to reproduce !
Trace:
42032 [ERROR] [OOB Thread,perf,10.68.32.33:32968] NAKACK.handleXmitReq(): -
(requester=10.68.28.33:33763, local_addr=10.68.32.33:32968) message
10.68.32.33:32968::47176 not found in sent msgs [48059 - 48090] (32)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:46 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-455) Problem with retransmission: message not found
[ http://jira.jboss.com/jira/browse/JGRP-455?page=all ]
Galder Zamarreno updated JGRP-455:
----------------------------------
Fix Version/s: 2.4.1 SP4
Problem with retransmission: message not found
----------------------------------------------
Key: JGRP-455
URL: http://jira.jboss.com/jira/browse/JGRP-455
Project: JGroups
Issue Type: Bug
Affects Versions: 2.4.1 SP1
Reporter: Bela Ban
Assigned To: Bela Ban
Priority: Blocker
Fix For: 2.5, 2.4.1 SP4
Attachments: tmp.txt, tmp2.txt, udp-atl.xml
Sometimes a sender cannot find a message with a seqno as requested by a receiver. The
seqno requested (47176) was already purged, sent_msgs: [48059 - 48090] (32)
This could be either an issue in STABLE, which prematurely purges messages although not
all members agreed on the highest seqnos seen, or NAKACK, which might incorrectly
'change' seqnos. Very hard to reproduce !
Trace:
42032 [ERROR] [OOB Thread,perf,10.68.32.33:32968] NAKACK.handleXmitReq(): -
(requester=10.68.28.33:33763, local_addr=10.68.32.33:32968) message
10.68.32.33:32968::47176 not found in sent msgs [48059 - 48090] (32)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:28 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-455) Problem with retransmission: message not found
[ http://jira.jboss.com/jira/browse/JGRP-455?page=comments#action_12364895 ]
Bela Ban commented on JGRP-455:
-------------------------------
Side effect on FC: since a receiver would not deliver any more data, credit requests would
arrive at a receiver after FC.max_block_time and (because we send back the full credits
after two times), after time there would be an OOME
Problem with retransmission: message not found
----------------------------------------------
Key: JGRP-455
URL: http://jira.jboss.com/jira/browse/JGRP-455
Project: JGroups
Issue Type: Bug
Affects Versions: 2.4.1 SP1
Reporter: Bela Ban
Assigned To: Bela Ban
Priority: Blocker
Fix For: 2.5, 2.4.1 SP4
Attachments: tmp.txt, tmp2.txt, udp-atl.xml
Sometimes a sender cannot find a message with a seqno as requested by a receiver. The
seqno requested (47176) was already purged, sent_msgs: [48059 - 48090] (32)
This could be either an issue in STABLE, which prematurely purges messages although not
all members agreed on the highest seqnos seen, or NAKACK, which might incorrectly
'change' seqnos. Very hard to reproduce !
Trace:
42032 [ERROR] [OOB Thread,perf,10.68.32.33:32968] NAKACK.handleXmitReq(): -
(requester=10.68.28.33:33763, local_addr=10.68.32.33:32968) message
10.68.32.33:32968::47176 not found in sent msgs [48059 - 48090] (32)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:59 a.m.
New subject: [JBoss JIRA] Reopened: (JGRP-455) Problem with retransmission: message not found
[ http://jira.jboss.com/jira/browse/JGRP-455?page=all ]
Bela Ban reopened JGRP-455:
---------------------------
Still happens with tcp.xml and 500 threads/sender (-num_threads 500)
Problem with retransmission: message not found
----------------------------------------------
Key: JGRP-455
URL: http://jira.jboss.com/jira/browse/JGRP-455
Project: JGroups
Issue Type: Bug
Affects Versions: 2.4.1 SP1
Reporter: Bela Ban
Assigned To: Bela Ban
Priority: Blocker
Fix For: 2.5, 2.4.1 SP4
Attachments: tmp.txt, tmp2.txt, udp-atl.xml
Sometimes a sender cannot find a message with a seqno as requested by a receiver. The
seqno requested (47176) was already purged, sent_msgs: [48059 - 48090] (32)
This could be either an issue in STABLE, which prematurely purges messages although not
all members agreed on the highest seqnos seen, or NAKACK, which might incorrectly
'change' seqnos. Very hard to reproduce !
Trace:
42032 [ERROR] [OOB Thread,perf,10.68.32.33:32968] NAKACK.handleXmitReq(): -
(requester=10.68.28.33:33763, local_addr=10.68.32.33:32968) message
10.68.32.33:32968::47176 not found in sent msgs [48059 - 48090] (32)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:14 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-455) Problem with retransmission: message not found
[ http://jira.jboss.com/jira/browse/JGRP-455?page=comments#action_12364993 ]
Bela Ban commented on JGRP-455:
-------------------------------
Looks like the retransmission requests are spurious requests due to timers cancelled a bit
too late every now and then. The tests complete, which means that we certainly didn't
hit issue 455, otherwise everyone would have been stuck !
I ran 3 tests on 6 nodes with 6 senders:
- Each sender sends 5M 1K messages, tests complete okay with 48'000 1K messages / sec
/ node (75MB/sec/node)
- ditto: OK
- Each sender sends 5M 5K messages, tests complete okay with 17'800 5K messages / sec
/ node (89MB/sec/node !)
Problem with retransmission: message not found
----------------------------------------------
Key: JGRP-455
URL: http://jira.jboss.com/jira/browse/JGRP-455
Project: JGroups
Issue Type: Bug
Affects Versions: 2.4.1 SP1
Reporter: Bela Ban
Assigned To: Bela Ban
Priority: Blocker
Fix For: 2.5, 2.4.1 SP4
Attachments: tmp.txt, tmp2.txt, udp-atl.xml
Sometimes a sender cannot find a message with a seqno as requested by a receiver. The
seqno requested (47176) was already purged, sent_msgs: [48059 - 48090] (32)
This could be either an issue in STABLE, which prematurely purges messages although not
all members agreed on the highest seqnos seen, or NAKACK, which might incorrectly
'change' seqnos. Very hard to reproduce !
Trace:
42032 [ERROR] [OOB Thread,perf,10.68.32.33:32968] NAKACK.handleXmitReq(): -
(requester=10.68.28.33:33763, local_addr=10.68.32.33:32968) message
10.68.32.33:32968::47176 not found in sent msgs [48059 - 48090] (32)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:14 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-455) Problem with retransmission: message not found
[ http://jira.jboss.com/jira/browse/JGRP-455?page=comments#action_12364995 ]
Bela Ban commented on JGRP-455:
-------------------------------
Results (1K and 5K):
-- results:
10.68.12.33:7800 (myself):
num_msgs_expected=30000000, num_msgs_received=30000000 (loss rate=0.0%), receive
d=150GB, time=1986111ms, msgs/sec=15104.9, throughput=75.52MB
10.68.20.33:7800:
num_msgs_expected=30000000, num_msgs_received=30000000 (loss rate=0.0%), receive
d=150GB, time=1986050ms, msgs/sec=15105.36, throughput=75.53MB
10.68.8.33:7800:
num_msgs_expected=30000000, num_msgs_received=30000000 (loss rate=0.0%), receive
d=150GB, time=1986069ms, msgs/sec=15105.22, throughput=75.53MB
10.68.16.33:7800:
num_msgs_expected=30000000, num_msgs_received=30000000 (loss rate=0.0%), receive
d=150GB, time=1986148ms, msgs/sec=15104.61, throughput=75.52MB
10.68.4.33:7800:
num_msgs_expected=30000000, num_msgs_received=30000000 (loss rate=0.0%), receive
d=150GB, time=1986112ms, msgs/sec=15104.89, throughput=75.52MB
10.68.0.33:7800:
num_msgs_expected=30000000, num_msgs_received=30000000 (loss rate=0.0%), receive
d=150GB, time=1986093ms, msgs/sec=15105.03, throughput=75.53MB
combined: 15105 msgs/sec averaged over all receivers (throughput=75.52MB/sec)
-- results:
10.68.12.33:7800:
num_msgs_expected=30000000, num_msgs_received=30000000 (loss rate=0.0%), receive
d=150GB, time=1682369ms, msgs/sec=17832, throughput=89.16MB
10.68.20.33:7800:
num_msgs_expected=30000000, num_msgs_received=30000000 (loss rate=0.0%), receive
d=150GB, time=1682386ms, msgs/sec=17831.82, throughput=89.16MB
10.68.8.33:7800:
num_msgs_expected=30000000, num_msgs_received=30000000 (loss rate=0.0%), receive
d=150GB, time=1682349ms, msgs/sec=17832.21, throughput=89.16MB
10.68.16.33:7800:
num_msgs_expected=30000000, num_msgs_received=30000000 (loss rate=0.0%), receive
d=150GB, time=1682134ms, msgs/sec=17834.49, throughput=89.17MB
10.68.0.33:7800:
num_msgs_expected=30000000, num_msgs_received=30000000 (loss rate=0.0%), receive
d=150GB, time=1682365ms, msgs/sec=17832.04, throughput=89.16MB
10.68.4.33:7800 (myself):
num_msgs_expected=30000000, num_msgs_received=30000000 (loss rate=0.0%), receive
d=150GB, time=1682362ms, msgs/sec=17832.07, throughput=89.16MB
combined: 17832.44 msgs/sec averaged over all receivers (throughput=89.16MB/sec)
Problem with retransmission: message not found
----------------------------------------------
Key: JGRP-455
URL: http://jira.jboss.com/jira/browse/JGRP-455
Project: JGroups
Issue Type: Bug
Affects Versions: 2.4.1 SP1
Reporter: Bela Ban
Assigned To: Bela Ban
Priority: Blocker
Fix For: 2.5, 2.4.1 SP4
Attachments: tmp.txt, tmp2.txt, udp-atl.xml
Sometimes a sender cannot find a message with a seqno as requested by a receiver. The
seqno requested (47176) was already purged, sent_msgs: [48059 - 48090] (32)
This could be either an issue in STABLE, which prematurely purges messages although not
all members agreed on the highest seqnos seen, or NAKACK, which might incorrectly
'change' seqnos. Very hard to reproduce !
Trace:
42032 [ERROR] [OOB Thread,perf,10.68.32.33:32968] NAKACK.handleXmitReq(): -
(requester=10.68.28.33:33763, local_addr=10.68.32.33:32968) message
10.68.32.33:32968::47176 not found in sent msgs [48059 - 48090] (32)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:46 a.m.
New subject: [JBoss JIRA] Resolved: (JGRP-455) Problem with retransmission: message not found
[ http://jira.jboss.com/jira/browse/JGRP-455?page=all ]
Bela Ban resolved JGRP-455.
---------------------------
Resolution: Done
Okay, tested some more with heavy loads (e.g. 8 nodes, ever node sending 8M 5K messages
with 500 threads): works fine. There are spurious retransmission requests, but while they
are annoying, they don't lead to incorrect behavior. This (528) issue will be tackled
in 2.6.
Problem with retransmission: message not found
----------------------------------------------
Key: JGRP-455
URL: http://jira.jboss.com/jira/browse/JGRP-455
Project: JGroups
Issue Type: Bug
Affects Versions: 2.4.1 SP1
Reporter: Bela Ban
Assigned To: Bela Ban
Priority: Blocker
Fix For: 2.4.1 SP4, 2.5
Attachments: tmp.txt, tmp2.txt, udp-atl.xml
Sometimes a sender cannot find a message with a seqno as requested by a receiver. The
seqno requested (47176) was already purged, sent_msgs: [48059 - 48090] (32)
This could be either an issue in STABLE, which prematurely purges messages although not
all members agreed on the highest seqnos seen, or NAKACK, which might incorrectly
'change' seqnos. Very hard to reproduce !
Trace:
42032 [ERROR] [OOB Thread,perf,10.68.32.33:32968] NAKACK.handleXmitReq(): -
(requester=10.68.28.33:33763, local_addr=10.68.32.33:32968) message
10.68.32.33:32968::47176 not found in sent msgs [48059 - 48090] (32)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6996
days inactive
7067
days old
18 comments
2 participants
participants (2)
-
Bela Ban (JIRA) -
Galder Zamarreno (JIRA)