7:50 p.m.
Admin Join problem -GMS flush by coordinator failed
----------------------------------------------------
Key: JGRP-985
URL: https://jira.jboss.org/jira/browse/JGRP-985
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7, 2.6.5, 2.6.4, 2.6.3
Environment: linux redhat 2.9.6, jdk 1.5
Reporter: Ronn C
Assignee: Bela Ban
Attachments: jgroup.tar.gz
I am experiencing a problem with jgroups trying to join existing cluster.
Occasionally, new node joining a existing cluster can experience this problem.
2009-05-21 12:04:02,568 [main] WARN org.jgroups.protocols.pbcast.GMS:144 -
join(callisto.tmca.com.au-18715) sent to callisto.tmca.com.au-8185 timed out (after 3000
ms), retrying
Retries can varies from a couple of times to infinitely retrying.
Debugging the code, I've discovered that before join the coordinator will perform a
GMS flush and unless that GMS flush success it won't reply with a join response.
So sure enough at the coordinator, I see this log.
2009-05-21 12:05:25,902 [ViewHandler,callisto.tmca.com.au-8185] WARN
org.jgroups.protocols.pbcast.GMS:749 - GMS flush by coordinator at
callisto.tmca.com.au-8185 failed
I've originally come across this problem in our prod environment with 2.6.3. I have
been able to replicate it reliably with 2.6.3. I have tested with 2.7.0 and 2.8.0.alpha3
and retries still occurs but generally it would sort itself out within a minute. However,
I've found that retries can still occurs infinitely on 2.8.0 if you keep repeating the
test often enough.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:50 p.m.
New subject: [JBoss JIRA] Updated: (JGRP-985) Admin Join problem -GMS flush by coordinator failed
[
https://jira.jboss.org/jira/browse/JGRP-985?page=com.atlassian.jira.plugi...
]
Ronn C updated JGRP-985:
------------------------
Attachment: jgroup.tar.gz
Test case is attached in this Jira. It is a class that instantiate a channel in a thread
three times. What I then do is open three terminal sessions which I run one of this
program for each terminal session. In order to induce the problem, I circle through each
of the terminal session, control-c and restart the program quickly. After a couple of
cycles, the error starts to appear.
Admin Join problem -GMS flush by coordinator failed
----------------------------------------------------
Key: JGRP-985
URL: https://jira.jboss.org/jira/browse/JGRP-985
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3, 2.6.4, 2.6.5, 2.7
Environment: linux redhat 2.9.6, jdk 1.5
Reporter: Ronn C
Assignee: Bela Ban
Attachments: jgroup.tar.gz
I am experiencing a problem with jgroups trying to join existing cluster.
Occasionally, new node joining a existing cluster can experience this problem.
2009-05-21 12:04:02,568 [main] WARN org.jgroups.protocols.pbcast.GMS:144 -
join(callisto.tmca.com.au-18715) sent to callisto.tmca.com.au-8185 timed out (after 3000
ms), retrying
Retries can varies from a couple of times to infinitely retrying.
Debugging the code, I've discovered that before join the coordinator will perform a
GMS flush and unless that GMS flush success it won't reply with a join response.
So sure enough at the coordinator, I see this log.
2009-05-21 12:05:25,902 [ViewHandler,callisto.tmca.com.au-8185] WARN
org.jgroups.protocols.pbcast.GMS:749 - GMS flush by coordinator at
callisto.tmca.com.au-8185 failed
I've originally come across this problem in our prod environment with 2.6.3. I have
been able to replicate it reliably with 2.6.3. I have tested with 2.7.0 and 2.8.0.alpha3
and retries still occurs but generally it would sort itself out within a minute. However,
I've found that retries can still occurs infinitely on 2.8.0 if you keep repeating the
test often enough.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:26 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-985) Admin Join problem -GMS flush by coordinator failed
[
https://jira.jboss.org/jira/browse/JGRP-985?page=com.atlassian.jira.plugi...
]
Bela Ban updated JGRP-985:
--------------------------
Fix Version/s: 2.6.11
2.8
Assignee: Vladimir Blagojevic (was: Bela Ban)
Admin Join problem -GMS flush by coordinator failed
----------------------------------------------------
Key: JGRP-985
URL: https://jira.jboss.org/jira/browse/JGRP-985
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3, 2.6.4, 2.6.5, 2.7
Environment: linux redhat 2.9.6, jdk 1.5
Reporter: Ronn C
Assignee: Vladimir Blagojevic
Fix For: 2.6.11, 2.8
Attachments: jgroup.tar.gz
I am experiencing a problem with jgroups trying to join existing cluster.
Occasionally, new node joining a existing cluster can experience this problem.
2009-05-21 12:04:02,568 [main] WARN org.jgroups.protocols.pbcast.GMS:144 -
join(callisto.tmca.com.au-18715) sent to callisto.tmca.com.au-8185 timed out (after 3000
ms), retrying
Retries can varies from a couple of times to infinitely retrying.
Debugging the code, I've discovered that before join the coordinator will perform a
GMS flush and unless that GMS flush success it won't reply with a join response.
So sure enough at the coordinator, I see this log.
2009-05-21 12:05:25,902 [ViewHandler,callisto.tmca.com.au-8185] WARN
org.jgroups.protocols.pbcast.GMS:749 - GMS flush by coordinator at
callisto.tmca.com.au-8185 failed
I've originally come across this problem in our prod environment with 2.6.3. I have
been able to replicate it reliably with 2.6.3. I have tested with 2.7.0 and 2.8.0.alpha3
and retries still occurs but generally it would sort itself out within a minute. However,
I've found that retries can still occurs infinitely on 2.8.0 if you keep repeating the
test often enough.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:28 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-985) Admin Join problem -GMS flush by coordinator failed
[
https://jira.jboss.org/jira/browse/JGRP-985?page=com.atlassian.jira.plugi...
]
Vladimir Blagojevic commented on JGRP-985:
------------------------------------------
I've looked through this example and I am not sure about the following. First, why did
you comment out UNICAST protocol? In tests like this one use a default udp.xml stack
provided in distribution. Second, why are you getting state if there is no state in
application, you do not implement state installation callbacks?
Admin Join problem -GMS flush by coordinator failed
----------------------------------------------------
Key: JGRP-985
URL: https://jira.jboss.org/jira/browse/JGRP-985
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3, 2.6.4, 2.6.5, 2.7
Environment: linux redhat 2.9.6, jdk 1.5
Reporter: Ronn C
Assignee: Vladimir Blagojevic
Fix For: 2.6.11, 2.8
Attachments: jgroup.tar.gz
I am experiencing a problem with jgroups trying to join existing cluster.
Occasionally, new node joining a existing cluster can experience this problem.
2009-05-21 12:04:02,568 [main] WARN org.jgroups.protocols.pbcast.GMS:144 -
join(callisto.tmca.com.au-18715) sent to callisto.tmca.com.au-8185 timed out (after 3000
ms), retrying
Retries can varies from a couple of times to infinitely retrying.
Debugging the code, I've discovered that before join the coordinator will perform a
GMS flush and unless that GMS flush success it won't reply with a join response.
So sure enough at the coordinator, I see this log.
2009-05-21 12:05:25,902 [ViewHandler,callisto.tmca.com.au-8185] WARN
org.jgroups.protocols.pbcast.GMS:749 - GMS flush by coordinator at
callisto.tmca.com.au-8185 failed
I've originally come across this problem in our prod environment with 2.6.3. I have
been able to replicate it reliably with 2.6.3. I have tested with 2.7.0 and 2.8.0.alpha3
and retries still occurs but generally it would sort itself out within a minute. However,
I've found that retries can still occurs infinitely on 2.8.0 if you keep repeating the
test often enough.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:55 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-985) Admin Join problem -GMS flush by coordinator failed
[
https://jira.jboss.org/jira/browse/JGRP-985?page=com.atlassian.jira.plugi...
]
Vladimir Blagojevic commented on JGRP-985:
------------------------------------------
I am trying to reproduce this issue on my local machine using JGroups2.8.0Alpha3 and I am
unable to do so. I have augmented your test locally not to execute getState and I have
used default flush-udp.xml instead of your jgroups.xml since yours does not include
UNICAST protocol. To be honest, I do see a couple of timeouts and attempted rejoins but
they all eventually succeed. I used three terminals and I cycled through them just as you
described for about 10 minutes.
Admin Join problem -GMS flush by coordinator failed
----------------------------------------------------
Key: JGRP-985
URL: https://jira.jboss.org/jira/browse/JGRP-985
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3, 2.6.4, 2.6.5, 2.7
Environment: linux redhat 2.9.6, jdk 1.5
Reporter: Ronn C
Assignee: Vladimir Blagojevic
Fix For: 2.6.11, 2.8
Attachments: jgroup.tar.gz
I am experiencing a problem with jgroups trying to join existing cluster.
Occasionally, new node joining a existing cluster can experience this problem.
2009-05-21 12:04:02,568 [main] WARN org.jgroups.protocols.pbcast.GMS:144 -
join(callisto.tmca.com.au-18715) sent to callisto.tmca.com.au-8185 timed out (after 3000
ms), retrying
Retries can varies from a couple of times to infinitely retrying.
Debugging the code, I've discovered that before join the coordinator will perform a
GMS flush and unless that GMS flush success it won't reply with a join response.
So sure enough at the coordinator, I see this log.
2009-05-21 12:05:25,902 [ViewHandler,callisto.tmca.com.au-8185] WARN
org.jgroups.protocols.pbcast.GMS:749 - GMS flush by coordinator at
callisto.tmca.com.au-8185 failed
I've originally come across this problem in our prod environment with 2.6.3. I have
been able to replicate it reliably with 2.6.3. I have tested with 2.7.0 and 2.8.0.alpha3
and retries still occurs but generally it would sort itself out within a minute. However,
I've found that retries can still occurs infinitely on 2.8.0 if you keep repeating the
test often enough.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:30 p.m.
New subject: [JBoss JIRA] Commented: (JGRP-985) Admin Join problem -GMS flush by coordinator failed
[
https://jira.jboss.org/jira/browse/JGRP-985?page=com.atlassian.jira.plugi...
]
Ronn C commented on JGRP-985:
-----------------------------
Hi Vladimir,
No reason why UNICAST was commented out. I was trying different settings. My original
problem would have UNICAST entry.
We do implement state in our real application but in the test case I didn't need to.
I agree that it is hard to replicate on 2.8.0alpha3 (compare to 2.6.3) and I did spend a
whole afternoon constantly at it and thought it was fixed. The next morning I turned up
to work tried it a few times and it happened again.
Besides all of that, are you able to confirm that this is an old problem that 2.8.0alpha3
had specifically addressed? and if it has been ported to 2.6.10.merge? We need to deploy
to prod very soon and I'm hesitant to use library that is still in alpha version but
if it is not a known issue and has not been fixed in 2.6.10.merge then I need to know.
Admin Join problem -GMS flush by coordinator failed
----------------------------------------------------
Key: JGRP-985
URL: https://jira.jboss.org/jira/browse/JGRP-985
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3, 2.6.4, 2.6.5, 2.7
Environment: linux redhat 2.9.6, jdk 1.5
Reporter: Ronn C
Assignee: Vladimir Blagojevic
Fix For: 2.6.11, 2.8
Attachments: jgroup.tar.gz
I am experiencing a problem with jgroups trying to join existing cluster.
Occasionally, new node joining a existing cluster can experience this problem.
2009-05-21 12:04:02,568 [main] WARN org.jgroups.protocols.pbcast.GMS:144 -
join(callisto.tmca.com.au-18715) sent to callisto.tmca.com.au-8185 timed out (after 3000
ms), retrying
Retries can varies from a couple of times to infinitely retrying.
Debugging the code, I've discovered that before join the coordinator will perform a
GMS flush and unless that GMS flush success it won't reply with a join response.
So sure enough at the coordinator, I see this log.
2009-05-21 12:05:25,902 [ViewHandler,callisto.tmca.com.au-8185] WARN
org.jgroups.protocols.pbcast.GMS:749 - GMS flush by coordinator at
callisto.tmca.com.au-8185 failed
I've originally come across this problem in our prod environment with 2.6.3. I have
been able to replicate it reliably with 2.6.3. I have tested with 2.7.0 and 2.8.0.alpha3
and retries still occurs but generally it would sort itself out within a minute. However,
I've found that retries can still occurs infinitely on 2.8.0 if you keep repeating the
test often enough.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:33 p.m.
New subject: [JBoss JIRA] Commented: (JGRP-985) Admin Join problem -GMS flush by coordinator failed
[
https://jira.jboss.org/jira/browse/JGRP-985?page=com.atlassian.jira.plugi...
]
Ronn C commented on JGRP-985:
-----------------------------
BTW: I run this test on a linux redhat machine. Have not tried to replicate this on
Windows.
Admin Join problem -GMS flush by coordinator failed
----------------------------------------------------
Key: JGRP-985
URL: https://jira.jboss.org/jira/browse/JGRP-985
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3, 2.6.4, 2.6.5, 2.7
Environment: linux redhat 2.9.6, jdk 1.5
Reporter: Ronn C
Assignee: Vladimir Blagojevic
Fix For: 2.6.11, 2.8
Attachments: jgroup.tar.gz
I am experiencing a problem with jgroups trying to join existing cluster.
Occasionally, new node joining a existing cluster can experience this problem.
2009-05-21 12:04:02,568 [main] WARN org.jgroups.protocols.pbcast.GMS:144 -
join(callisto.tmca.com.au-18715) sent to callisto.tmca.com.au-8185 timed out (after 3000
ms), retrying
Retries can varies from a couple of times to infinitely retrying.
Debugging the code, I've discovered that before join the coordinator will perform a
GMS flush and unless that GMS flush success it won't reply with a join response.
So sure enough at the coordinator, I see this log.
2009-05-21 12:05:25,902 [ViewHandler,callisto.tmca.com.au-8185] WARN
org.jgroups.protocols.pbcast.GMS:749 - GMS flush by coordinator at
callisto.tmca.com.au-8185 failed
I've originally come across this problem in our prod environment with 2.6.3. I have
been able to replicate it reliably with 2.6.3. I have tested with 2.7.0 and 2.8.0.alpha3
and retries still occurs but generally it would sort itself out within a minute. However,
I've found that retries can still occurs infinitely on 2.8.0 if you keep repeating the
test often enough.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:37 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-985) Admin Join problem -GMS flush by coordinator failed
[
https://jira.jboss.org/jira/browse/JGRP-985?page=com.atlassian.jira.plugi...
]
Vladimir Blagojevic commented on JGRP-985:
------------------------------------------
Ronn, I am not sure if this is fixed on 2.6 branch. This test scenario is also a bit
unusual and hard to reproduce - having three nodes fail at the same time. I wonder if
there is a way to trigger a scenario. I believe when you say that this lookup happens and
I think the reason is that the view in coordinator gets corrupted somehow so that flush
can never succeed and thus no new member can ever join.
Admin Join problem -GMS flush by coordinator failed
----------------------------------------------------
Key: JGRP-985
URL: https://jira.jboss.org/jira/browse/JGRP-985
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3, 2.6.4, 2.6.5, 2.7
Environment: linux redhat 2.9.6, jdk 1.5
Reporter: Ronn C
Assignee: Vladimir Blagojevic
Fix For: 2.6.11, 2.8
Attachments: jgroup.tar.gz
I am experiencing a problem with jgroups trying to join existing cluster.
Occasionally, new node joining a existing cluster can experience this problem.
2009-05-21 12:04:02,568 [main] WARN org.jgroups.protocols.pbcast.GMS:144 -
join(callisto.tmca.com.au-18715) sent to callisto.tmca.com.au-8185 timed out (after 3000
ms), retrying
Retries can varies from a couple of times to infinitely retrying.
Debugging the code, I've discovered that before join the coordinator will perform a
GMS flush and unless that GMS flush success it won't reply with a join response.
So sure enough at the coordinator, I see this log.
2009-05-21 12:05:25,902 [ViewHandler,callisto.tmca.com.au-8185] WARN
org.jgroups.protocols.pbcast.GMS:749 - GMS flush by coordinator at
callisto.tmca.com.au-8185 failed
I've originally come across this problem in our prod environment with 2.6.3. I have
been able to replicate it reliably with 2.6.3. I have tested with 2.7.0 and 2.8.0.alpha3
and retries still occurs but generally it would sort itself out within a minute. However,
I've found that retries can still occurs infinitely on 2.8.0 if you keep repeating the
test often enough.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:21 p.m.
New subject: [JBoss JIRA] Commented: (JGRP-985) Admin Join problem -GMS flush by coordinator failed
[
https://jira.jboss.org/jira/browse/JGRP-985?page=com.atlassian.jira.plugi...
]
Ronn C commented on JGRP-985:
-----------------------------
I'm now using 2.6.10.merge. Will be deploying to prod soon but rather nervous about
it.
having three nodes fail at the same time.
Yes I think your guess is right, something in the coordinator must be corrupted and once
failure happens on one node, it will happen to other node that wants to join.
I had tried to debug the problem and I got as far as the coordinator had received JOIN
request and as part of processing the JOIN request it was trying to do a flush but flush
return false as a result it doesn't generate JOIN response and add the member in.
The code that implement flush rather hard to follow and I was hoping that there would be a
document that would explain to me how flush work in algorithmic form so that it would give
me a bit of context to what/where I should be looking out for. Is there such a thing? If
not could you explain it to me and I'll try to debug it again.
Admin Join problem -GMS flush by coordinator failed
----------------------------------------------------
Key: JGRP-985
URL: https://jira.jboss.org/jira/browse/JGRP-985
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3, 2.6.4, 2.6.5, 2.7
Environment: linux redhat 2.9.6, jdk 1.5
Reporter: Ronn C
Assignee: Vladimir Blagojevic
Fix For: 2.6.11, 2.8
Attachments: jgroup.tar.gz
I am experiencing a problem with jgroups trying to join existing cluster.
Occasionally, new node joining a existing cluster can experience this problem.
2009-05-21 12:04:02,568 [main] WARN org.jgroups.protocols.pbcast.GMS:144 -
join(callisto.tmca.com.au-18715) sent to callisto.tmca.com.au-8185 timed out (after 3000
ms), retrying
Retries can varies from a couple of times to infinitely retrying.
Debugging the code, I've discovered that before join the coordinator will perform a
GMS flush and unless that GMS flush success it won't reply with a join response.
So sure enough at the coordinator, I see this log.
2009-05-21 12:05:25,902 [ViewHandler,callisto.tmca.com.au-8185] WARN
org.jgroups.protocols.pbcast.GMS:749 - GMS flush by coordinator at
callisto.tmca.com.au-8185 failed
I've originally come across this problem in our prod environment with 2.6.3. I have
been able to replicate it reliably with 2.6.3. I have tested with 2.7.0 and 2.8.0.alpha3
and retries still occurs but generally it would sort itself out within a minute. However,
I've found that retries can still occurs infinitely on 2.8.0 if you keep repeating the
test often enough.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:08 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-985) Admin Join problem -GMS flush by coordinator failed
[
https://jira.jboss.org/jira/browse/JGRP-985?page=com.atlassian.jira.plugi...
]
Bulat Nigmatullin commented on JGRP-985:
----------------------------------------
I have the same issue with 2.8.0 Beta1. This is the logs of two servers (dont pay
attention to log timestamps, there is a small misconception with server datetimes):
Joining server:
2009.06.26 13:03:17 join(jabber01.rambler.ru-50481) sent to jabber02.rambler.ru-42629
timed out (after 7000 ms), retrying
2009.06.26 13:03:24 join(jabber01.rambler.ru-50481) sent to jabber02.rambler.ru-42629
timed out (after 7000 ms), retrying
2009.06.26 13:03:31 join(jabber01.rambler.ru-50481) sent to jabber02.rambler.ru-42629
timed out (after 7000 ms), retrying
2009.06.26 13:03:38 join(jabber01.rambler.ru-50481) sent to jabber02.rambler.ru-42629
timed out (after 7000 ms), retrying
2009.06.26 13:03:45 join(jabber01.rambler.ru-50481) sent to jabber02.rambler.ru-42629
timed out (after 7000 ms), retrying
2009.06.26 13:03:52 join(jabber01.rambler.ru-50481) sent to jabber02.rambler.ru-42629
timed out (after 7000 ms), retrying
2009.06.26 13:03:59 join(jabber01.rambler.ru-50481) sent to jabber02.rambler.ru-42629
timed out (after 7000 ms), retrying
2009.06.26 13:04:06 join(jabber01.rambler.ru-50481) sent to jabber02.rambler.ru-42629
timed out (after 7000 ms), retrying
2009.06.26 13:04:13 join(jabber01.rambler.ru-50481) sent to jabber02.rambler.ru-42629
timed out (after 7000 ms), retrying
2009.06.26 13:04:20 join(jabber01.rambler.ru-50481) sent to jabber02.rambler.ru-42629
timed out (after 7000 ms), retrying
2009.06.26 13:04:27 join(jabber01.rambler.ru-50481) sent to jabber02.rambler.ru-42629
timed out (after 7000 ms), retrying
Coordinator server:
2009.06.26 13:10:32 jabber02.rambler.ru-42629: GMS flush by coordinator failed
2009.06.26 13:10:47 jabber02.rambler.ru-42629: GMS flush by coordinator failed
2009.06.26 13:11:03 jabber02.rambler.ru-42629: GMS flush by coordinator failed
2009.06.26 13:11:20 jabber02.rambler.ru-42629: GMS flush by coordinator failed
2009.06.26 13:11:35 jabber02.rambler.ru-42629: GMS flush by coordinator failed
2009.06.26 13:11:37 jabber02.rambler.ru-42629: failed to collect all ACKs (expected=5) for
view [jabber02.rambler.ru-42629|38] [jabber02.rambler.ru-42629, jabber15.rambler.ru-14938,
jabber16.rambler.ru-24876, jabber17.rambler.ru-30400, jabber01.rambler.ru-48589,
jabber01.rambler.ru-50481] after 2000ms, missing ACKs from [jabber15.rambler.ru-14938,
jabber01.rambler.ru-48589] (received=[jabber17.rambler.ru-30400,
jabber16.rambler.ru-24876, jabber02.rambler.ru-42629])
2009.06.26 13:12:58 jabber02.rambler.ru-42629: GMS flush by coordinator failed
2009.06.26 13:13:00 jabber02.rambler.ru-42629: failed to collect all ACKs (expected=5) for
view [jabber02.rambler.ru-42629|39] [jabber02.rambler.ru-42629, jabber15.rambler.ru-14938,
jabber16.rambler.ru-24876, jabber17.rambler.ru-30400, jabber01.rambler.ru-50481] after
2000ms, missing ACKs from [jabber15.rambler.ru-14938]
(received=[jabber17.rambler.ru-30400, jabber16.rambler.ru-24876,
jabber02.rambler.ru-42629, jabber01.rambler.ru-50481])
After two minutes from joining server start GMS was OK, but other flushes was broken:
Joining server:
2009.06.26 13:12:08 Flush failed at jabber01.rambler.ru-50481:Sessions by Hostname
2009.06.26 13:12:10 failed to collect all service ACKs (5) for [dst: <null>, src:
jabber01.rambler.ru-50481 (4 headers), size=0 bytes] after 2000ms, missing ACKs from
[jabber15.rambler.ru-14938] (received=[jabber17.rambler.ru-30400,
jabber16.rambler.ru-24876, jabber02.rambler.ru-42629, jabber01.rambler.ru-50481]),
local_addr=jabber01.rambler.ru-50481
2009.06.26 13:12:36 Flush failed at jabber01.rambler.ru-50481:Remote Users Existence
2009.06.26 13:12:38 failed to collect all service ACKs (5) for [dst: <null>, src:
jabber01.rambler.ru-50481 (4 headers), size=0 bytes] after 2000ms, missing ACKs from
[jabber15.rambler.ru-14938] (received=[jabber17.rambler.ru-30400,
jabber16.rambler.ru-24876, jabber02.rambler.ru-42629, jabber01.rambler.ru-50481]),
local_addr=jabber01.rambler.ru-50481
Member server:
2009.06.26 13:06:34 jabber15.rambler.ru-14938: discarded message from non-member
jabber01.rambler.ru-50481, my view is [jabber02.rambler.ru-42629|17]
[jabber02.rambler.ru-42629, jabber15.rambler.ru-14938, jabber16.rambler.ru-24876,
jabber17.rambler.ru-30400]
2009.06.26 13:06:43 jabber15.rambler.ru-14938: discarded message from non-member
jabber01.rambler.ru-50481, my view is [jabber02.rambler.ru-42629|17]
[jabber02.rambler.ru-42629, jabber15.rambler.ru-14938, jabber16.rambler.ru-24876,
jabber17.rambler.ru-30400]
2009.06.26 13:07:01 jabber15.rambler.ru-14938: discarded message from non-member
jabber01.rambler.ru-50481, my view is [jabber02.rambler.ru-42629|17]
[jabber02.rambler.ru-42629, jabber15.rambler.ru-14938, jabber16.rambler.ru-24876,
jabber17.rambler.ru-30400]
2009.06.26 13:07:03 jabber15.rambler.ru-14938: discarded message from non-member
jabber01.rambler.ru-50481, my view is [jabber02.rambler.ru-42629|17]
[jabber02.rambler.ru-42629, jabber15.rambler.ru-14938, jabber16.rambler.ru-24876,
jabber17.rambler.ru-30400]
2009.06.26 13:07:03 jabber15.rambler.ru-14938: discarded message from non-member
jabber01.rambler.ru-50481, my view is [jabber02.rambler.ru-42629|17]
[jabber02.rambler.ru-42629, jabber15.rambler.ru-14938, jabber16.rambler.ru-24876,
jabber17.rambler.ru-30400]
Admin Join problem -GMS flush by coordinator failed
----------------------------------------------------
Key: JGRP-985
URL: https://jira.jboss.org/jira/browse/JGRP-985
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3, 2.6.4, 2.6.5, 2.7
Environment: linux redhat 2.9.6, jdk 1.5
Reporter: Ronn C
Assignee: Vladimir Blagojevic
Fix For: 2.6.11, 2.8
Attachments: jgroup.tar.gz
I am experiencing a problem with jgroups trying to join existing cluster.
Occasionally, new node joining a existing cluster can experience this problem.
2009-05-21 12:04:02,568 [main] WARN org.jgroups.protocols.pbcast.GMS:144 -
join(callisto.tmca.com.au-18715) sent to callisto.tmca.com.au-8185 timed out (after 3000
ms), retrying
Retries can varies from a couple of times to infinitely retrying.
Debugging the code, I've discovered that before join the coordinator will perform a
GMS flush and unless that GMS flush success it won't reply with a join response.
So sure enough at the coordinator, I see this log.
2009-05-21 12:05:25,902 [ViewHandler,callisto.tmca.com.au-8185] WARN
org.jgroups.protocols.pbcast.GMS:749 - GMS flush by coordinator at
callisto.tmca.com.au-8185 failed
I've originally come across this problem in our prod environment with 2.6.3. I have
been able to replicate it reliably with 2.6.3. I have tested with 2.7.0 and 2.8.0.alpha3
and retries still occurs but generally it would sort itself out within a minute. However,
I've found that retries can still occurs infinitely on 2.8.0 if you keep repeating the
test often enough.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:18 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-985) Flush and join problems during constant node failures and constant joins
[
https://jira.jboss.org/jira/browse/JGRP-985?page=com.atlassian.jira.plugi...
]
Vladimir Blagojevic updated JGRP-985:
-------------------------------------
Summary: Flush and join problems during constant node failures and constant joins
(was: Admin Join problem -GMS flush by coordinator failed)
Flush and join problems during constant node failures and constant
joins
------------------------------------------------------------------------
Key: JGRP-985
URL: https://jira.jboss.org/jira/browse/JGRP-985
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3, 2.6.4, 2.6.5, 2.7
Environment: linux redhat 2.9.6, jdk 1.5
Reporter: Ronn C
Assignee: Vladimir Blagojevic
Fix For: 2.6.11, 2.8
Attachments: jgroup.tar.gz
I am experiencing a problem with jgroups trying to join existing cluster.
Occasionally, new node joining a existing cluster can experience this problem.
2009-05-21 12:04:02,568 [main] WARN org.jgroups.protocols.pbcast.GMS:144 -
join(callisto.tmca.com.au-18715) sent to callisto.tmca.com.au-8185 timed out (after 3000
ms), retrying
Retries can varies from a couple of times to infinitely retrying.
Debugging the code, I've discovered that before join the coordinator will perform a
GMS flush and unless that GMS flush success it won't reply with a join response.
So sure enough at the coordinator, I see this log.
2009-05-21 12:05:25,902 [ViewHandler,callisto.tmca.com.au-8185] WARN
org.jgroups.protocols.pbcast.GMS:749 - GMS flush by coordinator at
callisto.tmca.com.au-8185 failed
I've originally come across this problem in our prod environment with 2.6.3. I have
been able to replicate it reliably with 2.6.3. I have tested with 2.7.0 and 2.8.0.alpha3
and retries still occurs but generally it would sort itself out within a minute. However,
I've found that retries can still occurs infinitely on 2.8.0 if you keep repeating the
test often enough.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:10 a.m.
New subject: [JBoss JIRA] Resolved: (JGRP-985) Flush and join problems during constant node failures and constant joins
[
https://jira.jboss.org/jira/browse/JGRP-985?page=com.atlassian.jira.plugi...
]
Vladimir Blagojevic resolved JGRP-985.
--------------------------------------
Resolution: Won't Fix
While we acknowledge that this issue is possible we are confident that it is highly
unlikely to occur in normal range of operations. We have added a unit test
FlushWithChannelJoinsAndFailuresTest in an attempt to capture the scenario you describe
while we have concluded that we can deal with similar cases a bit better in the upcoming
releases. Therefore, we have opened [JGRP-1007] as a continuation of flush issues first
discovered in this case.
Flush and join problems during constant node failures and constant
joins
------------------------------------------------------------------------
Key: JGRP-985
URL: https://jira.jboss.org/jira/browse/JGRP-985
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3, 2.6.4, 2.6.5, 2.7
Environment: linux redhat 2.9.6, jdk 1.5
Reporter: Ronn C
Assignee: Vladimir Blagojevic
Fix For: 2.6.11, 2.8
Attachments: jgroup.tar.gz
I am experiencing a problem with jgroups trying to join existing cluster.
Occasionally, new node joining a existing cluster can experience this problem.
2009-05-21 12:04:02,568 [main] WARN org.jgroups.protocols.pbcast.GMS:144 -
join(callisto.tmca.com.au-18715) sent to callisto.tmca.com.au-8185 timed out (after 3000
ms), retrying
Retries can varies from a couple of times to infinitely retrying.
Debugging the code, I've discovered that before join the coordinator will perform a
GMS flush and unless that GMS flush success it won't reply with a join response.
So sure enough at the coordinator, I see this log.
2009-05-21 12:05:25,902 [ViewHandler,callisto.tmca.com.au-8185] WARN
org.jgroups.protocols.pbcast.GMS:749 - GMS flush by coordinator at
callisto.tmca.com.au-8185 failed
I've originally come across this problem in our prod environment with 2.6.3. I have
been able to replicate it reliably with 2.6.3. I have tested with 2.7.0 and 2.8.0.alpha3
and retries still occurs but generally it would sort itself out within a minute. However,
I've found that retries can still occurs infinitely on 2.8.0 if you keep repeating the
test often enough.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:22 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-985) Flush and join problems during constant node failures and constant joins
[
https://jira.jboss.org/jira/browse/JGRP-985?page=com.atlassian.jira.plugi...
]
Vladimir Blagojevic updated JGRP-985:
-------------------------------------
Comment: was deleted
Flush and join problems during constant node failures and constant
joins
------------------------------------------------------------------------
Key: JGRP-985
URL: https://jira.jboss.org/jira/browse/JGRP-985
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3, 2.6.4, 2.6.5, 2.7
Environment: linux redhat 2.9.6, jdk 1.5
Reporter: Ronn C
Assignee: Vladimir Blagojevic
Fix For: 2.6.11, 2.8
Attachments: jgroup.tar.gz
I am experiencing a problem with jgroups trying to join existing cluster.
Occasionally, new node joining a existing cluster can experience this problem.
2009-05-21 12:04:02,568 [main] WARN org.jgroups.protocols.pbcast.GMS:144 -
join(callisto.tmca.com.au-18715) sent to callisto.tmca.com.au-8185 timed out (after 3000
ms), retrying
Retries can varies from a couple of times to infinitely retrying.
Debugging the code, I've discovered that before join the coordinator will perform a
GMS flush and unless that GMS flush success it won't reply with a join response.
So sure enough at the coordinator, I see this log.
2009-05-21 12:05:25,902 [ViewHandler,callisto.tmca.com.au-8185] WARN
org.jgroups.protocols.pbcast.GMS:749 - GMS flush by coordinator at
callisto.tmca.com.au-8185 failed
I've originally come across this problem in our prod environment with 2.6.3. I have
been able to replicate it reliably with 2.6.3. I have tested with 2.7.0 and 2.8.0.alpha3
and retries still occurs but generally it would sort itself out within a minute. However,
I've found that retries can still occurs infinitely on 2.8.0 if you keep repeating the
test often enough.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:22 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-985) Flush and join problems during constant node failures and constant joins
[
https://jira.jboss.org/jira/browse/JGRP-985?page=com.atlassian.jira.plugi...
]
Vladimir Blagojevic commented on JGRP-985:
------------------------------------------
While we acknowledge that this issue is possible we are confident that it is highly
unlikely to occur in normal range of operations. We have added a unit test
FlushWithChannelJoinsAndFailuresTest in an attempt to capture the scenario you describe.
We have concluded that we can deal with similar cases a bit better in the upcoming
releases. Therefore, we have opened [JGRP-1007] as a continuation of flush issues first
discovered in this case.
Flush and join problems during constant node failures and constant
joins
------------------------------------------------------------------------
Key: JGRP-985
URL: https://jira.jboss.org/jira/browse/JGRP-985
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3, 2.6.4, 2.6.5, 2.7
Environment: linux redhat 2.9.6, jdk 1.5
Reporter: Ronn C
Assignee: Vladimir Blagojevic
Fix For: 2.6.11, 2.8
Attachments: jgroup.tar.gz
I am experiencing a problem with jgroups trying to join existing cluster.
Occasionally, new node joining a existing cluster can experience this problem.
2009-05-21 12:04:02,568 [main] WARN org.jgroups.protocols.pbcast.GMS:144 -
join(callisto.tmca.com.au-18715) sent to callisto.tmca.com.au-8185 timed out (after 3000
ms), retrying
Retries can varies from a couple of times to infinitely retrying.
Debugging the code, I've discovered that before join the coordinator will perform a
GMS flush and unless that GMS flush success it won't reply with a join response.
So sure enough at the coordinator, I see this log.
2009-05-21 12:05:25,902 [ViewHandler,callisto.tmca.com.au-8185] WARN
org.jgroups.protocols.pbcast.GMS:749 - GMS flush by coordinator at
callisto.tmca.com.au-8185 failed
I've originally come across this problem in our prod environment with 2.6.3. I have
been able to replicate it reliably with 2.6.3. I have tested with 2.7.0 and 2.8.0.alpha3
and retries still occurs but generally it would sort itself out within a minute. However,
I've found that retries can still occurs infinitely on 2.8.0 if you keep repeating the
test often enough.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:54 p.m.
New subject: [JBoss JIRA] Commented: (JGRP-985) Flush and join problems during constant node failures and constant joins
[
https://jira.jboss.org/jira/browse/JGRP-985?page=com.atlassian.jira.plugi...
]
Yong Zhang commented on JGRP-985:
---------------------------------
We keep getting this error recently in production environment:
* JBoss 5.01.GA + jgroup 2.6.7.GA
Joining server:
2009-10-15 13:26:16,061 [main] WARN jgroups.protocols.pbcast.GMS : join(prod01:7700)
sent to prod02:7700 timed out (after 3000 ms), retrying
2009-10-15 13:26:19,077 [main] WARN jgroups.protocols.pbcast.GMS : join(prod01:7700)
sent to prod02:7700 timed out (after 3000 ms), retrying
2009-10-15 13:26:22,093 [main] WARN jgroups.protocols.pbcast.GMS : join(prod01:7700)
sent to prod02:7700 timed out (after 3000 ms), retrying
2009-10-15 13:26:25,108 [main] WARN jgroups.protocols.pbcast.GMS : join(prod01:7700)
sent to prod02:7700 timed out (after 3000 ms), retrying
Member server:
2009-10-15 13:24:30,873 [ViewHandler,CCPRODR-SFSBCache,prod02:7700] WARN
jgroups.protocols.pbcast.GMS : GMS flush by coordinator at prod02:7700 failed
2009-10-15 13:24:56,200 [ViewHandler,CCPRODR-SFSBCache,prod02:7700] WARN
jgroups.protocols.pbcast.GMS : GMS flush by coordinator at prod02:7700 failed
The only work-around is to restart the whole cluster which causes a production outage to
business, or leave the failed server down.
Is there any chance that this issue can be fixed in a jgroup 2.6.x patch? or any
suggestions that we can use as a better workaround? Thanks.
Flush and join problems during constant node failures and constant
joins
------------------------------------------------------------------------
Key: JGRP-985
URL: https://jira.jboss.org/jira/browse/JGRP-985
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3, 2.6.4, 2.6.5, 2.7
Environment: linux redhat 2.9.6, jdk 1.5
Reporter: Ronn C
Assignee: Vladimir Blagojevic
Fix For: 2.6.11, 2.8
Attachments: jgroup.tar.gz
I am experiencing a problem with jgroups trying to join existing cluster.
Occasionally, new node joining a existing cluster can experience this problem.
2009-05-21 12:04:02,568 [main] WARN org.jgroups.protocols.pbcast.GMS:144 -
join(callisto.tmca.com.au-18715) sent to callisto.tmca.com.au-8185 timed out (after 3000
ms), retrying
Retries can varies from a couple of times to infinitely retrying.
Debugging the code, I've discovered that before join the coordinator will perform a
GMS flush and unless that GMS flush success it won't reply with a join response.
So sure enough at the coordinator, I see this log.
2009-05-21 12:05:25,902 [ViewHandler,callisto.tmca.com.au-8185] WARN
org.jgroups.protocols.pbcast.GMS:749 - GMS flush by coordinator at
callisto.tmca.com.au-8185 failed
I've originally come across this problem in our prod environment with 2.6.3. I have
been able to replicate it reliably with 2.6.3. I have tested with 2.7.0 and 2.8.0.alpha3
and retries still occurs but generally it would sort itself out within a minute. However,
I've found that retries can still occurs infinitely on 2.8.0 if you keep repeating the
test often enough.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6126
days inactive
6269
days old
14 comments
5 participants
participants (5)
-
Bela Ban (JIRA) -
Bulat Nigmatullin (JIRA)
-
Ronn C (JIRA)
-
Vladimir Blagojevic (JIRA)
-
Yong Zhang (JIRA)