1:13 p.m.
JGroups threads get stuck and stop communicating
------------------------------------------------
Key: JGRP-1157
URL: https://jira.jboss.org/jira/browse/JGRP-1157
Project: JGroups
Issue Type: Bug
Affects Versions: 2.9, 2.8
Environment: Linux
Reporter: vivek v
Assignee: Bela Ban
We are having problem where a node gets isolated after some intermittent network outage
and is never able to join back. Bela suspected some issue /w RouterStub and fixed a bug -
JGRP-1151. But, we are were able to reproduce this problem /w even JGroups 2.9 GA. Looks
like the problem is that the node that gets isolated becomes unresponsive as all its
JGroups threads hang. Here is how we reproduced the error /w 3 nodes (Node A -
coordinator also running
Gossip Router, Node B, Node C),
1) We added WANem between A and B - so there are random disconnects,high packet loss and
200 msec of delay
2) Due to our WANem setting B loses connectivity /w GR - in and out
3) We restart A and it becomes isolated. A becomes singleton and never joins back the
group. We see NAKACK on the node C - as A is still able to get to C, but not B. C keeps
dropping messages from A as A is not in its transmission table.
4) We turned on tracing on A, but after a while (couple of hours) we see no JGroups trace
on A - we suspected that some of the JGroups threads might have got stuck. So we took the
thread dump of the java process on A (attached). As you can see there are quite a few
JGroups threads in the waiting state and all are for TCP.send
We are not clear on how or why will the JGroups threads hang. Could outgoing messages be
queued up and not moving for some reason?
The only solution to fix this was to restart all the nodes, which is not desirable.
Attached are the stack trace (thread-dump) and our protocol stack.
This jira was originated from discussion at
http://sourceforge.net/mailarchive/forum.php?thread_name=4B7BC107.9060304...
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:25 p.m.
New subject: [JBoss JIRA] Updated: (JGRP-1157) JGroups threads get stuck and stop communicating
[
https://jira.jboss.org/jira/browse/JGRP-1157?page=com.atlassian.jira.plug...
]
vivek v updated JGRP-1157:
--------------------------
Attachment: threaddump-jg.log
jgroups_stack.txt
Adding attachments
JGroups threads get stuck and stop communicating
------------------------------------------------
Key: JGRP-1157
URL: https://jira.jboss.org/jira/browse/JGRP-1157
Project: JGroups
Issue Type: Bug
Affects Versions: 2.8, 2.9
Environment: Linux
Reporter: vivek v
Assignee: Bela Ban
Attachments: jgroups_stack.txt, threaddump-jg.log
We are having problem where a node gets isolated after some intermittent network outage
and is never able to join back. Bela suspected some issue /w RouterStub and fixed a bug -
JGRP-1151. But, we are were able to reproduce this problem /w even JGroups 2.9 GA. Looks
like the problem is that the node that gets isolated becomes unresponsive as all its
JGroups threads hang. Here is how we reproduced the error /w 3 nodes (Node A -
coordinator also running
Gossip Router, Node B, Node C),
1) We added WANem between A and B - so there are random disconnects,high packet loss and
200 msec of delay
2) Due to our WANem setting B loses connectivity /w GR - in and out
3) We restart A and it becomes isolated. A becomes singleton and never joins back the
group. We see NAKACK on the node C - as A is still able to get to C, but not B. C keeps
dropping messages from A as A is not in its transmission table.
4) We turned on tracing on A, but after a while (couple of hours) we see no JGroups trace
on A - we suspected that some of the JGroups threads might have got stuck. So we took the
thread dump of the java process on A (attached). As you can see there are quite a few
JGroups threads in the waiting state and all are for TCP.send
We are not clear on how or why will the JGroups threads hang. Could outgoing messages be
queued up and not moving for some reason?
The only solution to fix this was to restart all the nodes, which is not desirable.
Attached are the stack trace (thread-dump) and our protocol stack.
This jira was originated from discussion at
http://sourceforge.net/mailarchive/forum.php?thread_name=4B7BC107.9060304...
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:22 p.m.
New subject: [JBoss JIRA] Updated: (JGRP-1157) JGroups threads get stuck and stop communicating
[
https://jira.jboss.org/jira/browse/JGRP-1157?page=com.atlassian.jira.plug...
]
Bela Ban updated JGRP-1157:
---------------------------
Fix Version/s: 2.10
JGroups threads get stuck and stop communicating
------------------------------------------------
Key: JGRP-1157
URL: https://jira.jboss.org/jira/browse/JGRP-1157
Project: JGroups
Issue Type: Bug
Affects Versions: 2.8, 2.9
Environment: Linux
Reporter: vivek v
Assignee: Bela Ban
Fix For: 2.10
Attachments: jgroups_stack.txt, threaddump-jg.log
We are having problem where a node gets isolated after some intermittent network outage
and is never able to join back. Bela suspected some issue /w RouterStub and fixed a bug -
JGRP-1151. But, we are were able to reproduce this problem /w even JGroups 2.9 GA. Looks
like the problem is that the node that gets isolated becomes unresponsive as all its
JGroups threads hang. Here is how we reproduced the error /w 3 nodes (Node A -
coordinator also running
Gossip Router, Node B, Node C),
1) We added WANem between A and B - so there are random disconnects,high packet loss and
200 msec of delay
2) Due to our WANem setting B loses connectivity /w GR - in and out
3) We restart A and it becomes isolated. A becomes singleton and never joins back the
group. We see NAKACK on the node C - as A is still able to get to C, but not B. C keeps
dropping messages from A as A is not in its transmission table.
4) We turned on tracing on A, but after a while (couple of hours) we see no JGroups trace
on A - we suspected that some of the JGroups threads might have got stuck. So we took the
thread dump of the java process on A (attached). As you can see there are quite a few
JGroups threads in the waiting state and all are for TCP.send
We are not clear on how or why will the JGroups threads hang. Could outgoing messages be
queued up and not moving for some reason?
The only solution to fix this was to restart all the nodes, which is not desirable.
Attached are the stack trace (thread-dump) and our protocol stack.
This jira was originated from discussion at
http://sourceforge.net/mailarchive/forum.php?thread_name=4B7BC107.9060304...
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:36 p.m.
New subject: [JBoss JIRA] Commented: (JGRP-1157) JGroups threads get stuck and stop communicating
[
https://jira.jboss.org/jira/browse/JGRP-1157?page=com.atlassian.jira.plug...
]
vivek v commented on JGRP-1157:
-------------------------------
We debugged this issue little more (by attaching the debugger to one of the instance,
which is having problem). Here is what I found by stepping into some of the stuck
threads,
1) There were two sender threads (Node A -> B and Node A -> C). Both these sender
threads were waiting on queue.take() , basically there was nothing to be read from the
sender_queue, i.e, no messages.
2) There were lots of other threads, which were waiting on queue.put(). After tracing
through those threads I found the TCPConnection object for this queue was different than
the sender threads. This TCPConnection was for A-> B, but a different object
altogether
3) I also found that there were no sender thread for this new TCPConnection object (# 2).
So, basically all the messages were getting written to the queue, but there was no Sender
thread for it to read it.
In theory there should only be one TCPConnection (and one sender thread) for each
node-to-node connection. In this case, we got two TCPConnection objects for the same
node-to-node connection. I'm not sure how it is possible. Looks like one TCPConnection
was created, sender thread started, but later due to some socket connect problem it was
removed from the TCPConnectionMap, but the sender thread remained from that connection.
Now later a new TCPConnection was started and added to the TCPConnectionMap, but for some
reason its sender thread never got started - is it possible since, they both have the same
thread-name they might fail (though unlikely).
I'm attaching the snapshots to prove that we got two TCPConnections for the same
peer_addr.
Not sure if this is a race-condition, but this definitely seems like a problem.
Is there a work-around for this problem? Like not using sender or something? What are the
implications of not using sender (use_send_queues="false")?
JGroups threads get stuck and stop communicating
------------------------------------------------
Key: JGRP-1157
URL: https://jira.jboss.org/jira/browse/JGRP-1157
Project: JGroups
Issue Type: Bug
Affects Versions: 2.8, 2.9
Environment: Linux
Reporter: vivek v
Assignee: Bela Ban
Fix For: 2.10
Attachments: jgroups_stack.txt, threaddump-jg.log
We are having problem where a node gets isolated after some intermittent network outage
and is never able to join back. Bela suspected some issue /w RouterStub and fixed a bug -
JGRP-1151. But, we are were able to reproduce this problem /w even JGroups 2.9 GA. Looks
like the problem is that the node that gets isolated becomes unresponsive as all its
JGroups threads hang. Here is how we reproduced the error /w 3 nodes (Node A -
coordinator also running
Gossip Router, Node B, Node C),
1) We added WANem between A and B - so there are random disconnects,high packet loss and
200 msec of delay
2) Due to our WANem setting B loses connectivity /w GR - in and out
3) We restart A and it becomes isolated. A becomes singleton and never joins back the
group. We see NAKACK on the node C - as A is still able to get to C, but not B. C keeps
dropping messages from A as A is not in its transmission table.
4) We turned on tracing on A, but after a while (couple of hours) we see no JGroups trace
on A - we suspected that some of the JGroups threads might have got stuck. So we took the
thread dump of the java process on A (attached). As you can see there are quite a few
JGroups threads in the waiting state and all are for TCP.send
We are not clear on how or why will the JGroups threads hang. Could outgoing messages be
queued up and not moving for some reason?
The only solution to fix this was to restart all the nodes, which is not desirable.
Attached are the stack trace (thread-dump) and our protocol stack.
This jira was originated from discussion at
http://sourceforge.net/mailarchive/forum.php?thread_name=4B7BC107.9060304...
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:47 p.m.
New subject: [JBoss JIRA] Updated: (JGRP-1157) JGroups threads get stuck and stop communicating
[
https://jira.jboss.org/jira/browse/JGRP-1157?page=com.atlassian.jira.plug...
]
vivek v updated JGRP-1157:
--------------------------
Attachment: TCPConnection-1.png
TCPConnection-2.png
Adding snapshots for two TCPConnection connected to the same ip address.
JGroups threads get stuck and stop communicating
------------------------------------------------
Key: JGRP-1157
URL: https://jira.jboss.org/jira/browse/JGRP-1157
Project: JGroups
Issue Type: Bug
Affects Versions: 2.8, 2.9
Environment: Linux
Reporter: vivek v
Assignee: Bela Ban
Fix For: 2.10
Attachments: jgroups_stack.txt, TCPConnection-1.png, TCPConnection-2.png,
threaddump-jg.log
We are having problem where a node gets isolated after some intermittent network outage
and is never able to join back. Bela suspected some issue /w RouterStub and fixed a bug -
JGRP-1151. But, we are were able to reproduce this problem /w even JGroups 2.9 GA. Looks
like the problem is that the node that gets isolated becomes unresponsive as all its
JGroups threads hang. Here is how we reproduced the error /w 3 nodes (Node A -
coordinator also running
Gossip Router, Node B, Node C),
1) We added WANem between A and B - so there are random disconnects,high packet loss and
200 msec of delay
2) Due to our WANem setting B loses connectivity /w GR - in and out
3) We restart A and it becomes isolated. A becomes singleton and never joins back the
group. We see NAKACK on the node C - as A is still able to get to C, but not B. C keeps
dropping messages from A as A is not in its transmission table.
4) We turned on tracing on A, but after a while (couple of hours) we see no JGroups trace
on A - we suspected that some of the JGroups threads might have got stuck. So we took the
thread dump of the java process on A (attached). As you can see there are quite a few
JGroups threads in the waiting state and all are for TCP.send
We are not clear on how or why will the JGroups threads hang. Could outgoing messages be
queued up and not moving for some reason?
The only solution to fix this was to restart all the nodes, which is not desirable.
Attached are the stack trace (thread-dump) and our protocol stack.
This jira was originated from discussion at
http://sourceforge.net/mailarchive/forum.php?thread_name=4B7BC107.9060304...
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:09 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-1157) GossipRouter: JGroups threads get stuck and stop communicating
[
https://jira.jboss.org/jira/browse/JGRP-1157?page=com.atlassian.jira.plug...
]
Bela Ban updated JGRP-1157:
---------------------------
Summary: GossipRouter: JGroups threads get stuck and stop communicating (was: JGroups
threads get stuck and stop communicating)
GossipRouter: JGroups threads get stuck and stop communicating
--------------------------------------------------------------
Key: JGRP-1157
URL: https://jira.jboss.org/jira/browse/JGRP-1157
Project: JGroups
Issue Type: Bug
Affects Versions: 2.8, 2.9
Environment: Linux
Reporter: vivek v
Assignee: Bela Ban
Fix For: 2.10
Attachments: jgroups_stack.txt, TCPConnection-1.png, TCPConnection-2.png,
threaddump-jg.log
We are having problem where a node gets isolated after some intermittent network outage
and is never able to join back. Bela suspected some issue /w RouterStub and fixed a bug -
JGRP-1151. But, we are were able to reproduce this problem /w even JGroups 2.9 GA. Looks
like the problem is that the node that gets isolated becomes unresponsive as all its
JGroups threads hang. Here is how we reproduced the error /w 3 nodes (Node A -
coordinator also running
Gossip Router, Node B, Node C),
1) We added WANem between A and B - so there are random disconnects,high packet loss and
200 msec of delay
2) Due to our WANem setting B loses connectivity /w GR - in and out
3) We restart A and it becomes isolated. A becomes singleton and never joins back the
group. We see NAKACK on the node C - as A is still able to get to C, but not B. C keeps
dropping messages from A as A is not in its transmission table.
4) We turned on tracing on A, but after a while (couple of hours) we see no JGroups trace
on A - we suspected that some of the JGroups threads might have got stuck. So we took the
thread dump of the java process on A (attached). As you can see there are quite a few
JGroups threads in the waiting state and all are for TCP.send
We are not clear on how or why will the JGroups threads hang. Could outgoing messages be
queued up and not moving for some reason?
The only solution to fix this was to restart all the nodes, which is not desirable.
Attached are the stack trace (thread-dump) and our protocol stack.
This jira was originated from discussion at
http://sourceforge.net/mailarchive/forum.php?thread_name=4B7BC107.9060304...
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:49 a.m.
New subject: [JBoss JIRA] Assigned: (JGRP-1157) GossipRouter: JGroups threads get stuck and stop communicating
[
https://jira.jboss.org/jira/browse/JGRP-1157?page=com.atlassian.jira.plug...
]
Bela Ban reassigned JGRP-1157:
------------------------------
Assignee: Vladimir Blagojevic (was: Bela Ban)
Vladimir, I think this might be a moot issue once you resolve the other GossipRouter
related issues...
GossipRouter: JGroups threads get stuck and stop communicating
--------------------------------------------------------------
Key: JGRP-1157
URL: https://jira.jboss.org/jira/browse/JGRP-1157
Project: JGroups
Issue Type: Bug
Affects Versions: 2.8, 2.9
Environment: Linux
Reporter: vivek v
Assignee: Vladimir Blagojevic
Fix For: 2.10
Attachments: jgroups_stack.txt, TCPConnection-1.png, TCPConnection-2.png,
threaddump-jg.log
We are having problem where a node gets isolated after some intermittent network outage
and is never able to join back. Bela suspected some issue /w RouterStub and fixed a bug -
JGRP-1151. But, we are were able to reproduce this problem /w even JGroups 2.9 GA. Looks
like the problem is that the node that gets isolated becomes unresponsive as all its
JGroups threads hang. Here is how we reproduced the error /w 3 nodes (Node A -
coordinator also running
Gossip Router, Node B, Node C),
1) We added WANem between A and B - so there are random disconnects,high packet loss and
200 msec of delay
2) Due to our WANem setting B loses connectivity /w GR - in and out
3) We restart A and it becomes isolated. A becomes singleton and never joins back the
group. We see NAKACK on the node C - as A is still able to get to C, but not B. C keeps
dropping messages from A as A is not in its transmission table.
4) We turned on tracing on A, but after a while (couple of hours) we see no JGroups trace
on A - we suspected that some of the JGroups threads might have got stuck. So we took the
thread dump of the java process on A (attached). As you can see there are quite a few
JGroups threads in the waiting state and all are for TCP.send
We are not clear on how or why will the JGroups threads hang. Could outgoing messages be
queued up and not moving for some reason?
The only solution to fix this was to restart all the nodes, which is not desirable.
Attached are the stack trace (thread-dump) and our protocol stack.
This jira was originated from discussion at
http://sourceforge.net/mailarchive/forum.php?thread_name=4B7BC107.9060304...
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:10 p.m.
New subject: [JBoss JIRA] Commented: (JGRP-1157) GossipRouter: JGroups threads get stuck and stop communicating
[
https://jira.jboss.org/jira/browse/JGRP-1157?page=com.atlassian.jira.plug...
]
vivek v commented on JGRP-1157:
-------------------------------
Note, this issue is not related to Gossip Router, but TCP. So, the fix for GR may not
apply here. We still need to figure out where are multiple TCPConnections coming from for
the same destination. It sounds similar to GR issues where RouterStub wasn't closing
the connections properly.
GossipRouter: JGroups threads get stuck and stop communicating
--------------------------------------------------------------
Key: JGRP-1157
URL: https://jira.jboss.org/jira/browse/JGRP-1157
Project: JGroups
Issue Type: Bug
Affects Versions: 2.8, 2.9
Environment: Linux
Reporter: vivek v
Assignee: Vladimir Blagojevic
Fix For: 2.10
Attachments: jgroups_stack.txt, TCPConnection-1.png, TCPConnection-2.png,
threaddump-jg.log
We are having problem where a node gets isolated after some intermittent network outage
and is never able to join back. Bela suspected some issue /w RouterStub and fixed a bug -
JGRP-1151. But, we are were able to reproduce this problem /w even JGroups 2.9 GA. Looks
like the problem is that the node that gets isolated becomes unresponsive as all its
JGroups threads hang. Here is how we reproduced the error /w 3 nodes (Node A -
coordinator also running
Gossip Router, Node B, Node C),
1) We added WANem between A and B - so there are random disconnects,high packet loss and
200 msec of delay
2) Due to our WANem setting B loses connectivity /w GR - in and out
3) We restart A and it becomes isolated. A becomes singleton and never joins back the
group. We see NAKACK on the node C - as A is still able to get to C, but not B. C keeps
dropping messages from A as A is not in its transmission table.
4) We turned on tracing on A, but after a while (couple of hours) we see no JGroups trace
on A - we suspected that some of the JGroups threads might have got stuck. So we took the
thread dump of the java process on A (attached). As you can see there are quite a few
JGroups threads in the waiting state and all are for TCP.send
We are not clear on how or why will the JGroups threads hang. Could outgoing messages be
queued up and not moving for some reason?
The only solution to fix this was to restart all the nodes, which is not desirable.
Attached are the stack trace (thread-dump) and our protocol stack.
This jira was originated from discussion at
http://sourceforge.net/mailarchive/forum.php?thread_name=4B7BC107.9060304...
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:32 p.m.
New subject: [JBoss JIRA] Commented: (JGRP-1157) GossipRouter: JGroups threads get stuck and stop communicating
[
https://jira.jboss.org/jira/browse/JGRP-1157?page=com.atlassian.jira.plug...
]
Pete Matern commented on JGRP-1157:
-----------------------------------
I have run into a startup hang with similar stacks when spinning up three jgroups nodes at
essentially the same time. I can hit it roughly 1 out of 15 times on my box. I'm using
TCPPING, so certainly not a GR issue for me. Logs, dumps and stack configuration available
if you'd like it.
GossipRouter: JGroups threads get stuck and stop communicating
--------------------------------------------------------------
Key: JGRP-1157
URL: https://jira.jboss.org/jira/browse/JGRP-1157
Project: JGroups
Issue Type: Bug
Affects Versions: 2.8, 2.9
Environment: Linux
Reporter: vivek v
Assignee: Vladimir Blagojevic
Fix For: 2.10
Attachments: jgroups_stack.txt, TCPConnection-1.png, TCPConnection-2.png,
threaddump-jg.log
We are having problem where a node gets isolated after some intermittent network outage
and is never able to join back. Bela suspected some issue /w RouterStub and fixed a bug -
JGRP-1151. But, we are were able to reproduce this problem /w even JGroups 2.9 GA. Looks
like the problem is that the node that gets isolated becomes unresponsive as all its
JGroups threads hang. Here is how we reproduced the error /w 3 nodes (Node A -
coordinator also running
Gossip Router, Node B, Node C),
1) We added WANem between A and B - so there are random disconnects,high packet loss and
200 msec of delay
2) Due to our WANem setting B loses connectivity /w GR - in and out
3) We restart A and it becomes isolated. A becomes singleton and never joins back the
group. We see NAKACK on the node C - as A is still able to get to C, but not B. C keeps
dropping messages from A as A is not in its transmission table.
4) We turned on tracing on A, but after a while (couple of hours) we see no JGroups trace
on A - we suspected that some of the JGroups threads might have got stuck. So we took the
thread dump of the java process on A (attached). As you can see there are quite a few
JGroups threads in the waiting state and all are for TCP.send
We are not clear on how or why will the JGroups threads hang. Could outgoing messages be
queued up and not moving for some reason?
The only solution to fix this was to restart all the nodes, which is not desirable.
Attached are the stack trace (thread-dump) and our protocol stack.
This jira was originated from discussion at
http://sourceforge.net/mailarchive/forum.php?thread_name=4B7BC107.9060304...
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:21 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-1157) TCP: JGroups threads get stuck and stop communicating
[
https://jira.jboss.org/jira/browse/JGRP-1157?page=com.atlassian.jira.plug...
]
Vladimir Blagojevic updated JGRP-1157:
--------------------------------------
Summary: TCP: JGroups threads get stuck and stop communicating (was: GossipRouter:
JGroups threads get stuck and stop communicating)
TCP: JGroups threads get stuck and stop communicating
-----------------------------------------------------
Key: JGRP-1157
URL: https://jira.jboss.org/jira/browse/JGRP-1157
Project: JGroups
Issue Type: Bug
Affects Versions: 2.8, 2.9
Environment: Linux
Reporter: vivek v
Assignee: Vladimir Blagojevic
Fix For: 2.10
Attachments: jgroups_stack.txt, TCPConnection-1.png, TCPConnection-2.png,
threaddump-jg.log
We are having problem where a node gets isolated after some intermittent network outage
and is never able to join back. Bela suspected some issue /w RouterStub and fixed a bug -
JGRP-1151. But, we are were able to reproduce this problem /w even JGroups 2.9 GA. Looks
like the problem is that the node that gets isolated becomes unresponsive as all its
JGroups threads hang. Here is how we reproduced the error /w 3 nodes (Node A -
coordinator also running
Gossip Router, Node B, Node C),
1) We added WANem between A and B - so there are random disconnects,high packet loss and
200 msec of delay
2) Due to our WANem setting B loses connectivity /w GR - in and out
3) We restart A and it becomes isolated. A becomes singleton and never joins back the
group. We see NAKACK on the node C - as A is still able to get to C, but not B. C keeps
dropping messages from A as A is not in its transmission table.
4) We turned on tracing on A, but after a while (couple of hours) we see no JGroups trace
on A - we suspected that some of the JGroups threads might have got stuck. So we took the
thread dump of the java process on A (attached). As you can see there are quite a few
JGroups threads in the waiting state and all are for TCP.send
We are not clear on how or why will the JGroups threads hang. Could outgoing messages be
queued up and not moving for some reason?
The only solution to fix this was to restart all the nodes, which is not desirable.
Attached are the stack trace (thread-dump) and our protocol stack.
This jira was originated from discussion at
http://sourceforge.net/mailarchive/forum.php?thread_name=4B7BC107.9060304...
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:48 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1157) TCP: JGroups threads get stuck and stop communicating
[
https://jira.jboss.org/browse/JGRP-1157?page=com.atlassian.jira.plugin.sy...
]
Bela Ban commented on JGRP-1157:
--------------------------------
Vladimir made some changes to TCPConnectionMap, which ensures that receiver and sender
threads always terminate together; ie. it's not possible any longer for a sender
thread to terminate, but for the receiver thread to continue. The change was to close the
connection on sender thread termination (this was already in place for the receiver
thread).
When the connection is closed, the peer will need to re-establish it when trying to send
data.
TCP: JGroups threads get stuck and stop communicating
-----------------------------------------------------
Key: JGRP-1157
URL: https://jira.jboss.org/browse/JGRP-1157
Project: JGroups
Issue Type: Bug
Affects Versions: 2.8, 2.9
Environment: Linux
Reporter: vivek v
Assignee: Vladimir Blagojevic
Fix For: 2.10
Attachments: jgroups_stack.txt, TCPConnection-1.png, TCPConnection-2.png,
threaddump-jg.log
We are having problem where a node gets isolated after some intermittent network outage
and is never able to join back. Bela suspected some issue /w RouterStub and fixed a bug -
JGRP-1151. But, we are were able to reproduce this problem /w even JGroups 2.9 GA. Looks
like the problem is that the node that gets isolated becomes unresponsive as all its
JGroups threads hang. Here is how we reproduced the error /w 3 nodes (Node A -
coordinator also running
Gossip Router, Node B, Node C),
1) We added WANem between A and B - so there are random disconnects,high packet loss and
200 msec of delay
2) Due to our WANem setting B loses connectivity /w GR - in and out
3) We restart A and it becomes isolated. A becomes singleton and never joins back the
group. We see NAKACK on the node C - as A is still able to get to C, but not B. C keeps
dropping messages from A as A is not in its transmission table.
4) We turned on tracing on A, but after a while (couple of hours) we see no JGroups trace
on A - we suspected that some of the JGroups threads might have got stuck. So we took the
thread dump of the java process on A (attached). As you can see there are quite a few
JGroups threads in the waiting state and all are for TCP.send
We are not clear on how or why will the JGroups threads hang. Could outgoing messages be
queued up and not moving for some reason?
The only solution to fix this was to restart all the nodes, which is not desirable.
Attached are the stack trace (thread-dump) and our protocol stack.
This jira was originated from discussion at
http://sourceforge.net/mailarchive/forum.php?thread_name=4B7BC107.9060304...
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:50 a.m.
New subject: [JBoss JIRA] Resolved: (JGRP-1157) TCP: JGroups threads get stuck and stop communicating
[
https://jira.jboss.org/browse/JGRP-1157?page=com.atlassian.jira.plugin.sy...
]
Bela Ban resolved JGRP-1157.
----------------------------
Resolution: Done
Please re-open should this fix not solve the issue. If the issue persists, it would be
good to have a reproduceable test case attached !
TCP: JGroups threads get stuck and stop communicating
-----------------------------------------------------
Key: JGRP-1157
URL: https://jira.jboss.org/browse/JGRP-1157
Project: JGroups
Issue Type: Bug
Affects Versions: 2.8, 2.9
Environment: Linux
Reporter: vivek v
Assignee: Vladimir Blagojevic
Fix For: 2.10
Attachments: jgroups_stack.txt, TCPConnection-1.png, TCPConnection-2.png,
threaddump-jg.log
We are having problem where a node gets isolated after some intermittent network outage
and is never able to join back. Bela suspected some issue /w RouterStub and fixed a bug -
JGRP-1151. But, we are were able to reproduce this problem /w even JGroups 2.9 GA. Looks
like the problem is that the node that gets isolated becomes unresponsive as all its
JGroups threads hang. Here is how we reproduced the error /w 3 nodes (Node A -
coordinator also running
Gossip Router, Node B, Node C),
1) We added WANem between A and B - so there are random disconnects,high packet loss and
200 msec of delay
2) Due to our WANem setting B loses connectivity /w GR - in and out
3) We restart A and it becomes isolated. A becomes singleton and never joins back the
group. We see NAKACK on the node C - as A is still able to get to C, but not B. C keeps
dropping messages from A as A is not in its transmission table.
4) We turned on tracing on A, but after a while (couple of hours) we see no JGroups trace
on A - we suspected that some of the JGroups threads might have got stuck. So we took the
thread dump of the java process on A (attached). As you can see there are quite a few
JGroups threads in the waiting state and all are for TCP.send
We are not clear on how or why will the JGroups threads hang. Could outgoing messages be
queued up and not moving for some reason?
The only solution to fix this was to restart all the nodes, which is not desirable.
Attached are the stack trace (thread-dump) and our protocol stack.
This jira was originated from discussion at
http://sourceforge.net/mailarchive/forum.php?thread_name=4B7BC107.9060304...
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:03 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1157) TCP: JGroups threads get stuck and stop communicating
[
https://issues.jboss.org/browse/JGRP-1157?page=com.atlassian.jira.plugin....
]
Victor N commented on JGRP-1157:
--------------------------------
I believe I had the same bug (JGRP-1003) some time ago, and also found that there were 2
different TCPConnection objects.
Now we are using 2.10 and 2.11 for a long time and it never reproduced. Thanks for this
fix, it was really annoying bug!
TCP: JGroups threads get stuck and stop communicating
-----------------------------------------------------
Key: JGRP-1157
URL: https://issues.jboss.org/browse/JGRP-1157
Project: JGroups
Issue Type: Bug
Affects Versions: 2.8, 2.9
Environment: Linux
Reporter: vivek v
Assignee: Vladimir Blagojevic
Fix For: 2.10
Attachments: jgroups_stack.txt, TCPConnection-1.png, TCPConnection-2.png,
threaddump-jg.log
We are having problem where a node gets isolated after some intermittent network outage
and is never able to join back. Bela suspected some issue /w RouterStub and fixed a bug -
JGRP-1151. But, we are were able to reproduce this problem /w even JGroups 2.9 GA. Looks
like the problem is that the node that gets isolated becomes unresponsive as all its
JGroups threads hang. Here is how we reproduced the error /w 3 nodes (Node A -
coordinator also running
Gossip Router, Node B, Node C),
1) We added WANem between A and B - so there are random disconnects,high packet loss and
200 msec of delay
2) Due to our WANem setting B loses connectivity /w GR - in and out
3) We restart A and it becomes isolated. A becomes singleton and never joins back the
group. We see NAKACK on the node C - as A is still able to get to C, but not B. C keeps
dropping messages from A as A is not in its transmission table.
4) We turned on tracing on A, but after a while (couple of hours) we see no JGroups trace
on A - we suspected that some of the JGroups threads might have got stuck. So we took the
thread dump of the java process on A (attached). As you can see there are quite a few
JGroups threads in the waiting state and all are for TCP.send
We are not clear on how or why will the JGroups threads hang. Could outgoing messages be
queued up and not moving for some reason?
The only solution to fix this was to restart all the nodes, which is not desirable.
Attached are the stack trace (thread-dump) and our protocol stack.
This jira was originated from discussion at
http://sourceforge.net/mailarchive/forum.php?thread_name=4B7BC107.9060304...
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5684
days inactive
5999
days old
12 comments
5 participants
participants (5)
-
Bela Ban (JIRA) -
Pete Matern (JIRA)
-
Victor N (JIRA)
-
vivek v (JIRA)
-
Vladimir Blagojevic (JIRA)