2:09 a.m.
Member can not join cluster after heavy load
--------------------------------------------
Key: JGRP-1265
URL: https://issues.jboss.org/browse/JGRP-1265
Project: JGroups
Issue Type: Bug
Affects Versions: 2.11
Environment: linux, kernel 2.6.18
Reporter: Victor N
Assignee: Bela Ban
In our production system I can see that a node desappers from the cluster if its server
was heavily-loaded. It's OK, but the node never comes back to the cluster even after
its server is working normally, without load. I can easily reproduce the problem in 2
cases:
1) by taking a memory dump on the node: jmap -dump:format=b,file=dump.hprof <pid>
Since we have 8-16 GB of RAM, this operation takes much time and blocks JVM - so other
members exclude this node from View.
2) GC (garbage collection) - if JVM is doing GC constantly (and almost can not work)
In both situations the stuck node never reappears in the cluster (even after 1 h). Below
are more details.
We have 12 nodes in our cluster, we problematic node is "gate5".
View on gate5: [gate11.mydomain|869] [gate11.mydomain, gate2.mydomain, gate6.mydomain,
gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain, gate10.mydomain,
gate8.mydomain, gate9.mydomain, gate14.mydomain, gate5.mydomain]
View on gate11 (coordinator): [gate11.mydomain|870] [gate11.mydomain, gate2.mydomain,
gate6.mydomain, gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain,
gate10.mydomain, gate8.mydomain, gate9.mydomain, gate14.mydomain]
The coordinator (gate11) is sending GET_MBRS_REQ periodically - I see them in gate5. But I
do NOT see response to this request!
All jgroups threads are alive, not dead (I took stack traces).
Another strange thing is that the problematic gate5 sends messages to other nodes and even
receives messages from SOME of them! How is it possible - I double-checked that ALL other
nodes have view_id=870 (without gate5)?
The only assumption I have is race-conditions which occurs (as always) under high load.
In normal situations such as temporary network failure everything works perfectly - gate5
joins the cluster.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:11 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-1265) Member can not join cluster after JVM high load
[
https://issues.jboss.org/browse/JGRP-1265?page=com.atlassian.jira.plugin....
]
Victor N updated JGRP-1265:
---------------------------
Summary: Member can not join cluster after JVM high load (was: Member
can not join cluster after heavy load)
Workaround Description: disconnect from JChannel and connect again. (was: Workaround:
disconnect from JChannel and connect again.)
Steps to Reproduce: Start several nodes (>3 I think), set Xmx for JVM to 8 or
even 16 Gbytes, then use jmap tool (mentioned above) to take a memory dump. You should set
config accordingly - so that other nodes will update their view while taking the dump. In
my tests the problematic node where I did tests was NOT coordinator. (was: Steps to
reproduce:
Start several nodes (>3 I think), set Xmx for JVM to 8 or even 16 Gbytes, then use jmap
tool (mentioned above) to take a memory dump. You should set config accordingly - so that
other nodes will update their view while taking the dump. In my tests the problematic node
where I did tests was NOT coordinator.)
Member can not join cluster after JVM high load
-----------------------------------------------
Key: JGRP-1265
URL: https://issues.jboss.org/browse/JGRP-1265
Project: JGroups
Issue Type: Bug
Affects Versions: 2.11
Environment: linux, kernel 2.6.18
Reporter: Victor N
Assignee: Bela Ban
In our production system I can see that a node desappers from the cluster if its server
was heavily-loaded. It's OK, but the node never comes back to the cluster even after
its server is working normally, without load. I can easily reproduce the problem in 2
cases:
1) by taking a memory dump on the node: jmap -dump:format=b,file=dump.hprof <pid>
Since we have 8-16 GB of RAM, this operation takes much time and blocks JVM - so other
members exclude this node from View.
2) GC (garbage collection) - if JVM is doing GC constantly (and almost can not work)
In both situations the stuck node never reappears in the cluster (even after 1 h). Below
are more details.
We have 12 nodes in our cluster, we problematic node is "gate5".
View on gate5: [gate11.mydomain|869] [gate11.mydomain, gate2.mydomain, gate6.mydomain,
gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain, gate10.mydomain,
gate8.mydomain, gate9.mydomain, gate14.mydomain, gate5.mydomain]
View on gate11 (coordinator): [gate11.mydomain|870] [gate11.mydomain, gate2.mydomain,
gate6.mydomain, gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain,
gate10.mydomain, gate8.mydomain, gate9.mydomain, gate14.mydomain]
The coordinator (gate11) is sending GET_MBRS_REQ periodically - I see them in gate5. But
I do NOT see response to this request!
All jgroups threads are alive, not dead (I took stack traces).
Another strange thing is that the problematic gate5 sends messages to other nodes and
even receives messages from SOME of them! How is it possible - I double-checked that ALL
other nodes have view_id=870 (without gate5)?
The only assumption I have is race-conditions which occurs (as always) under high load.
In normal situations such as temporary network failure everything works perfectly - gate5
joins the cluster.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:27 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-1265) Member can not join cluster after JVM high load
[
https://issues.jboss.org/browse/JGRP-1265?page=com.atlassian.jira.plugin....
]
Bela Ban updated JGRP-1265:
---------------------------
Fix Version/s: 2.12
What's the XML config ? Do you have MERGE2 in it ? Is UDP.loopback set to true ?
Member can not join cluster after JVM high load
-----------------------------------------------
Key: JGRP-1265
URL: https://issues.jboss.org/browse/JGRP-1265
Project: JGroups
Issue Type: Bug
Affects Versions: 2.11
Environment: linux, kernel 2.6.18
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.12
In our production system I can see that a node desappers from the cluster if its server
was heavily-loaded. It's OK, but the node never comes back to the cluster even after
its server is working normally, without load. I can easily reproduce the problem in 2
cases:
1) by taking a memory dump on the node: jmap -dump:format=b,file=dump.hprof <pid>
Since we have 8-16 GB of RAM, this operation takes much time and blocks JVM - so other
members exclude this node from View.
2) GC (garbage collection) - if JVM is doing GC constantly (and almost can not work)
In both situations the stuck node never reappears in the cluster (even after 1 h). Below
are more details.
We have 12 nodes in our cluster, we problematic node is "gate5".
View on gate5: [gate11.mydomain|869] [gate11.mydomain, gate2.mydomain, gate6.mydomain,
gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain, gate10.mydomain,
gate8.mydomain, gate9.mydomain, gate14.mydomain, gate5.mydomain]
View on gate11 (coordinator): [gate11.mydomain|870] [gate11.mydomain, gate2.mydomain,
gate6.mydomain, gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain,
gate10.mydomain, gate8.mydomain, gate9.mydomain, gate14.mydomain]
The coordinator (gate11) is sending GET_MBRS_REQ periodically - I see them in gate5. But
I do NOT see response to this request!
All jgroups threads are alive, not dead (I took stack traces).
Another strange thing is that the problematic gate5 sends messages to other nodes and
even receives messages from SOME of them! How is it possible - I double-checked that ALL
other nodes have view_id=870 (without gate5)?
The only assumption I have is race-conditions which occurs (as always) under high load.
In normal situations such as temporary network failure everything works perfectly - gate5
joins the cluster.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:33 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1265) Member can not join cluster after JVM high load
[
https://issues.jboss.org/browse/JGRP-1265?page=com.atlassian.jira.plugin....
]
Bela Ban commented on JGRP-1265:
--------------------------------
I don't have access to machines with that much memory, can this be reproduced
differently ?
Member can not join cluster after JVM high load
-----------------------------------------------
Key: JGRP-1265
URL: https://issues.jboss.org/browse/JGRP-1265
Project: JGroups
Issue Type: Bug
Affects Versions: 2.11
Environment: linux, kernel 2.6.18
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.12
In our production system I can see that a node desappers from the cluster if its server
was heavily-loaded. It's OK, but the node never comes back to the cluster even after
its server is working normally, without load. I can easily reproduce the problem in 2
cases:
1) by taking a memory dump on the node: jmap -dump:format=b,file=dump.hprof <pid>
Since we have 8-16 GB of RAM, this operation takes much time and blocks JVM - so other
members exclude this node from View.
2) GC (garbage collection) - if JVM is doing GC constantly (and almost can not work)
In both situations the stuck node never reappears in the cluster (even after 1 h). Below
are more details.
We have 12 nodes in our cluster, we problematic node is "gate5".
View on gate5: [gate11.mydomain|869] [gate11.mydomain, gate2.mydomain, gate6.mydomain,
gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain, gate10.mydomain,
gate8.mydomain, gate9.mydomain, gate14.mydomain, gate5.mydomain]
View on gate11 (coordinator): [gate11.mydomain|870] [gate11.mydomain, gate2.mydomain,
gate6.mydomain, gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain,
gate10.mydomain, gate8.mydomain, gate9.mydomain, gate14.mydomain]
The coordinator (gate11) is sending GET_MBRS_REQ periodically - I see them in gate5. But
I do NOT see response to this request!
All jgroups threads are alive, not dead (I took stack traces).
Another strange thing is that the problematic gate5 sends messages to other nodes and
even receives messages from SOME of them! How is it possible - I double-checked that ALL
other nodes have view_id=870 (without gate5)?
The only assumption I have is race-conditions which occurs (as always) under high load.
In normal situations such as temporary network failure everything works perfectly - gate5
joins the cluster.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:44 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-1265) Member can not join cluster after JVM high load
[
https://issues.jboss.org/browse/JGRP-1265?page=com.atlassian.jira.plugin....
]
Victor N updated JGRP-1265:
---------------------------
Attachment: jgroups-tcp.xml
sorry, forgot to attach my config
Member can not join cluster after JVM high load
-----------------------------------------------
Key: JGRP-1265
URL: https://issues.jboss.org/browse/JGRP-1265
Project: JGroups
Issue Type: Bug
Affects Versions: 2.11
Environment: linux, kernel 2.6.18
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.12
Attachments: jgroups-tcp.xml
In our production system I can see that a node desappers from the cluster if its server
was heavily-loaded. It's OK, but the node never comes back to the cluster even after
its server is working normally, without load. I can easily reproduce the problem in 2
cases:
1) by taking a memory dump on the node: jmap -dump:format=b,file=dump.hprof <pid>
Since we have 8-16 GB of RAM, this operation takes much time and blocks JVM - so other
members exclude this node from View.
2) GC (garbage collection) - if JVM is doing GC constantly (and almost can not work)
In both situations the stuck node never reappears in the cluster (even after 1 h). Below
are more details.
We have 12 nodes in our cluster, we problematic node is "gate5".
View on gate5: [gate11.mydomain|869] [gate11.mydomain, gate2.mydomain, gate6.mydomain,
gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain, gate10.mydomain,
gate8.mydomain, gate9.mydomain, gate14.mydomain, gate5.mydomain]
View on gate11 (coordinator): [gate11.mydomain|870] [gate11.mydomain, gate2.mydomain,
gate6.mydomain, gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain,
gate10.mydomain, gate8.mydomain, gate9.mydomain, gate14.mydomain]
The coordinator (gate11) is sending GET_MBRS_REQ periodically - I see them in gate5. But
I do NOT see response to this request!
All jgroups threads are alive, not dead (I took stack traces).
Another strange thing is that the problematic gate5 sends messages to other nodes and
even receives messages from SOME of them! How is it possible - I double-checked that ALL
other nodes have view_id=870 (without gate5)?
The only assumption I have is race-conditions which occurs (as always) under high load.
In normal situations such as temporary network failure everything works perfectly - gate5
joins the cluster.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:48 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1265) Member can not join cluster after JVM high load
[
https://issues.jboss.org/browse/JGRP-1265?page=com.atlassian.jira.plugin....
]
Victor N commented on JGRP-1265:
--------------------------------
I think it can also be reproduced by forcing GC to work constantly and heavily - by
allocating too much memory for some time (so other nodes could exclude this one) and then
stop allocating memory, so JVM will work normally.
Member can not join cluster after JVM high load
-----------------------------------------------
Key: JGRP-1265
URL: https://issues.jboss.org/browse/JGRP-1265
Project: JGroups
Issue Type: Bug
Affects Versions: 2.11
Environment: linux, kernel 2.6.18
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.12
Attachments: jgroups-tcp.xml
In our production system I can see that a node desappers from the cluster if its server
was heavily-loaded. It's OK, but the node never comes back to the cluster even after
its server is working normally, without load. I can easily reproduce the problem in 2
cases:
1) by taking a memory dump on the node: jmap -dump:format=b,file=dump.hprof <pid>
Since we have 8-16 GB of RAM, this operation takes much time and blocks JVM - so other
members exclude this node from View.
2) GC (garbage collection) - if JVM is doing GC constantly (and almost can not work)
In both situations the stuck node never reappears in the cluster (even after 1 h). Below
are more details.
We have 12 nodes in our cluster, we problematic node is "gate5".
View on gate5: [gate11.mydomain|869] [gate11.mydomain, gate2.mydomain, gate6.mydomain,
gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain, gate10.mydomain,
gate8.mydomain, gate9.mydomain, gate14.mydomain, gate5.mydomain]
View on gate11 (coordinator): [gate11.mydomain|870] [gate11.mydomain, gate2.mydomain,
gate6.mydomain, gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain,
gate10.mydomain, gate8.mydomain, gate9.mydomain, gate14.mydomain]
The coordinator (gate11) is sending GET_MBRS_REQ periodically - I see them in gate5. But
I do NOT see response to this request!
All jgroups threads are alive, not dead (I took stack traces).
Another strange thing is that the problematic gate5 sends messages to other nodes and
even receives messages from SOME of them! How is it possible - I double-checked that ALL
other nodes have view_id=870 (without gate5)?
The only assumption I have is race-conditions which occurs (as always) under high load.
In normal situations such as temporary network failure everything works perfectly - gate5
joins the cluster.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:38 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1265) Member can not join cluster after JVM high load
[
https://issues.jboss.org/browse/JGRP-1265?page=com.atlassian.jira.plugin....
]
Bela Ban commented on JGRP-1265:
--------------------------------
I can't reproduce this: I wrote a test which allocates memory, until it (a) runs out
of memory and (b) is excluded from the cluster. Then, it de-allocates the memory and after
a short time, it gets merged back into the cluster...
Member can not join cluster after JVM high load
-----------------------------------------------
Key: JGRP-1265
URL: https://issues.jboss.org/browse/JGRP-1265
Project: JGroups
Issue Type: Bug
Affects Versions: 2.11
Environment: linux, kernel 2.6.18
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.12
Attachments: jgroups-tcp.xml
In our production system I can see that a node desappers from the cluster if its server
was heavily-loaded. It's OK, but the node never comes back to the cluster even after
its server is working normally, without load. I can easily reproduce the problem in 2
cases:
1) by taking a memory dump on the node: jmap -dump:format=b,file=dump.hprof <pid>
Since we have 8-16 GB of RAM, this operation takes much time and blocks JVM - so other
members exclude this node from View.
2) GC (garbage collection) - if JVM is doing GC constantly (and almost can not work)
In both situations the stuck node never reappears in the cluster (even after 1 h). Below
are more details.
We have 12 nodes in our cluster, we problematic node is "gate5".
View on gate5: [gate11.mydomain|869] [gate11.mydomain, gate2.mydomain, gate6.mydomain,
gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain, gate10.mydomain,
gate8.mydomain, gate9.mydomain, gate14.mydomain, gate5.mydomain]
View on gate11 (coordinator): [gate11.mydomain|870] [gate11.mydomain, gate2.mydomain,
gate6.mydomain, gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain,
gate10.mydomain, gate8.mydomain, gate9.mydomain, gate14.mydomain]
The coordinator (gate11) is sending GET_MBRS_REQ periodically - I see them in gate5. But
I do NOT see response to this request!
All jgroups threads are alive, not dead (I took stack traces).
Another strange thing is that the problematic gate5 sends messages to other nodes and
even receives messages from SOME of them! How is it possible - I double-checked that ALL
other nodes have view_id=870 (without gate5)?
The only assumption I have is race-conditions which occurs (as always) under high load.
In normal situations such as temporary network failure everything works perfectly - gate5
joins the cluster.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:46 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1265) Member can not join cluster after JVM high load
[
https://issues.jboss.org/browse/JGRP-1265?page=com.atlassian.jira.plugin....
]
Bela Ban commented on JGRP-1265:
--------------------------------
Same with generating CPU load: I generate load for 30 seconds, the nodes gets excluded
from the cluster but later rejoins.
Member can not join cluster after JVM high load
-----------------------------------------------
Key: JGRP-1265
URL: https://issues.jboss.org/browse/JGRP-1265
Project: JGroups
Issue Type: Bug
Affects Versions: 2.11
Environment: linux, kernel 2.6.18
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.12
Attachments: jgroups-tcp.xml
In our production system I can see that a node desappers from the cluster if its server
was heavily-loaded. It's OK, but the node never comes back to the cluster even after
its server is working normally, without load. I can easily reproduce the problem in 2
cases:
1) by taking a memory dump on the node: jmap -dump:format=b,file=dump.hprof <pid>
Since we have 8-16 GB of RAM, this operation takes much time and blocks JVM - so other
members exclude this node from View.
2) GC (garbage collection) - if JVM is doing GC constantly (and almost can not work)
In both situations the stuck node never reappears in the cluster (even after 1 h). Below
are more details.
We have 12 nodes in our cluster, we problematic node is "gate5".
View on gate5: [gate11.mydomain|869] [gate11.mydomain, gate2.mydomain, gate6.mydomain,
gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain, gate10.mydomain,
gate8.mydomain, gate9.mydomain, gate14.mydomain, gate5.mydomain]
View on gate11 (coordinator): [gate11.mydomain|870] [gate11.mydomain, gate2.mydomain,
gate6.mydomain, gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain,
gate10.mydomain, gate8.mydomain, gate9.mydomain, gate14.mydomain]
The coordinator (gate11) is sending GET_MBRS_REQ periodically - I see them in gate5. But
I do NOT see response to this request!
All jgroups threads are alive, not dead (I took stack traces).
Another strange thing is that the problematic gate5 sends messages to other nodes and
even receives messages from SOME of them! How is it possible - I double-checked that ALL
other nodes have view_id=870 (without gate5)?
The only assumption I have is race-conditions which occurs (as always) under high load.
In normal situations such as temporary network failure everything works perfectly - gate5
joins the cluster.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:48 a.m.
New subject: [JBoss JIRA] Issue Comment Edited: (JGRP-1265) Member can not join cluster after JVM high load
[
https://issues.jboss.org/browse/JGRP-1265?page=com.atlassian.jira.plugin....
]
Bela Ban edited comment on JGRP-1265 at 1/4/11 4:46 AM:
--------------------------------------------------------
Same with generating CPU load: I generate load for 30 seconds, the nodes gets excluded
from the cluster but is later merged back into the cluster.
was (Author: bela(a)jboss.com):
Same with generating CPU load: I generate load for 30 seconds, the nodes gets excluded
from the cluster but later rejoins.
Member can not join cluster after JVM high load
-----------------------------------------------
Key: JGRP-1265
URL: https://issues.jboss.org/browse/JGRP-1265
Project: JGroups
Issue Type: Bug
Affects Versions: 2.11
Environment: linux, kernel 2.6.18
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.12
Attachments: jgroups-tcp.xml
In our production system I can see that a node desappers from the cluster if its server
was heavily-loaded. It's OK, but the node never comes back to the cluster even after
its server is working normally, without load. I can easily reproduce the problem in 2
cases:
1) by taking a memory dump on the node: jmap -dump:format=b,file=dump.hprof <pid>
Since we have 8-16 GB of RAM, this operation takes much time and blocks JVM - so other
members exclude this node from View.
2) GC (garbage collection) - if JVM is doing GC constantly (and almost can not work)
In both situations the stuck node never reappears in the cluster (even after 1 h). Below
are more details.
We have 12 nodes in our cluster, we problematic node is "gate5".
View on gate5: [gate11.mydomain|869] [gate11.mydomain, gate2.mydomain, gate6.mydomain,
gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain, gate10.mydomain,
gate8.mydomain, gate9.mydomain, gate14.mydomain, gate5.mydomain]
View on gate11 (coordinator): [gate11.mydomain|870] [gate11.mydomain, gate2.mydomain,
gate6.mydomain, gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain,
gate10.mydomain, gate8.mydomain, gate9.mydomain, gate14.mydomain]
The coordinator (gate11) is sending GET_MBRS_REQ periodically - I see them in gate5. But
I do NOT see response to this request!
All jgroups threads are alive, not dead (I took stack traces).
Another strange thing is that the problematic gate5 sends messages to other nodes and
even receives messages from SOME of them! How is it possible - I double-checked that ALL
other nodes have view_id=870 (without gate5)?
The only assumption I have is race-conditions which occurs (as always) under high load.
In normal situations such as temporary network failure everything works perfectly - gate5
joins the cluster.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:32 p.m.
New subject: [JBoss JIRA] Commented: (JGRP-1265) Member can not join cluster after JVM high load
[
https://issues.jboss.org/browse/JGRP-1265?page=com.atlassian.jira.plugin....
]
ronald yang commented on JGRP-1265:
-----------------------------------
I have seen something very similar to this. I have a load test where every node in a
cluster broadcasts to everyone else incessantly with no wait time. If I send a sigstop to
one process, wait a while, then resume it with a sigcont, there's a 50/50 chance it
won't be allowed back in. If this is the case then one or more of the other nodes
will experience OOME in their NakReceiverWindow (or thereabouts). I apologize in advance
for not producing a self contained test case, but time hasn't permitted.
Member can not join cluster after JVM high load
-----------------------------------------------
Key: JGRP-1265
URL: https://issues.jboss.org/browse/JGRP-1265
Project: JGroups
Issue Type: Bug
Affects Versions: 2.11
Environment: linux, kernel 2.6.18
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.12
Attachments: jgroups-tcp.xml
In our production system I can see that a node desappers from the cluster if its server
was heavily-loaded. It's OK, but the node never comes back to the cluster even after
its server is working normally, without load. I can easily reproduce the problem in 2
cases:
1) by taking a memory dump on the node: jmap -dump:format=b,file=dump.hprof <pid>
Since we have 8-16 GB of RAM, this operation takes much time and blocks JVM - so other
members exclude this node from View.
2) GC (garbage collection) - if JVM is doing GC constantly (and almost can not work)
In both situations the stuck node never reappears in the cluster (even after 1 h). Below
are more details.
We have 12 nodes in our cluster, we problematic node is "gate5".
View on gate5: [gate11.mydomain|869] [gate11.mydomain, gate2.mydomain, gate6.mydomain,
gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain, gate10.mydomain,
gate8.mydomain, gate9.mydomain, gate14.mydomain, gate5.mydomain]
View on gate11 (coordinator): [gate11.mydomain|870] [gate11.mydomain, gate2.mydomain,
gate6.mydomain, gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain,
gate10.mydomain, gate8.mydomain, gate9.mydomain, gate14.mydomain]
The coordinator (gate11) is sending GET_MBRS_REQ periodically - I see them in gate5. But
I do NOT see response to this request!
All jgroups threads are alive, not dead (I took stack traces).
Another strange thing is that the problematic gate5 sends messages to other nodes and
even receives messages from SOME of them! How is it possible - I double-checked that ALL
other nodes have view_id=870 (without gate5)?
The only assumption I have is race-conditions which occurs (as always) under high load.
In normal situations such as temporary network failure everything works perfectly - gate5
joins the cluster.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:58 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1265) Member can not join cluster after JVM high load
[
https://issues.jboss.org/browse/JGRP-1265?page=com.atlassian.jira.plugin....
]
Victor N commented on JGRP-1265:
--------------------------------
I can not reproduce it in artificial tests, but the problem reproduced frequently last
days of 2010 when we had too many users! I looked into TRACE logs but did not see any
useful info. If possible, it would be greate to write even more detailed logs at TRACE
level or use deatiled JMX monitoring. Is it possible?
Member can not join cluster after JVM high load
-----------------------------------------------
Key: JGRP-1265
URL: https://issues.jboss.org/browse/JGRP-1265
Project: JGroups
Issue Type: Bug
Affects Versions: 2.11
Environment: linux, kernel 2.6.18
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.12
Attachments: jgroups-tcp.xml
In our production system I can see that a node desappers from the cluster if its server
was heavily-loaded. It's OK, but the node never comes back to the cluster even after
its server is working normally, without load. I can easily reproduce the problem in 2
cases:
1) by taking a memory dump on the node: jmap -dump:format=b,file=dump.hprof <pid>
Since we have 8-16 GB of RAM, this operation takes much time and blocks JVM - so other
members exclude this node from View.
2) GC (garbage collection) - if JVM is doing GC constantly (and almost can not work)
In both situations the stuck node never reappears in the cluster (even after 1 h). Below
are more details.
We have 12 nodes in our cluster, we problematic node is "gate5".
View on gate5: [gate11.mydomain|869] [gate11.mydomain, gate2.mydomain, gate6.mydomain,
gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain, gate10.mydomain,
gate8.mydomain, gate9.mydomain, gate14.mydomain, gate5.mydomain]
View on gate11 (coordinator): [gate11.mydomain|870] [gate11.mydomain, gate2.mydomain,
gate6.mydomain, gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain,
gate10.mydomain, gate8.mydomain, gate9.mydomain, gate14.mydomain]
The coordinator (gate11) is sending GET_MBRS_REQ periodically - I see them in gate5. But
I do NOT see response to this request!
All jgroups threads are alive, not dead (I took stack traces).
Another strange thing is that the problematic gate5 sends messages to other nodes and
even receives messages from SOME of them! How is it possible - I double-checked that ALL
other nodes have view_id=870 (without gate5)?
The only assumption I have is race-conditions which occurs (as always) under high load.
In normal situations such as temporary network failure everything works perfectly - gate5
joins the cluster.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:08 a.m.
New subject: [JBoss JIRA] Resolved: (JGRP-1265) Member can not join cluster after JVM high load
[
https://issues.jboss.org/browse/JGRP-1265?page=com.atlassian.jira.plugin....
]
Bela Ban resolved JGRP-1265.
----------------------------
Resolution: Cannot Reproduce Bug
Member can not join cluster after JVM high load
-----------------------------------------------
Key: JGRP-1265
URL: https://issues.jboss.org/browse/JGRP-1265
Project: JGroups
Issue Type: Bug
Affects Versions: 2.11
Environment: linux, kernel 2.6.18
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.12
Attachments: jgroups-tcp.xml
In our production system I can see that a node desappers from the cluster if its server
was heavily-loaded. It's OK, but the node never comes back to the cluster even after
its server is working normally, without load. I can easily reproduce the problem in 2
cases:
1) by taking a memory dump on the node: jmap -dump:format=b,file=dump.hprof <pid>
Since we have 8-16 GB of RAM, this operation takes much time and blocks JVM - so other
members exclude this node from View.
2) GC (garbage collection) - if JVM is doing GC constantly (and almost can not work)
In both situations the stuck node never reappears in the cluster (even after 1 h). Below
are more details.
We have 12 nodes in our cluster, we problematic node is "gate5".
View on gate5: [gate11.mydomain|869] [gate11.mydomain, gate2.mydomain, gate6.mydomain,
gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain, gate10.mydomain,
gate8.mydomain, gate9.mydomain, gate14.mydomain, gate5.mydomain]
View on gate11 (coordinator): [gate11.mydomain|870] [gate11.mydomain, gate2.mydomain,
gate6.mydomain, gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain,
gate10.mydomain, gate8.mydomain, gate9.mydomain, gate14.mydomain]
The coordinator (gate11) is sending GET_MBRS_REQ periodically - I see them in gate5. But
I do NOT see response to this request!
All jgroups threads are alive, not dead (I took stack traces).
Another strange thing is that the problematic gate5 sends messages to other nodes and
even receives messages from SOME of them! How is it possible - I double-checked that ALL
other nodes have view_id=870 (without gate5)?
The only assumption I have is race-conditions which occurs (as always) under high load.
In normal situations such as temporary network failure everything works perfectly - gate5
joins the cluster.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:08 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1265) Member can not join cluster after JVM high load
[
https://issues.jboss.org/browse/JGRP-1265?page=com.atlassian.jira.plugin....
]
Bela Ban commented on JGRP-1265:
--------------------------------
TRACE is the highest level we can log things at...
@Victor and @Ronald: The perf.Test I run frequently has everyone send messages at high
rates and I've never seen this issue...
I'm afraid I'll have to close this case, as I couldn't reproduce the issue,
with either high CPU load or high memory consumption triggered by the test driver: the
excluded node was always able to merge back into the cluster once this memory/CPU usage
decreased.
However, I'm very interested in getting a reproduceable case that I can test on my
system. Please re-open the case if you have such a case.
Cheers,
Member can not join cluster after JVM high load
-----------------------------------------------
Key: JGRP-1265
URL: https://issues.jboss.org/browse/JGRP-1265
Project: JGroups
Issue Type: Bug
Affects Versions: 2.11
Environment: linux, kernel 2.6.18
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.12
Attachments: jgroups-tcp.xml
In our production system I can see that a node desappers from the cluster if its server
was heavily-loaded. It's OK, but the node never comes back to the cluster even after
its server is working normally, without load. I can easily reproduce the problem in 2
cases:
1) by taking a memory dump on the node: jmap -dump:format=b,file=dump.hprof <pid>
Since we have 8-16 GB of RAM, this operation takes much time and blocks JVM - so other
members exclude this node from View.
2) GC (garbage collection) - if JVM is doing GC constantly (and almost can not work)
In both situations the stuck node never reappears in the cluster (even after 1 h). Below
are more details.
We have 12 nodes in our cluster, we problematic node is "gate5".
View on gate5: [gate11.mydomain|869] [gate11.mydomain, gate2.mydomain, gate6.mydomain,
gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain, gate10.mydomain,
gate8.mydomain, gate9.mydomain, gate14.mydomain, gate5.mydomain]
View on gate11 (coordinator): [gate11.mydomain|870] [gate11.mydomain, gate2.mydomain,
gate6.mydomain, gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain,
gate10.mydomain, gate8.mydomain, gate9.mydomain, gate14.mydomain]
The coordinator (gate11) is sending GET_MBRS_REQ periodically - I see them in gate5. But
I do NOT see response to this request!
All jgroups threads are alive, not dead (I took stack traces).
Another strange thing is that the problematic gate5 sends messages to other nodes and
even receives messages from SOME of them! How is it possible - I double-checked that ALL
other nodes have view_id=870 (without gate5)?
The only assumption I have is race-conditions which occurs (as always) under high load.
In normal situations such as temporary network failure everything works perfectly - gate5
joins the cluster.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:48 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1265) Member can not join cluster after JVM high load
[
https://issues.jboss.org/browse/JGRP-1265?page=com.atlassian.jira.plugin....
]
Victor N commented on JGRP-1265:
--------------------------------
Bela, yes I understand how it is difficult to reproduce the problem without test at your
side. Sorry I do not know what test it should be :-( The only thing I know exactly is that
the problem is reproduced sometimes. And seems not only for me. I can gather any info you
ask - I can build jgroups myself with any additional logs and maybe JMX monitoring - just
give me some points where to go please! It is sad that bugs are closed just because we can
not provide a ready-to-use test.
Member can not join cluster after JVM high load
-----------------------------------------------
Key: JGRP-1265
URL: https://issues.jboss.org/browse/JGRP-1265
Project: JGroups
Issue Type: Bug
Affects Versions: 2.11
Environment: linux, kernel 2.6.18
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.12
Attachments: jgroups-tcp.xml
In our production system I can see that a node desappers from the cluster if its server
was heavily-loaded. It's OK, but the node never comes back to the cluster even after
its server is working normally, without load. I can easily reproduce the problem in 2
cases:
1) by taking a memory dump on the node: jmap -dump:format=b,file=dump.hprof <pid>
Since we have 8-16 GB of RAM, this operation takes much time and blocks JVM - so other
members exclude this node from View.
2) GC (garbage collection) - if JVM is doing GC constantly (and almost can not work)
In both situations the stuck node never reappears in the cluster (even after 1 h). Below
are more details.
We have 12 nodes in our cluster, we problematic node is "gate5".
View on gate5: [gate11.mydomain|869] [gate11.mydomain, gate2.mydomain, gate6.mydomain,
gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain, gate10.mydomain,
gate8.mydomain, gate9.mydomain, gate14.mydomain, gate5.mydomain]
View on gate11 (coordinator): [gate11.mydomain|870] [gate11.mydomain, gate2.mydomain,
gate6.mydomain, gate7.mydomain, gate12.mydomain, gate4.mydomain, gate3.mydomain,
gate10.mydomain, gate8.mydomain, gate9.mydomain, gate14.mydomain]
The coordinator (gate11) is sending GET_MBRS_REQ periodically - I see them in gate5. But
I do NOT see response to this request!
All jgroups threads are alive, not dead (I took stack traces).
Another strange thing is that the problematic gate5 sends messages to other nodes and
even receives messages from SOME of them! How is it possible - I double-checked that ALL
other nodes have view_id=870 (without gate5)?
The only assumption I have is race-conditions which occurs (as always) under high load.
In normal situations such as temporary network failure everything works perfectly - gate5
joins the cluster.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5581
days inactive
5599
days old
13 comments
3 participants
participants (3)
-
Bela Ban (JIRA) -
ronald yang (JIRA)
-
Victor N (JIRA)