11:17 p.m.
Unable to recover from suspect/merge, with auto-reconnect
---------------------------------------------------------
Key: JGRP-692
URL: http://jira.jboss.com/jira/browse/JGRP-692
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.1
Environment: JBoss 4.2.2 on Linux 2.4.21-47.0.1.ELsmp i686 using Java HotSpot(TM)
Server VM (build 1.5.0_10-b03, mixed mode)
Reporter: Matt Magoffin
Assigned To: Bela Ban
Attachments: jgroups-logs.tbz2
I'm having an issue with a 2-machine cluster using a TCP stack
based on the tcp.xml from JGroups 2.6.1. On each machine I have 8 separate
channels running, on different ports, with 4 groups in 2 JVM instances.
After some period of time, one machine will fail to respond to a FD ping,
and gets suspected. The machine that failed is not responding in time it
seems from high CPU use, and many of the channels will fail FD around the
same time. The channels are configured with auto-reconnect. My
understanding was that the channel should "heal" itself and eventually
re-form into a new view with the same 2 members in the cluster, which
should apply to this situation because the machine that failed to respond
eventually will respond.
However, the group does not always seem to "heal" (sometimes it does,
sometimes not). Once it stops healing, it never seems to ever do so again,
and I get tons of NAKACK "message X not found in retransmission table"
ERROR logs. The only way to get the channel working agin is to shut down
the channel on both machines and then start them up again.
I'm not using muxed channels... just normal channels.
I have for now disabled shunning, and the channels seem to be able to
re-connect after a shun situation occurs, but after time I'm still seeing
something wrong with the channel in that the nodes are not able to send
messages to each other successfully, and I have tons of
2008-02-21 21:01:53,572 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
log entries, even while FD is receiving acks on that channel.
this is a degradation problem for me while my servers are running,
not just during deployment. After a while of running, the nodes get
shunned/disconnected some how (often from a slow response from the other
node) and then fail to ever merge back to form the original view of the 2
nodes again.
Now even with shun set to false in both FD and pbcast.GMS on both nodes
for all channels, eventually still the nodes reach the same state of never
re-forming... and I have tons eventual
2008-02-21 21:28:24,857 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
After a while in my application I'll force the channel to close wait a
minute, and reopen, and sometimes this gets the channel working again, but
doesn't always seem to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:17 p.m.
New subject: [JBoss JIRA] Updated: (JGRP-692) Unable to recover from suspect/merge, with auto-reconnect
[ http://jira.jboss.com/jira/browse/JGRP-692?page=all ]
Matt Magoffin updated JGRP-692:
-------------------------------
Attachment: jgroups-logs.tbz2
Hopefully these 2 logs files can be of help to you. There is one long from
each node, app1 and app2 (.233 and .234 IP addresses). There are several
channels running, all on different ports.
I've started both logs at about the same point in time, when app2 was shut
restarted in order to deploy some new code. This way you can see how app1
responds to a normal exit by app2 from the group, and see how app1 sees
app2 come back up again (line 412 in app1 log, at 2008-02-21
18:47:32,014).
On line 752 in app1 at 2008-02-21 19:42:34,794, FD does not get acks, and
suspects app2.
On line 979 in app1 at 2008-02-21 19:43:49,829 there is a NAKACK discard
message to a message from app2, followed by some
[org.jgroups.protocols.FD] Recevied Ack. is invalid (was from:
172.16.172.234:19282)
type stuff. Things then don't seem to work again on app1, but you can see
the application forces one channel closed, which happens on line 6186
(2008-02-21 21:13:10,946), in an attempt to get the channel working again.
By line 6216 (2008-02-21 21:13:14,444) that reset chanel is up and
working, and messages are getting sent to app2. But I still a NAKACK
warning coming from this newly reset channel on lin 6221:
2008-02-21 21:13:15,119 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
I was hoping you could glean a pattern from this stuff.
Unable to recover from suspect/merge, with auto-reconnect
---------------------------------------------------------
Key: JGRP-692
URL: http://jira.jboss.com/jira/browse/JGRP-692
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.1
Environment: JBoss 4.2.2 on Linux 2.4.21-47.0.1.ELsmp i686 using Java HotSpot(TM)
Server VM (build 1.5.0_10-b03, mixed mode)
Reporter: Matt Magoffin
Assigned To: Bela Ban
Attachments: jgroups-logs.tbz2
I'm having an issue with a 2-machine cluster using a TCP stack
based on the tcp.xml from JGroups 2.6.1. On each machine I have 8 separate
channels running, on different ports, with 4 groups in 2 JVM instances.
After some period of time, one machine will fail to respond to a FD ping,
and gets suspected. The machine that failed is not responding in time it
seems from high CPU use, and many of the channels will fail FD around the
same time. The channels are configured with auto-reconnect. My
understanding was that the channel should "heal" itself and eventually
re-form into a new view with the same 2 members in the cluster, which
should apply to this situation because the machine that failed to respond
eventually will respond.
However, the group does not always seem to "heal" (sometimes it does,
sometimes not). Once it stops healing, it never seems to ever do so again,
and I get tons of NAKACK "message X not found in retransmission table"
ERROR logs. The only way to get the channel working agin is to shut down
the channel on both machines and then start them up again.
I'm not using muxed channels... just normal channels.
I have for now disabled shunning, and the channels seem to be able to
re-connect after a shun situation occurs, but after time I'm still seeing
something wrong with the channel in that the nodes are not able to send
messages to each other successfully, and I have tons of
2008-02-21 21:01:53,572 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
log entries, even while FD is receiving acks on that channel.
this is a degradation problem for me while my servers are running,
not just during deployment. After a while of running, the nodes get
shunned/disconnected some how (often from a slow response from the other
node) and then fail to ever merge back to form the original view of the 2
nodes again.
Now even with shun set to false in both FD and pbcast.GMS on both nodes
for all channels, eventually still the nodes reach the same state of never
re-forming... and I have tons eventual
2008-02-21 21:28:24,857 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
After a while in my application I'll force the channel to close wait a
minute, and reopen, and sometimes this gets the channel working again, but
doesn't always seem to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:53 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-692) Unable to recover from suspect/merge, with auto-reconnect
[ http://jira.jboss.com/jira/browse/JGRP-692?page=all ]
Bela Ban updated JGRP-692:
--------------------------
Fix Version/s: 2.6.3
2.7
Unable to recover from suspect/merge, with auto-reconnect
---------------------------------------------------------
Key: JGRP-692
URL: http://jira.jboss.com/jira/browse/JGRP-692
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.1
Environment: JBoss 4.2.2 on Linux 2.4.21-47.0.1.ELsmp i686 using Java HotSpot(TM)
Server VM (build 1.5.0_10-b03, mixed mode)
Reporter: Matt Magoffin
Assigned To: Bela Ban
Fix For: 2.7, 2.6.3
Attachments: jgroups-logs.tbz2
I'm having an issue with a 2-machine cluster using a TCP stack
based on the tcp.xml from JGroups 2.6.1. On each machine I have 8 separate
channels running, on different ports, with 4 groups in 2 JVM instances.
After some period of time, one machine will fail to respond to a FD ping,
and gets suspected. The machine that failed is not responding in time it
seems from high CPU use, and many of the channels will fail FD around the
same time. The channels are configured with auto-reconnect. My
understanding was that the channel should "heal" itself and eventually
re-form into a new view with the same 2 members in the cluster, which
should apply to this situation because the machine that failed to respond
eventually will respond.
However, the group does not always seem to "heal" (sometimes it does,
sometimes not). Once it stops healing, it never seems to ever do so again,
and I get tons of NAKACK "message X not found in retransmission table"
ERROR logs. The only way to get the channel working agin is to shut down
the channel on both machines and then start them up again.
I'm not using muxed channels... just normal channels.
I have for now disabled shunning, and the channels seem to be able to
re-connect after a shun situation occurs, but after time I'm still seeing
something wrong with the channel in that the nodes are not able to send
messages to each other successfully, and I have tons of
2008-02-21 21:01:53,572 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
log entries, even while FD is receiving acks on that channel.
this is a degradation problem for me while my servers are running,
not just during deployment. After a while of running, the nodes get
shunned/disconnected some how (often from a slow response from the other
node) and then fail to ever merge back to form the original view of the 2
nodes again.
Now even with shun set to false in both FD and pbcast.GMS on both nodes
for all channels, eventually still the nodes reach the same state of never
re-forming... and I have tons eventual
2008-02-21 21:28:24,857 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
After a while in my application I'll force the channel to close wait a
minute, and reopen, and sometimes this gets the channel working again, but
doesn't always seem to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:38 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-692) Unable to recover from suspect/merge, with auto-reconnect
[ http://jira.jboss.com/jira/browse/JGRP-692?page=comments#action_12400458 ]
Bela Ban commented on JGRP-692:
-------------------------------
The version used seems to be 2.4.x, *not* 2.6.x ! "Recevied Ack. is invalid"
occurs only in 2.4, but not in 2.5/2.6 !
Looks like http://jira.jboss.com/jira/browse/JGRP-699 fixes this issue, please validate
this. I'm closing this as duplicate of http://jira.jboss.com/jira/browse/JGRP-699.
Unable to recover from suspect/merge, with auto-reconnect
---------------------------------------------------------
Key: JGRP-692
URL: http://jira.jboss.com/jira/browse/JGRP-692
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.1
Environment: JBoss 4.2.2 on Linux 2.4.21-47.0.1.ELsmp i686 using Java HotSpot(TM)
Server VM (build 1.5.0_10-b03, mixed mode)
Reporter: Matt Magoffin
Assigned To: Bela Ban
Fix For: 2.6.2
Attachments: jgroups-logs.tbz2
I'm having an issue with a 2-machine cluster using a TCP stack
based on the tcp.xml from JGroups 2.6.1. On each machine I have 8 separate
channels running, on different ports, with 4 groups in 2 JVM instances.
After some period of time, one machine will fail to respond to a FD ping,
and gets suspected. The machine that failed is not responding in time it
seems from high CPU use, and many of the channels will fail FD around the
same time. The channels are configured with auto-reconnect. My
understanding was that the channel should "heal" itself and eventually
re-form into a new view with the same 2 members in the cluster, which
should apply to this situation because the machine that failed to respond
eventually will respond.
However, the group does not always seem to "heal" (sometimes it does,
sometimes not). Once it stops healing, it never seems to ever do so again,
and I get tons of NAKACK "message X not found in retransmission table"
ERROR logs. The only way to get the channel working agin is to shut down
the channel on both machines and then start them up again.
I'm not using muxed channels... just normal channels.
I have for now disabled shunning, and the channels seem to be able to
re-connect after a shun situation occurs, but after time I'm still seeing
something wrong with the channel in that the nodes are not able to send
messages to each other successfully, and I have tons of
2008-02-21 21:01:53,572 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
log entries, even while FD is receiving acks on that channel.
this is a degradation problem for me while my servers are running,
not just during deployment. After a while of running, the nodes get
shunned/disconnected some how (often from a slow response from the other
node) and then fail to ever merge back to form the original view of the 2
nodes again.
Now even with shun set to false in both FD and pbcast.GMS on both nodes
for all channels, eventually still the nodes reach the same state of never
re-forming... and I have tons eventual
2008-02-21 21:28:24,857 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
After a while in my application I'll force the channel to close wait a
minute, and reopen, and sometimes this gets the channel working again, but
doesn't always seem to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:38 a.m.
New subject: [JBoss JIRA] Resolved: (JGRP-692) Unable to recover from suspect/merge, with auto-reconnect
[ http://jira.jboss.com/jira/browse/JGRP-692?page=all ]
Bela Ban resolved JGRP-692.
---------------------------
Fix Version/s: 2.6.2
(was: 2.7)
(was: 2.6.3)
Resolution: Duplicate Issue
Duplicate of http://jira.jboss.com/jira/browse/JGRP-699
Unable to recover from suspect/merge, with auto-reconnect
---------------------------------------------------------
Key: JGRP-692
URL: http://jira.jboss.com/jira/browse/JGRP-692
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.1
Environment: JBoss 4.2.2 on Linux 2.4.21-47.0.1.ELsmp i686 using Java HotSpot(TM)
Server VM (build 1.5.0_10-b03, mixed mode)
Reporter: Matt Magoffin
Assigned To: Bela Ban
Fix For: 2.6.2
Attachments: jgroups-logs.tbz2
I'm having an issue with a 2-machine cluster using a TCP stack
based on the tcp.xml from JGroups 2.6.1. On each machine I have 8 separate
channels running, on different ports, with 4 groups in 2 JVM instances.
After some period of time, one machine will fail to respond to a FD ping,
and gets suspected. The machine that failed is not responding in time it
seems from high CPU use, and many of the channels will fail FD around the
same time. The channels are configured with auto-reconnect. My
understanding was that the channel should "heal" itself and eventually
re-form into a new view with the same 2 members in the cluster, which
should apply to this situation because the machine that failed to respond
eventually will respond.
However, the group does not always seem to "heal" (sometimes it does,
sometimes not). Once it stops healing, it never seems to ever do so again,
and I get tons of NAKACK "message X not found in retransmission table"
ERROR logs. The only way to get the channel working agin is to shut down
the channel on both machines and then start them up again.
I'm not using muxed channels... just normal channels.
I have for now disabled shunning, and the channels seem to be able to
re-connect after a shun situation occurs, but after time I'm still seeing
something wrong with the channel in that the nodes are not able to send
messages to each other successfully, and I have tons of
2008-02-21 21:01:53,572 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
log entries, even while FD is receiving acks on that channel.
this is a degradation problem for me while my servers are running,
not just during deployment. After a while of running, the nodes get
shunned/disconnected some how (often from a slow response from the other
node) and then fail to ever merge back to form the original view of the 2
nodes again.
Now even with shun set to false in both FD and pbcast.GMS on both nodes
for all channels, eventually still the nodes reach the same state of never
re-forming... and I have tons eventual
2008-02-21 21:28:24,857 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
After a while in my application I'll force the channel to close wait a
minute, and reopen, and sometimes this gets the channel working again, but
doesn't always seem to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:16 p.m.
New subject: [JBoss JIRA] Commented: (JGRP-692) Unable to recover from suspect/merge, with auto-reconnect
[ http://jira.jboss.com/jira/browse/JGRP-692?page=comments#action_12400510 ]
Matt Magoffin commented on JGRP-692:
------------------------------------
If you look in the attached log files, you'll see:
2008-02-21 21:13:10,947 INFO [org.jgroups.JChannel] JGroups version: 2.6.1
We are definitely using 2.6.1, not 2.4.x. Also, in the FD class on line 261 of the 2.6.1
release, I see:
synchronized(this) {
Address previewNextPingDest =
(Address)getPingDest(pingable_mbrs);
/* We are only interested to stop or restart the monitor
thread iff the current target ping_dest is going
change */
if(log.isDebugEnabled()) log.debug("Recevied Ack. is
invalid (was from: " + hdr.from + "), ");
if ((previewNextPingDest != null && ping_dest !=
null && !previewNextPingDest.equals(ping_dest)) ||
(previewNextPingDest != null && ping_dest
== null) ||
(previewNextPingDest == null && ping_dest
!= null)) {
Do you think this is still a duplicate of 699?
Unable to recover from suspect/merge, with auto-reconnect
---------------------------------------------------------
Key: JGRP-692
URL: http://jira.jboss.com/jira/browse/JGRP-692
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.1
Environment: JBoss 4.2.2 on Linux 2.4.21-47.0.1.ELsmp i686 using Java HotSpot(TM)
Server VM (build 1.5.0_10-b03, mixed mode)
Reporter: Matt Magoffin
Assigned To: Bela Ban
Fix For: 2.6.2
Attachments: jgroups-logs.tbz2
I'm having an issue with a 2-machine cluster using a TCP stack
based on the tcp.xml from JGroups 2.6.1. On each machine I have 8 separate
channels running, on different ports, with 4 groups in 2 JVM instances.
After some period of time, one machine will fail to respond to a FD ping,
and gets suspected. The machine that failed is not responding in time it
seems from high CPU use, and many of the channels will fail FD around the
same time. The channels are configured with auto-reconnect. My
understanding was that the channel should "heal" itself and eventually
re-form into a new view with the same 2 members in the cluster, which
should apply to this situation because the machine that failed to respond
eventually will respond.
However, the group does not always seem to "heal" (sometimes it does,
sometimes not). Once it stops healing, it never seems to ever do so again,
and I get tons of NAKACK "message X not found in retransmission table"
ERROR logs. The only way to get the channel working agin is to shut down
the channel on both machines and then start them up again.
I'm not using muxed channels... just normal channels.
I have for now disabled shunning, and the channels seem to be able to
re-connect after a shun situation occurs, but after time I'm still seeing
something wrong with the channel in that the nodes are not able to send
messages to each other successfully, and I have tons of
2008-02-21 21:01:53,572 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
log entries, even while FD is receiving acks on that channel.
this is a degradation problem for me while my servers are running,
not just during deployment. After a while of running, the nodes get
shunned/disconnected some how (often from a slow response from the other
node) and then fail to ever merge back to form the original view of the 2
nodes again.
Now even with shun set to false in both FD and pbcast.GMS on both nodes
for all channels, eventually still the nodes reach the same state of never
re-forming... and I have tons eventual
2008-02-21 21:28:24,857 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
After a while in my application I'll force the channel to close wait a
minute, and reopen, and sometimes this gets the channel working again, but
doesn't always seem to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:27 a.m.
New subject: [JBoss JIRA] Reopened: (JGRP-692) Unable to recover from suspect/merge, with auto-reconnect
[ http://jira.jboss.com/jira/browse/JGRP-692?page=all ]
Bela Ban reopened JGRP-692:
---------------------------
Unable to recover from suspect/merge, with auto-reconnect
---------------------------------------------------------
Key: JGRP-692
URL: http://jira.jboss.com/jira/browse/JGRP-692
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.1
Environment: JBoss 4.2.2 on Linux 2.4.21-47.0.1.ELsmp i686 using Java HotSpot(TM)
Server VM (build 1.5.0_10-b03, mixed mode)
Reporter: Matt Magoffin
Assigned To: Bela Ban
Fix For: 2.6.3
Attachments: jgroups-logs.tbz2
I'm having an issue with a 2-machine cluster using a TCP stack
based on the tcp.xml from JGroups 2.6.1. On each machine I have 8 separate
channels running, on different ports, with 4 groups in 2 JVM instances.
After some period of time, one machine will fail to respond to a FD ping,
and gets suspected. The machine that failed is not responding in time it
seems from high CPU use, and many of the channels will fail FD around the
same time. The channels are configured with auto-reconnect. My
understanding was that the channel should "heal" itself and eventually
re-form into a new view with the same 2 members in the cluster, which
should apply to this situation because the machine that failed to respond
eventually will respond.
However, the group does not always seem to "heal" (sometimes it does,
sometimes not). Once it stops healing, it never seems to ever do so again,
and I get tons of NAKACK "message X not found in retransmission table"
ERROR logs. The only way to get the channel working agin is to shut down
the channel on both machines and then start them up again.
I'm not using muxed channels... just normal channels.
I have for now disabled shunning, and the channels seem to be able to
re-connect after a shun situation occurs, but after time I'm still seeing
something wrong with the channel in that the nodes are not able to send
messages to each other successfully, and I have tons of
2008-02-21 21:01:53,572 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
log entries, even while FD is receiving acks on that channel.
this is a degradation problem for me while my servers are running,
not just during deployment. After a while of running, the nodes get
shunned/disconnected some how (often from a slow response from the other
node) and then fail to ever merge back to form the original view of the 2
nodes again.
Now even with shun set to false in both FD and pbcast.GMS on both nodes
for all channels, eventually still the nodes reach the same state of never
re-forming... and I have tons eventual
2008-02-21 21:28:24,857 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
After a while in my application I'll force the channel to close wait a
minute, and reopen, and sometimes this gets the channel working again, but
doesn't always seem to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:27 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-692) Unable to recover from suspect/merge, with auto-reconnect
[ http://jira.jboss.com/jira/browse/JGRP-692?page=all ]
Bela Ban updated JGRP-692:
--------------------------
Fix Version/s: 2.6.3
(was: 2.6.2)
Unable to recover from suspect/merge, with auto-reconnect
---------------------------------------------------------
Key: JGRP-692
URL: http://jira.jboss.com/jira/browse/JGRP-692
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.1
Environment: JBoss 4.2.2 on Linux 2.4.21-47.0.1.ELsmp i686 using Java HotSpot(TM)
Server VM (build 1.5.0_10-b03, mixed mode)
Reporter: Matt Magoffin
Assigned To: Bela Ban
Fix For: 2.6.3
Attachments: jgroups-logs.tbz2
I'm having an issue with a 2-machine cluster using a TCP stack
based on the tcp.xml from JGroups 2.6.1. On each machine I have 8 separate
channels running, on different ports, with 4 groups in 2 JVM instances.
After some period of time, one machine will fail to respond to a FD ping,
and gets suspected. The machine that failed is not responding in time it
seems from high CPU use, and many of the channels will fail FD around the
same time. The channels are configured with auto-reconnect. My
understanding was that the channel should "heal" itself and eventually
re-form into a new view with the same 2 members in the cluster, which
should apply to this situation because the machine that failed to respond
eventually will respond.
However, the group does not always seem to "heal" (sometimes it does,
sometimes not). Once it stops healing, it never seems to ever do so again,
and I get tons of NAKACK "message X not found in retransmission table"
ERROR logs. The only way to get the channel working agin is to shut down
the channel on both machines and then start them up again.
I'm not using muxed channels... just normal channels.
I have for now disabled shunning, and the channels seem to be able to
re-connect after a shun situation occurs, but after time I'm still seeing
something wrong with the channel in that the nodes are not able to send
messages to each other successfully, and I have tons of
2008-02-21 21:01:53,572 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
log entries, even while FD is receiving acks on that channel.
this is a degradation problem for me while my servers are running,
not just during deployment. After a while of running, the nodes get
shunned/disconnected some how (often from a slow response from the other
node) and then fail to ever merge back to form the original view of the 2
nodes again.
Now even with shun set to false in both FD and pbcast.GMS on both nodes
for all channels, eventually still the nodes reach the same state of never
re-forming... and I have tons eventual
2008-02-21 21:28:24,857 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
After a while in my application I'll force the channel to close wait a
minute, and reopen, and sometimes this gets the channel working again, but
doesn't always seem to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:21 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-692) Unable to recover from suspect/merge, with auto-reconnect
[ http://jira.jboss.com/jira/browse/JGRP-692?page=comments#action_12402474 ]
Bela Ban commented on JGRP-692:
-------------------------------
Has this been fixed ? If I don't hear from you, I'm going to close this issue
Unable to recover from suspect/merge, with auto-reconnect
---------------------------------------------------------
Key: JGRP-692
URL: http://jira.jboss.com/jira/browse/JGRP-692
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.1
Environment: JBoss 4.2.2 on Linux 2.4.21-47.0.1.ELsmp i686 using Java HotSpot(TM)
Server VM (build 1.5.0_10-b03, mixed mode)
Reporter: Matt Magoffin
Assigned To: Bela Ban
Fix For: 2.6.3
Attachments: jgroups-logs.tbz2
I'm having an issue with a 2-machine cluster using a TCP stack
based on the tcp.xml from JGroups 2.6.1. On each machine I have 8 separate
channels running, on different ports, with 4 groups in 2 JVM instances.
After some period of time, one machine will fail to respond to a FD ping,
and gets suspected. The machine that failed is not responding in time it
seems from high CPU use, and many of the channels will fail FD around the
same time. The channels are configured with auto-reconnect. My
understanding was that the channel should "heal" itself and eventually
re-form into a new view with the same 2 members in the cluster, which
should apply to this situation because the machine that failed to respond
eventually will respond.
However, the group does not always seem to "heal" (sometimes it does,
sometimes not). Once it stops healing, it never seems to ever do so again,
and I get tons of NAKACK "message X not found in retransmission table"
ERROR logs. The only way to get the channel working agin is to shut down
the channel on both machines and then start them up again.
I'm not using muxed channels... just normal channels.
I have for now disabled shunning, and the channels seem to be able to
re-connect after a shun situation occurs, but after time I'm still seeing
something wrong with the channel in that the nodes are not able to send
messages to each other successfully, and I have tons of
2008-02-21 21:01:53,572 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
log entries, even while FD is receiving acks on that channel.
this is a degradation problem for me while my servers are running,
not just during deployment. After a while of running, the nodes get
shunned/disconnected some how (often from a slow response from the other
node) and then fail to ever merge back to form the original view of the 2
nodes again.
Now even with shun set to false in both FD and pbcast.GMS on both nodes
for all channels, eventually still the nodes reach the same state of never
re-forming... and I have tons eventual
2008-02-21 21:28:24,857 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
After a while in my application I'll force the channel to close wait a
minute, and reopen, and sometimes this gets the channel working again, but
doesn't always seem to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:32 p.m.
New subject: [JBoss JIRA] Commented: (JGRP-692) Unable to recover from suspect/merge, with auto-reconnect
[ http://jira.jboss.com/jira/browse/JGRP-692?page=comments#action_12402539 ]
Matt Magoffin commented on JGRP-692:
------------------------------------
It says the fix release is 2.6.3, but I don't see that available. Since I can only
reproduce this on a production environment, I can't use a CVS based release. When is
2.6.3 getting released?
Unable to recover from suspect/merge, with auto-reconnect
---------------------------------------------------------
Key: JGRP-692
URL: http://jira.jboss.com/jira/browse/JGRP-692
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.1
Environment: JBoss 4.2.2 on Linux 2.4.21-47.0.1.ELsmp i686 using Java HotSpot(TM)
Server VM (build 1.5.0_10-b03, mixed mode)
Reporter: Matt Magoffin
Assigned To: Bela Ban
Fix For: 2.6.3
Attachments: jgroups-logs.tbz2
I'm having an issue with a 2-machine cluster using a TCP stack
based on the tcp.xml from JGroups 2.6.1. On each machine I have 8 separate
channels running, on different ports, with 4 groups in 2 JVM instances.
After some period of time, one machine will fail to respond to a FD ping,
and gets suspected. The machine that failed is not responding in time it
seems from high CPU use, and many of the channels will fail FD around the
same time. The channels are configured with auto-reconnect. My
understanding was that the channel should "heal" itself and eventually
re-form into a new view with the same 2 members in the cluster, which
should apply to this situation because the machine that failed to respond
eventually will respond.
However, the group does not always seem to "heal" (sometimes it does,
sometimes not). Once it stops healing, it never seems to ever do so again,
and I get tons of NAKACK "message X not found in retransmission table"
ERROR logs. The only way to get the channel working agin is to shut down
the channel on both machines and then start them up again.
I'm not using muxed channels... just normal channels.
I have for now disabled shunning, and the channels seem to be able to
re-connect after a shun situation occurs, but after time I'm still seeing
something wrong with the channel in that the nodes are not able to send
messages to each other successfully, and I have tons of
2008-02-21 21:01:53,572 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
log entries, even while FD is receiving acks on that channel.
this is a degradation problem for me while my servers are running,
not just during deployment. After a while of running, the nodes get
shunned/disconnected some how (often from a slow response from the other
node) and then fail to ever merge back to form the original view of the 2
nodes again.
Now even with shun set to false in both FD and pbcast.GMS on both nodes
for all channels, eventually still the nodes reach the same state of never
re-forming... and I have tons eventual
2008-02-21 21:28:24,857 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
After a while in my application I'll force the channel to close wait a
minute, and reopen, and sometimes this gets the channel working again, but
doesn't always seem to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:26 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-692) Unable to recover from suspect/merge, with auto-reconnect
[ http://jira.jboss.com/jira/browse/JGRP-692?page=comments#action_12402639 ]
Bela Ban commented on JGRP-692:
-------------------------------
Yes, 2.6.3 has not yet been released. I wanted to wait for *this* issue to get fixed. No
chance you could test this (CVS checkout) in staging at least ?
Otherwise, if 2.6.3 won't fix your issue, we'd have to wait until 2.6.4...
I don't know when I'm going to cut 2.6.3 yet. The 2.6 branch is currently the
stable branch for JBoss AS 5, and we're back porting all the important bug fixes from
trunk... It's going to be a couple of weeks at least
Unable to recover from suspect/merge, with auto-reconnect
---------------------------------------------------------
Key: JGRP-692
URL: http://jira.jboss.com/jira/browse/JGRP-692
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.1
Environment: JBoss 4.2.2 on Linux 2.4.21-47.0.1.ELsmp i686 using Java HotSpot(TM)
Server VM (build 1.5.0_10-b03, mixed mode)
Reporter: Matt Magoffin
Assigned To: Bela Ban
Fix For: 2.6.3
Attachments: jgroups-logs.tbz2
I'm having an issue with a 2-machine cluster using a TCP stack
based on the tcp.xml from JGroups 2.6.1. On each machine I have 8 separate
channels running, on different ports, with 4 groups in 2 JVM instances.
After some period of time, one machine will fail to respond to a FD ping,
and gets suspected. The machine that failed is not responding in time it
seems from high CPU use, and many of the channels will fail FD around the
same time. The channels are configured with auto-reconnect. My
understanding was that the channel should "heal" itself and eventually
re-form into a new view with the same 2 members in the cluster, which
should apply to this situation because the machine that failed to respond
eventually will respond.
However, the group does not always seem to "heal" (sometimes it does,
sometimes not). Once it stops healing, it never seems to ever do so again,
and I get tons of NAKACK "message X not found in retransmission table"
ERROR logs. The only way to get the channel working agin is to shut down
the channel on both machines and then start them up again.
I'm not using muxed channels... just normal channels.
I have for now disabled shunning, and the channels seem to be able to
re-connect after a shun situation occurs, but after time I'm still seeing
something wrong with the channel in that the nodes are not able to send
messages to each other successfully, and I have tons of
2008-02-21 21:01:53,572 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
log entries, even while FD is receiving acks on that channel.
this is a degradation problem for me while my servers are running,
not just during deployment. After a while of running, the nodes get
shunned/disconnected some how (often from a slow response from the other
node) and then fail to ever merge back to form the original view of the 2
nodes again.
Now even with shun set to false in both FD and pbcast.GMS on both nodes
for all channels, eventually still the nodes reach the same state of never
re-forming... and I have tons eventual
2008-02-21 21:28:24,857 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
After a while in my application I'll force the channel to close wait a
minute, and reopen, and sometimes this gets the channel working again, but
doesn't always seem to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:20 p.m.
New subject: [JBoss JIRA] Commented: (JGRP-692) Unable to recover from suspect/merge, with auto-reconnect
[ http://jira.jboss.com/jira/browse/JGRP-692?page=comments#action_12402794 ]
Matt Magoffin commented on JGRP-692:
------------------------------------
I can see about getting this to our staging environment. The problem is we don't have
enough traffic on our staging environment to expose this issue... we've only seen it
on our production environment.
By CVS checkout, do you mean the HEAD branch?
Unable to recover from suspect/merge, with auto-reconnect
---------------------------------------------------------
Key: JGRP-692
URL: http://jira.jboss.com/jira/browse/JGRP-692
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.1
Environment: JBoss 4.2.2 on Linux 2.4.21-47.0.1.ELsmp i686 using Java HotSpot(TM)
Server VM (build 1.5.0_10-b03, mixed mode)
Reporter: Matt Magoffin
Assigned To: Bela Ban
Fix For: 2.6.3
Attachments: jgroups-logs.tbz2
I'm having an issue with a 2-machine cluster using a TCP stack
based on the tcp.xml from JGroups 2.6.1. On each machine I have 8 separate
channels running, on different ports, with 4 groups in 2 JVM instances.
After some period of time, one machine will fail to respond to a FD ping,
and gets suspected. The machine that failed is not responding in time it
seems from high CPU use, and many of the channels will fail FD around the
same time. The channels are configured with auto-reconnect. My
understanding was that the channel should "heal" itself and eventually
re-form into a new view with the same 2 members in the cluster, which
should apply to this situation because the machine that failed to respond
eventually will respond.
However, the group does not always seem to "heal" (sometimes it does,
sometimes not). Once it stops healing, it never seems to ever do so again,
and I get tons of NAKACK "message X not found in retransmission table"
ERROR logs. The only way to get the channel working agin is to shut down
the channel on both machines and then start them up again.
I'm not using muxed channels... just normal channels.
I have for now disabled shunning, and the channels seem to be able to
re-connect after a shun situation occurs, but after time I'm still seeing
something wrong with the channel in that the nodes are not able to send
messages to each other successfully, and I have tons of
2008-02-21 21:01:53,572 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
log entries, even while FD is receiving acks on that channel.
this is a degradation problem for me while my servers are running,
not just during deployment. After a while of running, the nodes get
shunned/disconnected some how (often from a slow response from the other
node) and then fail to ever merge back to form the original view of the 2
nodes again.
Now even with shun set to false in both FD and pbcast.GMS on both nodes
for all channels, eventually still the nodes reach the same state of never
re-forming... and I have tons eventual
2008-02-21 21:28:24,857 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
After a while in my application I'll force the channel to close wait a
minute, and reopen, and sometimes this gets the channel working again, but
doesn't always seem to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:38 p.m.
New subject: [JBoss JIRA] Commented: (JGRP-692) Unable to recover from suspect/merge, with auto-reconnect
[ http://jira.jboss.com/jira/browse/JGRP-692?page=comments#action_12403276 ]
Matt Magoffin commented on JGRP-692:
------------------------------------
I tried out the latest CVS code on our staging environment. It built as version
2.7.0.Beta1. I didn't see any problems there (but this was expected). Since our
production environment is effectively broken (in relation to JGroups) I was able to deploy
the JAR also there today. I'll let you know how it performs, we should know in a day
or so if the problem appears to be fixed.
Unable to recover from suspect/merge, with auto-reconnect
---------------------------------------------------------
Key: JGRP-692
URL: http://jira.jboss.com/jira/browse/JGRP-692
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.1
Environment: JBoss 4.2.2 on Linux 2.4.21-47.0.1.ELsmp i686 using Java HotSpot(TM)
Server VM (build 1.5.0_10-b03, mixed mode)
Reporter: Matt Magoffin
Assigned To: Bela Ban
Fix For: 2.6.3
Attachments: jgroups-logs.tbz2
I'm having an issue with a 2-machine cluster using a TCP stack
based on the tcp.xml from JGroups 2.6.1. On each machine I have 8 separate
channels running, on different ports, with 4 groups in 2 JVM instances.
After some period of time, one machine will fail to respond to a FD ping,
and gets suspected. The machine that failed is not responding in time it
seems from high CPU use, and many of the channels will fail FD around the
same time. The channels are configured with auto-reconnect. My
understanding was that the channel should "heal" itself and eventually
re-form into a new view with the same 2 members in the cluster, which
should apply to this situation because the machine that failed to respond
eventually will respond.
However, the group does not always seem to "heal" (sometimes it does,
sometimes not). Once it stops healing, it never seems to ever do so again,
and I get tons of NAKACK "message X not found in retransmission table"
ERROR logs. The only way to get the channel working agin is to shut down
the channel on both machines and then start them up again.
I'm not using muxed channels... just normal channels.
I have for now disabled shunning, and the channels seem to be able to
re-connect after a shun situation occurs, but after time I'm still seeing
something wrong with the channel in that the nodes are not able to send
messages to each other successfully, and I have tons of
2008-02-21 21:01:53,572 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
log entries, even while FD is receiving acks on that channel.
this is a degradation problem for me while my servers are running,
not just during deployment. After a while of running, the nodes get
shunned/disconnected some how (often from a slow response from the other
node) and then fail to ever merge back to form the original view of the 2
nodes again.
Now even with shun set to false in both FD and pbcast.GMS on both nodes
for all channels, eventually still the nodes reach the same state of never
re-forming... and I have tons eventual
2008-02-21 21:28:24,857 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
After a while in my application I'll force the channel to close wait a
minute, and reopen, and sometimes this gets the channel working again, but
doesn't always seem to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:01 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-692) Unable to recover from suspect/merge, with auto-reconnect
[ http://jira.jboss.com/jira/browse/JGRP-692?page=comments#action_12403295 ]
Matt Magoffin commented on JGRP-692:
------------------------------------
So on production so far, this is looking promising. I've seen successful merge events
occur after socket exceptions:
2008-03-18 00:46:44,508 ERROR [org.jgroups.blocks.ConnectionTable] failed sending data to
172.16.172.234:19282: java.net.SocketException: Broken pipe
2008-03-18 00:46:44,519 ERROR [org.jgroups.blocks.ConnectionTable] failed sending data to
172.16.172.234:19282: java.net.SocketException: Socket closed
...
2008-03-18 00:46:47,489 DEBUG [org.jgroups.protocols.MERGE2]
initial_mbrs=[[own_addr=172.16.172.234:19282, coord_addr=172.16.172.234:19282,
is_server=true], [own_addr=172.16.172.233
:19282, coord_addr=172.16.172.233:19282, is_server=true]]
2008-03-18 00:46:47,489 DEBUG [org.jgroups.protocols.MERGE2] found multiple coordinators:
[172.16.172.234:19282, 172.16.172.233:19282]; sending up MERGE event
2008-03-18 00:46:47,499 DEBUG [org.jgroups.protocols.MERGE2]
initial_mbrs=[[own_addr=172.16.172.234:19182, coord_addr=172.16.172.233:19182,
is_server=true], [own_addr=172.16.172.233
:19182, coord_addr=172.16.172.233:19182, is_server=true]]
2008-03-18 00:46:47,499 DEBUG [org.jgroups.protocols.MERGE2]
initial_mbrs=[[own_addr=172.16.172.234:19183, coord_addr=172.16.172.233:19183,
is_server=true], [own_addr=172.16.172.233
:19183, coord_addr=172.16.172.233:19183, is_server=true]]
2008-03-18 00:46:47,499 DEBUG [org.jgroups.protocols.MERGE2]
initial_mbrs=[[own_addr=172.16.172.234:19082, coord_addr=172.16.172.233:19082,
is_server=true], [own_addr=172.16.172.233
:19082, coord_addr=172.16.172.233:19082, is_server=true]]
2008-03-18 00:46:47,499 DEBUG [org.jgroups.protocols.MERGE2]
initial_mbrs=[[own_addr=172.16.172.234:19382, coord_addr=172.16.172.234:19382,
is_server=true], [own_addr=172.16.172.233
:19382, coord_addr=172.16.172.233:19382, is_server=true]]
2008-03-18 00:46:47,499 DEBUG [org.jgroups.protocols.MERGE2] found multiple coordinators:
[172.16.172.234:19382, 172.16.172.233:19382]; sending up MERGE event
2008-03-18 00:46:47,499 DEBUG [org.jgroups.protocols.MERGE2]
initial_mbrs=[[own_addr=172.16.172.234:19083, coord_addr=172.16.172.233:19083,
is_server=true], [own_addr=172.16.172.233
:19083, coord_addr=172.16.172.233:19083, is_server=true]]
2008-03-18 00:46:47,500 DEBUG [org.jgroups.protocols.MERGE2]
initial_mbrs=[[own_addr=172.16.172.234:19383, coord_addr=172.16.172.233:19383,
is_server=true], [own_addr=172.16.172.233
:19383, coord_addr=172.16.172.233:19383, is_server=true]]
2008-03-18 00:46:47,556 DEBUG [org.jgroups.protocols.pbcast.GMS] merge task started,
coordinators are [172.16.172.234:19282, 172.16.172.233:19282]
2008-03-18 00:46:47,556 DEBUG [org.jgroups.protocols.pbcast.GMS] sending MERGE_REQ to
[172.16.172.234:19282, 172.16.172.233:19282]
2008-03-18 00:46:47,557 DEBUG [org.jgroups.protocols.pbcast.GMS] waiting 10000 msecs for
merge responses
2008-03-18 00:46:47,557 DEBUG [org.jgroups.protocols.pbcast.STABLE] suspending message
garbage collection
2008-03-18 00:46:47,557 DEBUG [org.jgroups.protocols.pbcast.STABLE] resume task started,
max_suspend_time=200000
2008-03-18 00:46:47,557 DEBUG [org.jgroups.protocols.pbcast.GMS]
sender=172.16.172.233:19282, merge_id=[172.16.172.233:19282|1205819207556]
2008-03-18 00:46:47,557 DEBUG [org.jgroups.protocols.pbcast.GMS]
response=GmsHeader[MERGE_RSP]: view=[172.16.172.233:19282|3] [172.16.172.233:19282,
172.16.172.234:19282], digest=17
2.16.172.233:19282: [8391 : 8449 (8449)], 172.16.172.234:19282: [1043 : 1047 (1047)],
merge_rejected=false, merge_id=[172.16.172.233:19282|1205819207556]
2008-03-18 00:46:47,557 DEBUG [org.jgroups.protocols.pbcast.GMS] num_rsps_expected=2,
actual responses=1
2008-03-18 00:46:47,557 DEBUG [org.jgroups.protocols.pbcast.GMS] waiting 10000 msecs for
merge responses
2008-03-18 00:46:47,559 DEBUG [org.jgroups.protocols.pbcast.GMS] merge task started,
coordinators are [172.16.172.234:19382, 172.16.172.233:19382]
2008-03-18 00:46:47,559 DEBUG [org.jgroups.protocols.pbcast.GMS] sending MERGE_REQ to
[172.16.172.234:19382, 172.16.172.233:19382]
2008-03-18 00:46:47,559 DEBUG [org.jgroups.protocols.pbcast.GMS] waiting 10000 msecs for
merge responses
2008-03-18 00:46:47,559 DEBUG [org.jgroups.protocols.pbcast.STABLE] suspending message
garbage collection
2008-03-18 00:46:47,559 DEBUG [org.jgroups.protocols.pbcast.STABLE] resume task started,
max_suspend_time=200000
2008-03-18 00:46:47,559 DEBUG [org.jgroups.protocols.pbcast.GMS]
sender=172.16.172.233:19382, merge_id=[172.16.172.233:19382|1205819207559]
2008-03-18 00:46:47,559 DEBUG [org.jgroups.protocols.pbcast.GMS]
response=GmsHeader[MERGE_RSP]: view=[172.16.172.233:19382|5] [172.16.172.233:19382,
172.16.172.234:19382], digest=17
2.16.172.233:19382: [446 : 448 (448)], 172.16.172.234:19382: [1916 : 1933 (1933)],
merge_rejected=false, merge_id=[172.16.172.233:19382|1205819207559]
2008-03-18 00:46:47,560 DEBUG [org.jgroups.protocols.pbcast.GMS] num_rsps_expected=2,
actual responses=1
2008-03-18 00:46:47,560 DEBUG [org.jgroups.protocols.pbcast.GMS] waiting 9999 msecs for
merge responses
2008-03-18 00:46:47,592 DEBUG [org.jgroups.protocols.pbcast.GMS] num_rsps_expected=2,
actual responses=2
2008-03-18 00:46:47,593 DEBUG [org.jgroups.protocols.pbcast.GMS] num_rsps_expected=2,
actual responses=2
2008-03-18 00:46:47,593 DEBUG [org.jgroups.protocols.pbcast.GMS] merge data is
sender=172.16.172.233:19282, view=[172.16.172.233:19282|3] [172.16.172.233:19282,
172.16.172.234:19282
], digest=172.16.172.233:19282: [8391 : 8449 (8449)], 172.16.172.234:19282: [1043 : 1047
(1047)]
2008-03-18 00:46:47,593 DEBUG [org.jgroups.protocols.pbcast.GMS] merge data is
sender=172.16.172.233:19382, view=[172.16.172.233:19382|5] [172.16.172.233:19382,
172.16.172.234:19382
], digest=172.16.172.233:19382: [446 : 448 (448)], 172.16.172.234:19382: [1916 : 1933
(1933)]
2008-03-18 00:46:47,593 DEBUG [org.jgroups.protocols.pbcast.GMS] merge data is
sender=172.16.172.234:19282, view=[172.16.172.234:19282|4] [172.16.172.234:19282],
digest=172.16.172.2
34:19282: [1043 : 1048 (1048)]
2008-03-18 00:46:47,593 DEBUG [org.jgroups.protocols.pbcast.GMS] merge data is
sender=172.16.172.234:19382, view=[172.16.172.234:19382|6] [172.16.172.234:19382],
digest=172.16.172.2
34:19382: [1916 : 1934 (1934)]
2008-03-18 00:46:47,593 DEBUG [org.jgroups.protocols.pbcast.GMS] new merged view will be
MergeView::[172.16.172.233:19282|5] [172.16.172.233:19282, 172.16.172.234:19282],
subgroups=
[[172.16.172.233:19282|3] [172.16.172.233:19282, 172.16.172.234:19282],
[172.16.172.234:19282|4] [172.16.172.234:19282]]
2008-03-18 00:46:47,593 DEBUG [org.jgroups.protocols.pbcast.GMS] new merged view will be
MergeView::[172.16.172.233:19382|7] [172.16.172.233:19382, 172.16.172.234:19382],
subgroups=
[[172.16.172.233:19382|5] [172.16.172.233:19382, 172.16.172.234:19382],
[172.16.172.234:19382|6] [172.16.172.234:19382]]
2008-03-18 00:46:47,593 DEBUG [org.jgroups.protocols.pbcast.GMS] consolidated
digest=172.16.172.233:19282: [8391 : 8449 (8449)], 172.16.172.234:19282: [1043 : 1048
(1048)]
2008-03-18 00:46:47,593 DEBUG [org.jgroups.protocols.pbcast.GMS] consolidated
digest=172.16.172.233:19382: [446 : 448 (448)], 172.16.172.234:19382: [1916 : 1934
(1934)]
2008-03-18 00:46:47,594 DEBUG [org.jgroups.protocols.pbcast.STABLE] resuming message
garbage collection
2008-03-18 00:46:47,594 DEBUG [org.jgroups.protocols.pbcast.STABLE] resuming message
garbage collection
2008-03-18 00:46:47,594 DEBUG [org.jgroups.protocols.pbcast.GMS] merge task terminated
2008-03-18 00:46:47,594 DEBUG [org.jgroups.protocols.pbcast.GMS] merge task terminated
2008-03-18 00:46:47,649 DEBUG [org.jgroups.protocols.pbcast.GMS]
view=MergeView::[172.16.172.233:19282|5] [172.16.172.233:19282, 172.16.172.234:19282],
subgroups=[[172.16.172.233:19
282|3] [172.16.172.233:19282, 172.16.172.234:19282], [172.16.172.234:19282|4]
[172.16.172.234:19282]], digest=172.16.172.233:19282: [8391 : 8449 (8449)],
172.16.172.234:19282: [1043
: 1048 (1048)]
2008-03-18 00:46:47,649 DEBUG [org.jgroups.protocols.pbcast.GMS]
view=MergeView::[172.16.172.233:19382|7] [172.16.172.233:19382, 172.16.172.234:19382],
subgroups=[[172.16.172.233:19
382|5] [172.16.172.233:19382, 172.16.172.234:19382], [172.16.172.234:19382|6]
[172.16.172.234:19382]], digest=172.16.172.233:19382: [446 : 448 (448)],
172.16.172.234:19382: [1916 :
1934 (1934)]
2008-03-18 00:46:47,649 DEBUG [org.jgroups.protocols.pbcast.NAKACK] existing digest:
172.16.172.233:19282: [8391 : 8450 (8450)], 172.16.172.234:19282: [1043 : 1047 (1047)]
new digest: 172.16.172.233:19282: [8391 : 8449 (8449)], 172.16.172.234:19282: [1043
: 1048 (1048)]
resulting digest: 172.16.172.233:19282: [8391 : 8450 (8450)], 172.16.172.234:19282: [1043
: 1048 (1048)]
2008-03-18 00:46:47,649 DEBUG [org.jgroups.protocols.pbcast.GMS]
[local_addr=172.16.172.233:19282] view is MergeView::[172.16.172.233:19282|5]
[172.16.172.233:19282, 172.16.172.234:
19282], subgroups=[[172.16.172.233:19282|3] [172.16.172.233:19282, 172.16.172.234:19282],
[172.16.172.234:19282|4] [172.16.172.234:19282]]
2008-03-18 00:46:47,649 DEBUG [org.jgroups.protocols.pbcast.NAKACK] existing digest:
172.16.172.233:19382: [446 : 449 (449)], 172.16.172.234:19382: [1916 : 1933 (1933)]
new digest: 172.16.172.233:19382: [446 : 448 (448)], 172.16.172.234:19382: [1916 :
1934 (1934)]
resulting digest: 172.16.172.233:19382: [446 : 449 (449)], 172.16.172.234:19382: [1916 :
1934 (1934)]
2008-03-18 00:46:47,650 DEBUG [org.jgroups.protocols.pbcast.GMS]
[local_addr=172.16.172.233:19382] view is MergeView::[172.16.172.233:19382|7]
[172.16.172.233:19382, 172.16.172.234:
19382], subgroups=[[172.16.172.233:19382|5] [172.16.172.233:19382, 172.16.172.234:19382],
[172.16.172.234:19382|6] [172.16.172.234:19382]]
And afterwards, the channel seems to get back to normal.
Unable to recover from suspect/merge, with auto-reconnect
---------------------------------------------------------
Key: JGRP-692
URL: http://jira.jboss.com/jira/browse/JGRP-692
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.1
Environment: JBoss 4.2.2 on Linux 2.4.21-47.0.1.ELsmp i686 using Java HotSpot(TM)
Server VM (build 1.5.0_10-b03, mixed mode)
Reporter: Matt Magoffin
Assigned To: Bela Ban
Fix For: 2.6.3
Attachments: jgroups-logs.tbz2
I'm having an issue with a 2-machine cluster using a TCP stack
based on the tcp.xml from JGroups 2.6.1. On each machine I have 8 separate
channels running, on different ports, with 4 groups in 2 JVM instances.
After some period of time, one machine will fail to respond to a FD ping,
and gets suspected. The machine that failed is not responding in time it
seems from high CPU use, and many of the channels will fail FD around the
same time. The channels are configured with auto-reconnect. My
understanding was that the channel should "heal" itself and eventually
re-form into a new view with the same 2 members in the cluster, which
should apply to this situation because the machine that failed to respond
eventually will respond.
However, the group does not always seem to "heal" (sometimes it does,
sometimes not). Once it stops healing, it never seems to ever do so again,
and I get tons of NAKACK "message X not found in retransmission table"
ERROR logs. The only way to get the channel working agin is to shut down
the channel on both machines and then start them up again.
I'm not using muxed channels... just normal channels.
I have for now disabled shunning, and the channels seem to be able to
re-connect after a shun situation occurs, but after time I'm still seeing
something wrong with the channel in that the nodes are not able to send
messages to each other successfully, and I have tons of
2008-02-21 21:01:53,572 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
log entries, even while FD is receiving acks on that channel.
this is a degradation problem for me while my servers are running,
not just during deployment. After a while of running, the nodes get
shunned/disconnected some how (often from a slow response from the other
node) and then fail to ever merge back to form the original view of the 2
nodes again.
Now even with shun set to false in both FD and pbcast.GMS on both nodes
for all channels, eventually still the nodes reach the same state of never
re-forming... and I have tons eventual
2008-02-21 21:28:24,857 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
After a while in my application I'll force the channel to close wait a
minute, and reopen, and sometimes this gets the channel working again, but
doesn't always seem to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:38 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-692) Unable to recover from suspect/merge, with auto-reconnect
[ http://jira.jboss.com/jira/browse/JGRP-692?page=comments#action_12403297 ]
Bela Ban commented on JGRP-692:
-------------------------------
Good to know it works, but I was actually referring to 2.6.3 (the 2.6 branch) *not* 2.7 !
2.7 won't get released for some time !
The branch name for 2.6 is Branch_JGroups_2_6, see
http://wiki.jboss.org/wiki/Wiki.jsp?page=Branches.
Unable to recover from suspect/merge, with auto-reconnect
---------------------------------------------------------
Key: JGRP-692
URL: http://jira.jboss.com/jira/browse/JGRP-692
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.1
Environment: JBoss 4.2.2 on Linux 2.4.21-47.0.1.ELsmp i686 using Java HotSpot(TM)
Server VM (build 1.5.0_10-b03, mixed mode)
Reporter: Matt Magoffin
Assigned To: Bela Ban
Fix For: 2.6.3
Attachments: jgroups-logs.tbz2
I'm having an issue with a 2-machine cluster using a TCP stack
based on the tcp.xml from JGroups 2.6.1. On each machine I have 8 separate
channels running, on different ports, with 4 groups in 2 JVM instances.
After some period of time, one machine will fail to respond to a FD ping,
and gets suspected. The machine that failed is not responding in time it
seems from high CPU use, and many of the channels will fail FD around the
same time. The channels are configured with auto-reconnect. My
understanding was that the channel should "heal" itself and eventually
re-form into a new view with the same 2 members in the cluster, which
should apply to this situation because the machine that failed to respond
eventually will respond.
However, the group does not always seem to "heal" (sometimes it does,
sometimes not). Once it stops healing, it never seems to ever do so again,
and I get tons of NAKACK "message X not found in retransmission table"
ERROR logs. The only way to get the channel working agin is to shut down
the channel on both machines and then start them up again.
I'm not using muxed channels... just normal channels.
I have for now disabled shunning, and the channels seem to be able to
re-connect after a shun situation occurs, but after time I'm still seeing
something wrong with the channel in that the nodes are not able to send
messages to each other successfully, and I have tons of
2008-02-21 21:01:53,572 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
log entries, even while FD is receiving acks on that channel.
this is a degradation problem for me while my servers are running,
not just during deployment. After a while of running, the nodes get
shunned/disconnected some how (often from a slow response from the other
node) and then fail to ever merge back to form the original view of the 2
nodes again.
Now even with shun set to false in both FD and pbcast.GMS on both nodes
for all channels, eventually still the nodes reach the same state of never
re-forming... and I have tons eventual
2008-02-21 21:28:24,857 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
After a while in my application I'll force the channel to close wait a
minute, and reopen, and sometimes this gets the channel working again, but
doesn't always seem to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:16 p.m.
New subject: [JBoss JIRA] Commented: (JGRP-692) Unable to recover from suspect/merge, with auto-reconnect
[ http://jira.jboss.com/jira/browse/JGRP-692?page=comments#action_12403494 ]
Matt Magoffin commented on JGRP-692:
------------------------------------
I see, I guess that is why I asked in a previous comment about which branch to use... I
will try with the Branch_JGroups_2_6 and let you know.
Unable to recover from suspect/merge, with auto-reconnect
---------------------------------------------------------
Key: JGRP-692
URL: http://jira.jboss.com/jira/browse/JGRP-692
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.1
Environment: JBoss 4.2.2 on Linux 2.4.21-47.0.1.ELsmp i686 using Java HotSpot(TM)
Server VM (build 1.5.0_10-b03, mixed mode)
Reporter: Matt Magoffin
Assigned To: Bela Ban
Fix For: 2.6.3
Attachments: jgroups-logs.tbz2
I'm having an issue with a 2-machine cluster using a TCP stack
based on the tcp.xml from JGroups 2.6.1. On each machine I have 8 separate
channels running, on different ports, with 4 groups in 2 JVM instances.
After some period of time, one machine will fail to respond to a FD ping,
and gets suspected. The machine that failed is not responding in time it
seems from high CPU use, and many of the channels will fail FD around the
same time. The channels are configured with auto-reconnect. My
understanding was that the channel should "heal" itself and eventually
re-form into a new view with the same 2 members in the cluster, which
should apply to this situation because the machine that failed to respond
eventually will respond.
However, the group does not always seem to "heal" (sometimes it does,
sometimes not). Once it stops healing, it never seems to ever do so again,
and I get tons of NAKACK "message X not found in retransmission table"
ERROR logs. The only way to get the channel working agin is to shut down
the channel on both machines and then start them up again.
I'm not using muxed channels... just normal channels.
I have for now disabled shunning, and the channels seem to be able to
re-connect after a shun situation occurs, but after time I'm still seeing
something wrong with the channel in that the nodes are not able to send
messages to each other successfully, and I have tons of
2008-02-21 21:01:53,572 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
log entries, even while FD is receiving acks on that channel.
this is a degradation problem for me while my servers are running,
not just during deployment. After a while of running, the nodes get
shunned/disconnected some how (often from a slow response from the other
node) and then fail to ever merge back to form the original view of the 2
nodes again.
Now even with shun set to false in both FD and pbcast.GMS on both nodes
for all channels, eventually still the nodes reach the same state of never
re-forming... and I have tons eventual
2008-02-21 21:28:24,857 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
After a while in my application I'll force the channel to close wait a
minute, and reopen, and sometimes this gets the channel working again, but
doesn't always seem to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:20 p.m.
New subject: [JBoss JIRA] Commented: (JGRP-692) Unable to recover from suspect/merge, with auto-reconnect
[ http://jira.jboss.com/jira/browse/JGRP-692?page=comments#action_12403678 ]
Matt Magoffin commented on JGRP-692:
------------------------------------
I deployed the 2.6 branch, which built as version 2.6.3.GA to our production environment
yesterday, and so far it has been performing without problems. However I have not seen any
socket resets from the org.jgroups.blocks.ConnectionTable class in the logs yet, which
seemed to be the original event that exposed the problem. I'll continue monitoring.
Unable to recover from suspect/merge, with auto-reconnect
---------------------------------------------------------
Key: JGRP-692
URL: http://jira.jboss.com/jira/browse/JGRP-692
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.1
Environment: JBoss 4.2.2 on Linux 2.4.21-47.0.1.ELsmp i686 using Java HotSpot(TM)
Server VM (build 1.5.0_10-b03, mixed mode)
Reporter: Matt Magoffin
Assigned To: Bela Ban
Fix For: 2.6.3
Attachments: jgroups-logs.tbz2
I'm having an issue with a 2-machine cluster using a TCP stack
based on the tcp.xml from JGroups 2.6.1. On each machine I have 8 separate
channels running, on different ports, with 4 groups in 2 JVM instances.
After some period of time, one machine will fail to respond to a FD ping,
and gets suspected. The machine that failed is not responding in time it
seems from high CPU use, and many of the channels will fail FD around the
same time. The channels are configured with auto-reconnect. My
understanding was that the channel should "heal" itself and eventually
re-form into a new view with the same 2 members in the cluster, which
should apply to this situation because the machine that failed to respond
eventually will respond.
However, the group does not always seem to "heal" (sometimes it does,
sometimes not). Once it stops healing, it never seems to ever do so again,
and I get tons of NAKACK "message X not found in retransmission table"
ERROR logs. The only way to get the channel working agin is to shut down
the channel on both machines and then start them up again.
I'm not using muxed channels... just normal channels.
I have for now disabled shunning, and the channels seem to be able to
re-connect after a shun situation occurs, but after time I'm still seeing
something wrong with the channel in that the nodes are not able to send
messages to each other successfully, and I have tons of
2008-02-21 21:01:53,572 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
log entries, even while FD is receiving acks on that channel.
this is a degradation problem for me while my servers are running,
not just during deployment. After a while of running, the nodes get
shunned/disconnected some how (often from a slow response from the other
node) and then fail to ever merge back to form the original view of the 2
nodes again.
Now even with shun set to false in both FD and pbcast.GMS on both nodes
for all channels, eventually still the nodes reach the same state of never
re-forming... and I have tons eventual
2008-02-21 21:28:24,857 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
After a while in my application I'll force the channel to close wait a
minute, and reopen, and sometimes this gets the channel working again, but
doesn't always seem to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:23 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-692) Unable to recover from suspect/merge, with auto-reconnect
[ http://jira.jboss.com/jira/browse/JGRP-692?page=comments#action_12403759 ]
Matt Magoffin commented on JGRP-692:
------------------------------------
On our production environment (using 2.6.3.GA build) we just had a couple of channels get
disconnected and they have not yet been able to re-connect. I'll try to investigate in
the logs.
Unable to recover from suspect/merge, with auto-reconnect
---------------------------------------------------------
Key: JGRP-692
URL: http://jira.jboss.com/jira/browse/JGRP-692
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.1
Environment: JBoss 4.2.2 on Linux 2.4.21-47.0.1.ELsmp i686 using Java HotSpot(TM)
Server VM (build 1.5.0_10-b03, mixed mode)
Reporter: Matt Magoffin
Assigned To: Bela Ban
Fix For: 2.6.3
Attachments: jgroups-logs.tbz2
I'm having an issue with a 2-machine cluster using a TCP stack
based on the tcp.xml from JGroups 2.6.1. On each machine I have 8 separate
channels running, on different ports, with 4 groups in 2 JVM instances.
After some period of time, one machine will fail to respond to a FD ping,
and gets suspected. The machine that failed is not responding in time it
seems from high CPU use, and many of the channels will fail FD around the
same time. The channels are configured with auto-reconnect. My
understanding was that the channel should "heal" itself and eventually
re-form into a new view with the same 2 members in the cluster, which
should apply to this situation because the machine that failed to respond
eventually will respond.
However, the group does not always seem to "heal" (sometimes it does,
sometimes not). Once it stops healing, it never seems to ever do so again,
and I get tons of NAKACK "message X not found in retransmission table"
ERROR logs. The only way to get the channel working agin is to shut down
the channel on both machines and then start them up again.
I'm not using muxed channels... just normal channels.
I have for now disabled shunning, and the channels seem to be able to
re-connect after a shun situation occurs, but after time I'm still seeing
something wrong with the channel in that the nodes are not able to send
messages to each other successfully, and I have tons of
2008-02-21 21:01:53,572 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
log entries, even while FD is receiving acks on that channel.
this is a degradation problem for me while my servers are running,
not just during deployment. After a while of running, the nodes get
shunned/disconnected some how (often from a slow response from the other
node) and then fail to ever merge back to form the original view of the 2
nodes again.
Now even with shun set to false in both FD and pbcast.GMS on both nodes
for all channels, eventually still the nodes reach the same state of never
re-forming... and I have tons eventual
2008-02-21 21:28:24,857 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
After a while in my application I'll force the channel to close wait a
minute, and reopen, and sometimes this gets the channel working again, but
doesn't always seem to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:46 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-692) Unable to recover from suspect/merge, with auto-reconnect
[ http://jira.jboss.com/jira/browse/JGRP-692?page=comments#action_12403779 ]
Matt Magoffin commented on JGRP-692:
------------------------------------
So the problem is still happening in 2.6.3.GA. A member got shunned... but the FD logs are
not the same, all I see now is
2008-03-20 01:14:12,737 DEBUG [org.jgroups.protocols.FD] [172.16.172.233:19183]: received
no heartbeat ack from 172.16.172.234:19183 for 6 times (60000 milliseconds), suspecting
it
2008-03-20 01:14:12,738 DEBUG [org.jgroups.protocols.FD] broadcasting SUSPECT message
[suspected_mbrs=[172.16.172.234:19183]] to group
2008-03-20 01:14:14,248 DEBUG [org.jgroups.blocks.RequestCorrelator]
suspect=172.16.172.234:19183
2008-03-20 01:14:14,248 DEBUG [com.autoleadservice.aop.EmailEventInterceptor] Got suspect
address [172.16.172.234:19183]
2008-03-20 01:14:14,308 DEBUG [org.jgroups.protocols.pbcast.GMS] new=[],
suspected=[172.16.172.234:19183], leaving=[], new view: [172.16.172.233:19183|10]
[172.16.172.233:19183]
2008-03-20 01:14:14,308 DEBUG [org.jgroups.protocols.pbcast.GMS]
view=[172.16.172.233:19183|10] [172.16.172.233:19183]
2008-03-20 01:14:14,308 DEBUG [org.jgroups.protocols.pbcast.GMS]
[local_addr=172.16.172.233:19183] view is [172.16.172.233:19183|10]
[172.16.172.233:19183]
2008-03-20 01:14:14,308 DEBUG [org.jgroups.protocols.pbcast.NAKACK] removing
172.16.172.234:19183 from xmit_table (not member anymore)
2008-03-20 01:14:14,308 DEBUG [org.jgroups.protocols.FD_SOCK] VIEW_CHANGE received:
[172.16.172.233:19183]
2008-03-20 01:14:14,309 DEBUG [org.jgroups.protocols.FD_SOCK] socket to null was reset
2008-03-20 01:14:14,309 DEBUG [org.jgroups.protocols.FD_SOCK] pinger thread terminated
2008-03-20 01:14:14,309 DEBUG [com.autoleadservice.aop.EmailEventInterceptor] Got view
change, handling in new thread...
2008-03-20 01:14:14,310 INFO [com.autoleadservice.aop.EmailEventInterceptor] Address
[172.16.172.234] has left channel [email-bounce-kia], buffering updates
...
2008-03-20 01:14:25,439 DEBUG [org.jgroups.protocols.MERGE2]
initial_mbrs=[[own_addr=172.16.172.233:19183, coord_addr=172.16.172.233:19183,
is_server=true]]
...
2008-03-20 01:18:47,731 DEBUG [org.jgroups.protocols.MERGE2]
initial_mbrs=[[own_addr=172.16.172.234:19183, coord_addr=172.16.172.233:19183,
is_server=true], [own_addr=172.16.172.233:19183, coord_addr=172.16.172.233:19183,
is_server=true]]
...
2008-03-20 01:19:08,100 WARN [org.jgroups.protocols.pbcast.NAKACK] 172.16.172.233:19183]
discarded message from non-member 172.16.172.234:19183, my view is
[172.16.172.233:19183|10] [172.16.172.233:19183]
From this point on, even though there are those MERGE2 messages
showing both servers 234 and 233, the NAKACK layer always discards messages from 234. The
group never merges, and both 234 and 233 remain in singleton groups.
Unable to recover from suspect/merge, with auto-reconnect
---------------------------------------------------------
Key: JGRP-692
URL: http://jira.jboss.com/jira/browse/JGRP-692
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.1
Environment: JBoss 4.2.2 on Linux 2.4.21-47.0.1.ELsmp i686 using Java HotSpot(TM)
Server VM (build 1.5.0_10-b03, mixed mode)
Reporter: Matt Magoffin
Assigned To: Bela Ban
Fix For: 2.6.3
Attachments: jgroups-logs.tbz2
I'm having an issue with a 2-machine cluster using a TCP stack
based on the tcp.xml from JGroups 2.6.1. On each machine I have 8 separate
channels running, on different ports, with 4 groups in 2 JVM instances.
After some period of time, one machine will fail to respond to a FD ping,
and gets suspected. The machine that failed is not responding in time it
seems from high CPU use, and many of the channels will fail FD around the
same time. The channels are configured with auto-reconnect. My
understanding was that the channel should "heal" itself and eventually
re-form into a new view with the same 2 members in the cluster, which
should apply to this situation because the machine that failed to respond
eventually will respond.
However, the group does not always seem to "heal" (sometimes it does,
sometimes not). Once it stops healing, it never seems to ever do so again,
and I get tons of NAKACK "message X not found in retransmission table"
ERROR logs. The only way to get the channel working agin is to shut down
the channel on both machines and then start them up again.
I'm not using muxed channels... just normal channels.
I have for now disabled shunning, and the channels seem to be able to
re-connect after a shun situation occurs, but after time I'm still seeing
something wrong with the channel in that the nodes are not able to send
messages to each other successfully, and I have tons of
2008-02-21 21:01:53,572 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
log entries, even while FD is receiving acks on that channel.
this is a degradation problem for me while my servers are running,
not just during deployment. After a while of running, the nodes get
shunned/disconnected some how (often from a slow response from the other
node) and then fail to ever merge back to form the original view of the 2
nodes again.
Now even with shun set to false in both FD and pbcast.GMS on both nodes
for all channels, eventually still the nodes reach the same state of never
re-forming... and I have tons eventual
2008-02-21 21:28:24,857 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
After a while in my application I'll force the channel to close wait a
minute, and reopen, and sometimes this gets the channel working again, but
doesn't always seem to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:25 a.m.
New subject: [JBoss JIRA] Assigned: (JGRP-692) Unable to recover from suspect/merge, with auto-reconnect
[ http://jira.jboss.com/jira/browse/JGRP-692?page=all ]
Bela Ban reassigned JGRP-692:
-----------------------------
Assignee: Vladimir Blagojevic (was: Bela Ban)
Unable to recover from suspect/merge, with auto-reconnect
---------------------------------------------------------
Key: JGRP-692
URL: http://jira.jboss.com/jira/browse/JGRP-692
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.1
Environment: JBoss 4.2.2 on Linux 2.4.21-47.0.1.ELsmp i686 using Java HotSpot(TM)
Server VM (build 1.5.0_10-b03, mixed mode)
Reporter: Matt Magoffin
Assigned To: Vladimir Blagojevic
Fix For: 2.6.3
Attachments: jgroups-logs.tbz2
I'm having an issue with a 2-machine cluster using a TCP stack
based on the tcp.xml from JGroups 2.6.1. On each machine I have 8 separate
channels running, on different ports, with 4 groups in 2 JVM instances.
After some period of time, one machine will fail to respond to a FD ping,
and gets suspected. The machine that failed is not responding in time it
seems from high CPU use, and many of the channels will fail FD around the
same time. The channels are configured with auto-reconnect. My
understanding was that the channel should "heal" itself and eventually
re-form into a new view with the same 2 members in the cluster, which
should apply to this situation because the machine that failed to respond
eventually will respond.
However, the group does not always seem to "heal" (sometimes it does,
sometimes not). Once it stops healing, it never seems to ever do so again,
and I get tons of NAKACK "message X not found in retransmission table"
ERROR logs. The only way to get the channel working agin is to shut down
the channel on both machines and then start them up again.
I'm not using muxed channels... just normal channels.
I have for now disabled shunning, and the channels seem to be able to
re-connect after a shun situation occurs, but after time I'm still seeing
something wrong with the channel in that the nodes are not able to send
messages to each other successfully, and I have tons of
2008-02-21 21:01:53,572 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
log entries, even while FD is receiving acks on that channel.
this is a degradation problem for me while my servers are running,
not just during deployment. After a while of running, the nodes get
shunned/disconnected some how (often from a slow response from the other
node) and then fail to ever merge back to form the original view of the 2
nodes again.
Now even with shun set to false in both FD and pbcast.GMS on both nodes
for all channels, eventually still the nodes reach the same state of never
re-forming... and I have tons eventual
2008-02-21 21:28:24,857 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
After a while in my application I'll force the channel to close wait a
minute, and reopen, and sometimes this gets the channel working again, but
doesn't always seem to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:55 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-692) Unable to recover from suspect/merge, with auto-reconnect
[ http://jira.jboss.com/jira/browse/JGRP-692?page=comments#action_12411446 ]
Vladimir Blagojevic commented on JGRP-692:
------------------------------------------
2.6.3 GA is not available yet. I assume you were using latest 2.6 cvs branch. We made some
fixes that we believe were related to this issue. 2.6 branch is very stable now and is
being prepared for a release as 2.6.3. Would you give latest 2.6 branch a try and see if
this issue can be reproduced easily.
Unable to recover from suspect/merge, with auto-reconnect
---------------------------------------------------------
Key: JGRP-692
URL: http://jira.jboss.com/jira/browse/JGRP-692
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.1
Environment: JBoss 4.2.2 on Linux 2.4.21-47.0.1.ELsmp i686 using Java HotSpot(TM)
Server VM (build 1.5.0_10-b03, mixed mode)
Reporter: Matt Magoffin
Assigned To: Vladimir Blagojevic
Fix For: 2.6.3
Attachments: jgroups-logs.tbz2
I'm having an issue with a 2-machine cluster using a TCP stack
based on the tcp.xml from JGroups 2.6.1. On each machine I have 8 separate
channels running, on different ports, with 4 groups in 2 JVM instances.
After some period of time, one machine will fail to respond to a FD ping,
and gets suspected. The machine that failed is not responding in time it
seems from high CPU use, and many of the channels will fail FD around the
same time. The channels are configured with auto-reconnect. My
understanding was that the channel should "heal" itself and eventually
re-form into a new view with the same 2 members in the cluster, which
should apply to this situation because the machine that failed to respond
eventually will respond.
However, the group does not always seem to "heal" (sometimes it does,
sometimes not). Once it stops healing, it never seems to ever do so again,
and I get tons of NAKACK "message X not found in retransmission table"
ERROR logs. The only way to get the channel working agin is to shut down
the channel on both machines and then start them up again.
I'm not using muxed channels... just normal channels.
I have for now disabled shunning, and the channels seem to be able to
re-connect after a shun situation occurs, but after time I'm still seeing
something wrong with the channel in that the nodes are not able to send
messages to each other successfully, and I have tons of
2008-02-21 21:01:53,572 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
log entries, even while FD is receiving acks on that channel.
this is a degradation problem for me while my servers are running,
not just during deployment. After a while of running, the nodes get
shunned/disconnected some how (often from a slow response from the other
node) and then fail to ever merge back to form the original view of the 2
nodes again.
Now even with shun set to false in both FD and pbcast.GMS on both nodes
for all channels, eventually still the nodes reach the same state of never
re-forming... and I have tons eventual
2008-02-21 21:28:24,857 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
After a while in my application I'll force the channel to close wait a
minute, and reopen, and sometimes this gets the channel working again, but
doesn't always seem to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:48 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-692) Unable to recover from suspect/merge, with auto-reconnect
[ http://jira.jboss.com/jira/browse/JGRP-692?page=comments#action_12411450 ]
Matt Magoffin commented on JGRP-692:
------------------------------------
Yes I was. Since that time to work around the issue we have upgraded our machines with
more RAM, moved to 64-bit Java and given our JVM more RAM to work with. Since this time,
the JGroups channels have been working well... it's been not quite 2 weeks since this
change.
Since this was causing major issues on our production system, I'm reluctant to try
this just yet, and the issue only happened on our production system. I'll wait a
couple more weeks at least before trying out the new 2.6.3 branch code.
Unable to recover from suspect/merge, with auto-reconnect
---------------------------------------------------------
Key: JGRP-692
URL: http://jira.jboss.com/jira/browse/JGRP-692
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.1
Environment: JBoss 4.2.2 on Linux 2.4.21-47.0.1.ELsmp i686 using Java HotSpot(TM)
Server VM (build 1.5.0_10-b03, mixed mode)
Reporter: Matt Magoffin
Assigned To: Vladimir Blagojevic
Fix For: 2.6.3
Attachments: jgroups-logs.tbz2
I'm having an issue with a 2-machine cluster using a TCP stack
based on the tcp.xml from JGroups 2.6.1. On each machine I have 8 separate
channels running, on different ports, with 4 groups in 2 JVM instances.
After some period of time, one machine will fail to respond to a FD ping,
and gets suspected. The machine that failed is not responding in time it
seems from high CPU use, and many of the channels will fail FD around the
same time. The channels are configured with auto-reconnect. My
understanding was that the channel should "heal" itself and eventually
re-form into a new view with the same 2 members in the cluster, which
should apply to this situation because the machine that failed to respond
eventually will respond.
However, the group does not always seem to "heal" (sometimes it does,
sometimes not). Once it stops healing, it never seems to ever do so again,
and I get tons of NAKACK "message X not found in retransmission table"
ERROR logs. The only way to get the channel working agin is to shut down
the channel on both machines and then start them up again.
I'm not using muxed channels... just normal channels.
I have for now disabled shunning, and the channels seem to be able to
re-connect after a shun situation occurs, but after time I'm still seeing
something wrong with the channel in that the nodes are not able to send
messages to each other successfully, and I have tons of
2008-02-21 21:01:53,572 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
log entries, even while FD is receiving acks on that channel.
this is a degradation problem for me while my servers are running,
not just during deployment. After a while of running, the nodes get
shunned/disconnected some how (often from a slow response from the other
node) and then fail to ever merge back to form the original view of the 2
nodes again.
Now even with shun set to false in both FD and pbcast.GMS on both nodes
for all channels, eventually still the nodes reach the same state of never
re-forming... and I have tons eventual
2008-02-21 21:28:24,857 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
After a while in my application I'll force the channel to close wait a
minute, and reopen, and sometimes this gets the channel working again, but
doesn't always seem to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:19 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-692) Unable to recover from suspect/merge, with auto-reconnect
[ http://jira.jboss.com/jira/browse/JGRP-692?page=comments#action_12414307 ]
Vladimir Blagojevic commented on JGRP-692:
------------------------------------------
Matt, Can you update us on the status of this issue? Have you encountered it again? We are
thinking of closing it!
Unable to recover from suspect/merge, with auto-reconnect
---------------------------------------------------------
Key: JGRP-692
URL: http://jira.jboss.com/jira/browse/JGRP-692
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.1
Environment: JBoss 4.2.2 on Linux 2.4.21-47.0.1.ELsmp i686 using Java HotSpot(TM)
Server VM (build 1.5.0_10-b03, mixed mode)
Reporter: Matt Magoffin
Assigned To: Vladimir Blagojevic
Fix For: 2.6.3
Attachments: jgroups-logs.tbz2
I'm having an issue with a 2-machine cluster using a TCP stack
based on the tcp.xml from JGroups 2.6.1. On each machine I have 8 separate
channels running, on different ports, with 4 groups in 2 JVM instances.
After some period of time, one machine will fail to respond to a FD ping,
and gets suspected. The machine that failed is not responding in time it
seems from high CPU use, and many of the channels will fail FD around the
same time. The channels are configured with auto-reconnect. My
understanding was that the channel should "heal" itself and eventually
re-form into a new view with the same 2 members in the cluster, which
should apply to this situation because the machine that failed to respond
eventually will respond.
However, the group does not always seem to "heal" (sometimes it does,
sometimes not). Once it stops healing, it never seems to ever do so again,
and I get tons of NAKACK "message X not found in retransmission table"
ERROR logs. The only way to get the channel working agin is to shut down
the channel on both machines and then start them up again.
I'm not using muxed channels... just normal channels.
I have for now disabled shunning, and the channels seem to be able to
re-connect after a shun situation occurs, but after time I'm still seeing
something wrong with the channel in that the nodes are not able to send
messages to each other successfully, and I have tons of
2008-02-21 21:01:53,572 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
log entries, even while FD is receiving acks on that channel.
this is a degradation problem for me while my servers are running,
not just during deployment. After a while of running, the nodes get
shunned/disconnected some how (often from a slow response from the other
node) and then fail to ever merge back to form the original view of the 2
nodes again.
Now even with shun set to false in both FD and pbcast.GMS on both nodes
for all channels, eventually still the nodes reach the same state of never
re-forming... and I have tons eventual
2008-02-21 21:28:24,857 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
After a while in my application I'll force the channel to close wait a
minute, and reopen, and sometimes this gets the channel working again, but
doesn't always seem to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:14 a.m.
New subject: [JBoss JIRA] Closed: (JGRP-692) Unable to recover from suspect/merge, with auto-reconnect
[ http://jira.jboss.com/jira/browse/JGRP-692?page=all ]
Vladimir Blagojevic closed JGRP-692.
------------------------------------
Resolution: Cannot Reproduce Bug
Since we did not hear back from Matt consider this issue not reproducible.
Unable to recover from suspect/merge, with auto-reconnect
---------------------------------------------------------
Key: JGRP-692
URL: http://jira.jboss.com/jira/browse/JGRP-692
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.1
Environment: JBoss 4.2.2 on Linux 2.4.21-47.0.1.ELsmp i686 using Java HotSpot(TM)
Server VM (build 1.5.0_10-b03, mixed mode)
Reporter: Matt Magoffin
Assigned To: Vladimir Blagojevic
Fix For: 2.6.3
Attachments: jgroups-logs.tbz2
I'm having an issue with a 2-machine cluster using a TCP stack
based on the tcp.xml from JGroups 2.6.1. On each machine I have 8 separate
channels running, on different ports, with 4 groups in 2 JVM instances.
After some period of time, one machine will fail to respond to a FD ping,
and gets suspected. The machine that failed is not responding in time it
seems from high CPU use, and many of the channels will fail FD around the
same time. The channels are configured with auto-reconnect. My
understanding was that the channel should "heal" itself and eventually
re-form into a new view with the same 2 members in the cluster, which
should apply to this situation because the machine that failed to respond
eventually will respond.
However, the group does not always seem to "heal" (sometimes it does,
sometimes not). Once it stops healing, it never seems to ever do so again,
and I get tons of NAKACK "message X not found in retransmission table"
ERROR logs. The only way to get the channel working agin is to shut down
the channel on both machines and then start them up again.
I'm not using muxed channels... just normal channels.
I have for now disabled shunning, and the channels seem to be able to
re-connect after a shun situation occurs, but after time I'm still seeing
something wrong with the channel in that the nodes are not able to send
messages to each other successfully, and I have tons of
2008-02-21 21:01:53,572 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
log entries, even while FD is receiving acks on that channel.
this is a degradation problem for me while my servers are running,
not just during deployment. After a while of running, the nodes get
shunned/disconnected some how (often from a slow response from the other
node) and then fail to ever merge back to form the original view of the 2
nodes again.
Now even with shun set to false in both FD and pbcast.GMS on both nodes
for all channels, eventually still the nodes reach the same state of never
re-forming... and I have tons eventual
2008-02-21 21:28:24,857 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
After a while in my application I'll force the channel to close wait a
minute, and reopen, and sometimes this gets the channel working again, but
doesn't always seem to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:57 p.m.
New subject: [JBoss JIRA] Commented: (JGRP-692) Unable to recover from suspect/merge, with auto-reconnect
[
https://jira.jboss.org/jira/browse/JGRP-692?page=com.atlassian.jira.plugi...
]
Juan Pablo Sevilla Martín commented on JGRP-692:
-------------------------------------------------
I'm using Hibernate/JBossCache over JGroups 2.6.7 on two web applications and I'm
getting the same problem: tons of NAKACK "message X not found in retransmission
table" logs.
These are the versions of the libraries I'm using:
jbosscache-core 3.1.0.GA
jboss-logging-spi 2.0.5.GA
jbosscommon-core-2.2.10.GA
jgroups 2.6.7.GA
Unable to recover from suspect/merge, with auto-reconnect
---------------------------------------------------------
Key: JGRP-692
URL: https://jira.jboss.org/jira/browse/JGRP-692
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.1
Environment: JBoss 4.2.2 on Linux 2.4.21-47.0.1.ELsmp i686 using Java HotSpot(TM)
Server VM (build 1.5.0_10-b03, mixed mode)
Reporter: Matt Magoffin
Assignee: Vladimir Blagojevic
Fix For: 2.6.3
Attachments: jgroups-logs.tbz2
I'm having an issue with a 2-machine cluster using a TCP stack
based on the tcp.xml from JGroups 2.6.1. On each machine I have 8 separate
channels running, on different ports, with 4 groups in 2 JVM instances.
After some period of time, one machine will fail to respond to a FD ping,
and gets suspected. The machine that failed is not responding in time it
seems from high CPU use, and many of the channels will fail FD around the
same time. The channels are configured with auto-reconnect. My
understanding was that the channel should "heal" itself and eventually
re-form into a new view with the same 2 members in the cluster, which
should apply to this situation because the machine that failed to respond
eventually will respond.
However, the group does not always seem to "heal" (sometimes it does,
sometimes not). Once it stops healing, it never seems to ever do so again,
and I get tons of NAKACK "message X not found in retransmission table"
ERROR logs. The only way to get the channel working agin is to shut down
the channel on both machines and then start them up again.
I'm not using muxed channels... just normal channels.
I have for now disabled shunning, and the channels seem to be able to
re-connect after a shun situation occurs, but after time I'm still seeing
something wrong with the channel in that the nodes are not able to send
messages to each other successfully, and I have tons of
2008-02-21 21:01:53,572 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
log entries, even while FD is receiving acks on that channel.
this is a degradation problem for me while my servers are running,
not just during deployment. After a while of running, the nodes get
shunned/disconnected some how (often from a slow response from the other
node) and then fail to ever merge back to form the original view of the 2
nodes again.
Now even with shun set to false in both FD and pbcast.GMS on both nodes
for all channels, eventually still the nodes reach the same state of never
re-forming... and I have tons eventual
2008-02-21 21:28:24,857 WARN [org.jgroups.protocols.pbcast.NAKACK]
172.16.172.233:19182] discarded message from non-member
172.16.172.234:19182, my view is [172.16.172.233:19182|8]
[172.16.172.233:19182]
After a while in my application I'll force the channel to close wait a
minute, and reopen, and sometimes this gets the channel working again, but
doesn't always seem to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5113
days inactive
5912
days old
24 comments
4 participants
participants (4)
-
Bela Ban (JIRA) -
Juan Pablo Sevilla Martín (JIRA)
-
Matt Magoffin (JIRA)
-
Vladimir Blagojevic (JIRA)