[jboss-jira] [JBoss JIRA] Updated: (JGRP-692) Unable to recover from suspect/merge, with auto-reconnect

Unable to recover from suspect/merge, with auto-reconnect --------------------------------------------------------- Key: JGRP-692 URL: http://jira.jboss.com/jira/browse/JGRP-692 Project: JGroups Issue Type: Bug Affects Versions: 2.6.1 Environment: JBoss 4.2.2 on Linux 2.4.21-47.0.1.ELsmp i686 using Java HotSpot(TM) Server VM (build 1.5.0_10-b03, mixed mode) Reporter: Matt Magoffin Assigned To: Bela Ban Attachments: jgroups-logs.tbz2 I'm having an issue with a 2-machine cluster using a TCP stack based on the tcp.xml from JGroups 2.6.1. On each machine I have 8 separate channels running, on different ports, with 4 groups in 2 JVM instances. After some period of time, one machine will fail to respond to a FD ping, and gets suspected. The machine that failed is not responding in time it seems from high CPU use, and many of the channels will fail FD around the same time. The channels are configured with auto-reconnect. My understanding was that the channel should "heal" itself and eventually re-form into a new view with the same 2 members in the cluster, which should apply to this situation because the machine that failed to respond eventually will respond. However, the group does not always seem to "heal" (sometimes it does, sometimes not). Once it stops healing, it never seems to ever do so again, and I get tons of NAKACK "message X not found in retransmission table" ERROR logs. The only way to get the channel working agin is to shut down the channel on both machines and then start them up again. I'm not using muxed channels... just normal channels. I have for now disabled shunning, and the channels seem to be able to re-connect after a shun situation occurs, but after time I'm still seeing something wrong with the channel in that the nodes are not able to send messages to each other successfully, and I have tons of 2008-02-21 21:01:53,572 WARN [org.jgroups.protocols.pbcast.NAKACK] 172.16.172.233:19182] discarded message from non-member 172.16.172.234:19182, my view is [172.16.172.233:19182|8] [172.16.172.233:19182] log entries, even while FD is receiving acks on that channel. this is a degradation problem for me while my servers are running, not just during deployment. After a while of running, the nodes get shunned/disconnected some how (often from a slow response from the other node) and then fail to ever merge back to form the original view of the 2 nodes again. Now even with shun set to false in both FD and pbcast.GMS on both nodes for all channels, eventually still the nodes reach the same state of never re-forming... and I have tons eventual 2008-02-21 21:28:24,857 WARN [org.jgroups.protocols.pbcast.NAKACK] 172.16.172.233:19182] discarded message from non-member 172.16.172.234:19182, my view is [172.16.172.233:19182|8] [172.16.172.233:19182] After a while in my application I'll force the channel to close wait a minute, and reopen, and sometimes this gets the channel working again, but doesn't always seem to.