[jboss-jira] [JBoss JIRA] Updated: (JGRP-1299) Node does not re-join the cluster after several lost pings

Node does not re-join the cluster after several lost pings ---------------------------------------------------------- Key: JGRP-1299 URL: https://issues.jboss.org/browse/JGRP-1299 Project: JGroups Issue Type: Bug Affects Versions: 2.6.15 Environment: Solaris OS 10 & Java 1.5 & 1.6 Reporter: Igor M Assignee: Bela Ban Priority: Critical Attachments: Node1.log, Node2.log, stacks.xml This is what we see in production: 1. Node 1 does not send pings for 25 seconds 2. Node 2 notices 6 lost pings (in 15 seconds) 3. Node 2 starts sending "broadcast SUSPECT" 4. Node 1 replies to a few of them 5. Node 2 does not receive replies until after 15 seconds after it suspected node 1 6. Node 2 removes Node 1 from the view 7. Node 1 keeps sending "are-you-alive" and Node 2 is now discarding them At this time Node 1 believe there are two nodes in the cluster, and Node 2 only sees itself. In the lab we were able to reproduce the problem by stopping Node 1 process: pstop {PID} ; sleep 35 ; prun {PID} Once the process is resumed it can never join the cluster. Here is the log snipped from Node 1. The first two lines show 26 seconds interval between pings while it should have been 2.5 seconds. Node 2 logs for the same time interval are after Node 1 logs I traced the 26 seconds delay to the GC cycle on Node 1. pstop/sleep/prun have almost the same effect.