jGroups 3.4.3 timeouts after a node leaves and rejoins cluster -------------------------------------------------------------- Key: JGRP-1850 URL: https://issues.jboss.org/browse/JGRP-1850 Project: JGroups Issue Type: Bug Affects Versions: 3.4.3 Environment: Tomcat 7.0.54 Java 7u45 x32 HA-JDBC 3.0.0, 3.0.1, 3.0.2, 3.0.3-SNAPSHOT Debian 5 x64 Reporter: Justin Cranford Assignee: Bela Ban Attachments: catalina.out.bz2, ha-jdbc-jgroups-tcp.xml I have this open thread on HA-JDBC forum. The issue seems to be jGroups. I attached my Tomcat log there with steps to reproduce the issue. http://sourceforge.net/p/ha-jdbc/discussion/383397/thread/843c60ad/?limit... Here is an example of what my jGroups error looks like, copied from the HA-JDBC thread. I can reproduce this quite often, possibly every time I try it, but not sure if it happens 100% of the time. I have reproduced it at least half a dozen times. May 09, 2014 12:01:45 AM net.sf.hajdbc.logging.slf4j.SLF4JLogger log FINE: Received MemberAcquireLockCommand(writeLock()) from 10.0.0.187 May 09, 2014 12:01:46 AM org.jgroups.logging.JDKLogImpl debug FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187 May 09, 2014 12:01:49 AM org.jgroups.logging.JDKLogImpl debug FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187 May 09, 2014 12:01:50 AM org.jgroups.logging.JDKLogImpl debug FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187 May 09, 2014 12:01:51 AM net.sf.hajdbc.logging.slf4j.SLF4JLogger log WARNING: timeout sending message to 10.0.0.187 org.jgroups.TimeoutException: timeout sending message to 10.0.0.187 at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419) at net.sf.hajdbc.distributed.jgroups.JGroupsCommandDispatcher.execute(JGroupsCommandDispatcher.java:177) at net.sf.hajdbc.lock.distributed.DistributedLockManager$DistributedLock.lockCoordinator(DistributedLockManager.java:420) at net.sf.hajdbc.lock.distributed.DistributedLockManager$DistributedLock.lockInterruptibly(DistributedLockManager.java:320) at net.sf.hajdbc.sql.DatabaseClusterImpl.activate(DatabaseClusterImpl.java:864) at net.sf.hajdbc.sql.DatabaseClusterImpl.activate(DatabaseClusterImpl.java:156) May 09, 2014 12:01:51 AM net.sf.hajdbc.logging.slf4j.SLF4JLogger log FINE: Received MemberAcquireLockCommand(writeLock()) from 10.0.0.187 May 09, 2014 12:01:53 AM org.jgroups.logging.JDKLogImpl debug FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187 May 09, 2014 12:01:54 AM org.jgroups.logging.JDKLogImpl debug FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187 May 09, 2014 12:01:57 AM org.jgroups.logging.JDKLogImpl debug FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187 May 09, 2014 12:01:57 AM net.sf.hajdbc.logging.slf4j.SLF4JLogger log WARNING: timeout sending message to 10.0.0.187 org.jgroups.TimeoutException: timeout sending message to 10.0.0.187 at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419) at net.sf.hajdbc.distributed.jgroups.JGroupsCommandDispatcher.execute(JGroupsCommandDispatcher.java:177) at net.sf.hajdbc.lock.distributed.DistributedLockManager$DistributedLock.lockCoordinator(DistributedLockManager.java:420) at net.sf.hajdbc.lock.distributed.DistributedLockManager$DistributedLock.lockInterruptibly(DistributedLockManager.java:320) at net.sf.hajdbc.sql.DatabaseClusterImpl.activate(DatabaseClusterImpl.java:864) at net.sf.hajdbc.sql.DatabaseClusterImpl.activate(DatabaseClusterImpl.java:156) May 09, 2014 12:01:57 AM net.sf.hajdbc.logging.slf4j.SLF4JLogger log FINE: Received MemberAcquireLockCommand(writeLock()) from 10.0.0.187 May 09, 2014 12:01:58 AM org.jgroups.logging.JDKLogImpl debug FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187 May 09, 2014 12:02:01 AM org.jgroups.logging.JDKLogImpl debug FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187 May 09, 2014 12:02:02 AM org.jgroups.logging.JDKLogImpl debug FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187 May 09, 2014 12:02:03 AM net.sf.hajdbc.logging.slf4j.SLF4JLogger log WARNING: timeout sending message to 10.0.0.187 org.jgroups.TimeoutException: timeout sending message to 10.0.0.187 at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419) at net.sf.hajdbc.distributed.jgroups.JGroupsCommandDispatcher.execute(JGroupsCommandDispatcher.java:177) at net.sf.hajdbc.lock.distributed.DistributedLockManager$DistributedLock.lockCoordinator(DistributedLockManager.java:420) at net.sf.hajdbc.lock.distributed.DistributedLockManager$DistributedLock.lockInterruptibly(DistributedLockManager.java:320) at net.sf.hajdbc.sql.DatabaseClusterImpl.activate(DatabaseClusterImpl.java:864) at net.sf.hajdbc.sql.DatabaseClusterImpl.activate(DatabaseClusterImpl.java:156)