[jboss-jira] [JBoss JIRA] (JGRP-1850) jGroups 3.4.3 timeouts after a node leaves and rejoins cluster

Wed Jun 11 08:57:38 EDT 2014

    [ https://issues.jboss.org/browse/JGRP-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12975256#comment-12975256 ] 

Justin Cranford edited comment on JGRP-1850 at 6/11/14 8:55 AM:
----------------------------------------------------------------

I posed the question on the HA-JDBC forum. The maintainer (Peter Ferrero) looked at the problem and thinks it is similar to a bug fixed in jGroups 3.4.1.

I don't think anything is holding onto the lock, otherwise I would have seen it in the log. Here is the startup sequence of my application cluster to show how I reproduce it. Note that I am using 3 identical Linux servers with the same software stack.

Start MySQL 1
Start MySQL 2
Start MySQL 3
Start Tomcat 1 with jGroups in TCP mode (I can attach my config)
Start Tomcat 2
Start Tomcat 3
Restart MySQL 3 (HA-JDBC distributed state is updated to shun MySQL 3 from DB cluster)
Restart Tomcat 1
>From Tomcat 1, attempt to reactivate MySQL 3 (acquire lock before copying MySQL 1 contents to MySQL 3)

This is the first time the lock would be acquired, and it is timing out.

was (Author: justincranford):
I posed the question on the HA-JDBC forum. The maintainer (Peter Ferrero) looked at the problem and thinks it is similar to a bug fixed in jGroups 3.4.1.

I don't think anything is holding onto the lock, otherwise I would have seen it in the log. Here is the startup sequence of my application cluster to show how I reproduce it. Note that I am using 3 identical Linux servers with the same software stack.

Start MySQL 1
Start MySQL 2
Start MySQL 3
Start Tomcat 1 with jGroups in TCP mode (I can attach my config)
Start Tomcat 2
Start Tomcat 3
Restart MySQL 3 (HA-JDBC distributed state us updated to shun MySQL 3 from DB cluster)
Restart Tomcat 1
>From Tomcat 1, attempt to reactivate MySQL 3 (acquire lock before copying MySQL 1 contents to MySQL 3)

This is the first time the lock would be acquired, and it is timing out.

> jGroups 3.4.3 timeouts after a node leaves and rejoins cluster
> --------------------------------------------------------------
>
>                 Key: JGRP-1850
>                 URL: https://issues.jboss.org/browse/JGRP-1850
>             Project: JGroups
>          Issue Type: Bug
>    Affects Versions: 3.4.3
>         Environment: Tomcat 7.0.54
> Java 7u45 x32
> HA-JDBC 3.0.0, 3.0.1, 3.0.2, 3.0.3-SNAPSHOT
> Debian 5 x64
>            Reporter: Justin Cranford
>            Assignee: Bela Ban
>         Attachments: catalina.out.bz2
>
>
> I have this open thread on HA-JDBC forum. The issue seems to be jGroups. I attached my Tomcat log there with steps to reproduce the issue.
> http://sourceforge.net/p/ha-jdbc/discussion/383397/thread/843c60ad/?limit=25&page=2#338a
> Here is an example of what my jGroups error looks like, copied from the HA-JDBC thread. I can reproduce this quite often, possibly every time I try it, but not sure if it happens 100% of the time. I have reproduced it at least half a dozen times.
> May 09, 2014 12:01:45 AM net.sf.hajdbc.logging.slf4j.SLF4JLogger log
> FINE: Received MemberAcquireLockCommand(writeLock()) from 10.0.0.187
> May 09, 2014 12:01:46 AM org.jgroups.logging.JDKLogImpl debug
> FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187
> May 09, 2014 12:01:49 AM org.jgroups.logging.JDKLogImpl debug
> FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187
> May 09, 2014 12:01:50 AM org.jgroups.logging.JDKLogImpl debug
> FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187
> May 09, 2014 12:01:51 AM net.sf.hajdbc.logging.slf4j.SLF4JLogger log
> WARNING: timeout sending message to 10.0.0.187
> org.jgroups.TimeoutException: timeout sending message to 10.0.0.187
>         at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419)
>         at net.sf.hajdbc.distributed.jgroups.JGroupsCommandDispatcher.execute(JGroupsCommandDispatcher.java:177)
>         at net.sf.hajdbc.lock.distributed.DistributedLockManager$DistributedLock.lockCoordinator(DistributedLockManager.java:420)
>         at net.sf.hajdbc.lock.distributed.DistributedLockManager$DistributedLock.lockInterruptibly(DistributedLockManager.java:320)
>         at net.sf.hajdbc.sql.DatabaseClusterImpl.activate(DatabaseClusterImpl.java:864)
>         at net.sf.hajdbc.sql.DatabaseClusterImpl.activate(DatabaseClusterImpl.java:156)
> May 09, 2014 12:01:51 AM net.sf.hajdbc.logging.slf4j.SLF4JLogger log
> FINE: Received MemberAcquireLockCommand(writeLock()) from 10.0.0.187
> May 09, 2014 12:01:53 AM org.jgroups.logging.JDKLogImpl debug
> FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187
> May 09, 2014 12:01:54 AM org.jgroups.logging.JDKLogImpl debug
> FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187
> May 09, 2014 12:01:57 AM org.jgroups.logging.JDKLogImpl debug
> FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187
> May 09, 2014 12:01:57 AM net.sf.hajdbc.logging.slf4j.SLF4JLogger log
> WARNING: timeout sending message to 10.0.0.187
> org.jgroups.TimeoutException: timeout sending message to 10.0.0.187
>         at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419)
>         at net.sf.hajdbc.distributed.jgroups.JGroupsCommandDispatcher.execute(JGroupsCommandDispatcher.java:177)
>         at net.sf.hajdbc.lock.distributed.DistributedLockManager$DistributedLock.lockCoordinator(DistributedLockManager.java:420)
>         at net.sf.hajdbc.lock.distributed.DistributedLockManager$DistributedLock.lockInterruptibly(DistributedLockManager.java:320)
>         at net.sf.hajdbc.sql.DatabaseClusterImpl.activate(DatabaseClusterImpl.java:864)
>         at net.sf.hajdbc.sql.DatabaseClusterImpl.activate(DatabaseClusterImpl.java:156)
> May 09, 2014 12:01:57 AM net.sf.hajdbc.logging.slf4j.SLF4JLogger log
> FINE: Received MemberAcquireLockCommand(writeLock()) from 10.0.0.187
> May 09, 2014 12:01:58 AM org.jgroups.logging.JDKLogImpl debug
> FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187
> May 09, 2014 12:02:01 AM org.jgroups.logging.JDKLogImpl debug
> FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187
> May 09, 2014 12:02:02 AM org.jgroups.logging.JDKLogImpl debug
> FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187
> May 09, 2014 12:02:03 AM net.sf.hajdbc.logging.slf4j.SLF4JLogger log
> WARNING: timeout sending message to 10.0.0.187
> org.jgroups.TimeoutException: timeout sending message to 10.0.0.187
>         at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419)
>         at net.sf.hajdbc.distributed.jgroups.JGroupsCommandDispatcher.execute(JGroupsCommandDispatcher.java:177)
>         at net.sf.hajdbc.lock.distributed.DistributedLockManager$DistributedLock.lockCoordinator(DistributedLockManager.java:420)
>         at net.sf.hajdbc.lock.distributed.DistributedLockManager$DistributedLock.lockInterruptibly(DistributedLockManager.java:320)
>         at net.sf.hajdbc.sql.DatabaseClusterImpl.activate(DatabaseClusterImpl.java:864)
>         at net.sf.hajdbc.sql.DatabaseClusterImpl.activate(DatabaseClusterImpl.java:156)

--
This message was sent by Atlassian JIRA
(v6.2.6#6264)