[jboss-jira] [JBoss JIRA] (JGRP-1850) jGroups 3.4.3 timeouts after a node leaves and rejoins cluster

[Justin Cranford (JIRA)]

    [ https://issues.jboss.org/browse/JGRP-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12975324#comment-12975324 ] 

Justin Cranford commented on JGRP-1850:
---------------------------------------

I forgot to mention something else. This is why I suspect a problem in jGroups versus HA-JDBCx:

- If I repeat all the same steps, but I leave out the "Restart Tomcat 1" step, then the jGroups lock works. I can repeat the "Restart MySQL 3" and "Activate MySQL 3 From Tomcat 1" over and over without any timeouts.
- As soon as I Include the "Restart Tomcat 1" step, the jGroups lock times out. Leaving and rejoining the jGroups cluster is the only difference.


If you have a moment, can you skip the JGRP error messages in my catalina log to see if it points to a configuration issue or an unlikely bug?

> jGroups 3.4.3 timeouts after a node leaves and rejoins cluster
> --------------------------------------------------------------
>
>                 Key: JGRP-1850
>                 URL: https://issues.jboss.org/browse/JGRP-1850
>             Project: JGroups
>          Issue Type: Bug
>    Affects Versions: 3.4.3
>         Environment: Tomcat 7.0.54
> Java 7u45 x32
> HA-JDBC 3.0.0, 3.0.1, 3.0.2, 3.0.3-SNAPSHOT
> Debian 5 x64
>            Reporter: Justin Cranford
>            Assignee: Bela Ban
>         Attachments: catalina.out.bz2, ha-jdbc-jgroups-tcp.xml
>
>
> I have this open thread on HA-JDBC forum. The issue seems to be jGroups. I attached my Tomcat log there with steps to reproduce the issue.
> http://sourceforge.net/p/ha-jdbc/discussion/383397/thread/843c60ad/?limit=25&page=2#338a
> Here is an example of what my jGroups error looks like, copied from the HA-JDBC thread. I can reproduce this quite often, possibly every time I try it, but not sure if it happens 100% of the time. I have reproduced it at least half a dozen times.
> May 09, 2014 12:01:45 AM net.sf.hajdbc.logging.slf4j.SLF4JLogger log
> FINE: Received MemberAcquireLockCommand(writeLock()) from 10.0.0.187
> May 09, 2014 12:01:46 AM org.jgroups.logging.JDKLogImpl debug
> FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187
> May 09, 2014 12:01:49 AM org.jgroups.logging.JDKLogImpl debug
> FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187
> May 09, 2014 12:01:50 AM org.jgroups.logging.JDKLogImpl debug
> FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187
> May 09, 2014 12:01:51 AM net.sf.hajdbc.logging.slf4j.SLF4JLogger log
> WARNING: timeout sending message to 10.0.0.187
> org.jgroups.TimeoutException: timeout sending message to 10.0.0.187
>         at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419)
>         at net.sf.hajdbc.distributed.jgroups.JGroupsCommandDispatcher.execute(JGroupsCommandDispatcher.java:177)
>         at net.sf.hajdbc.lock.distributed.DistributedLockManager$DistributedLock.lockCoordinator(DistributedLockManager.java:420)
>         at net.sf.hajdbc.lock.distributed.DistributedLockManager$DistributedLock.lockInterruptibly(DistributedLockManager.java:320)
>         at net.sf.hajdbc.sql.DatabaseClusterImpl.activate(DatabaseClusterImpl.java:864)
>         at net.sf.hajdbc.sql.DatabaseClusterImpl.activate(DatabaseClusterImpl.java:156)
> May 09, 2014 12:01:51 AM net.sf.hajdbc.logging.slf4j.SLF4JLogger log
> FINE: Received MemberAcquireLockCommand(writeLock()) from 10.0.0.187
> May 09, 2014 12:01:53 AM org.jgroups.logging.JDKLogImpl debug
> FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187
> May 09, 2014 12:01:54 AM org.jgroups.logging.JDKLogImpl debug
> FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187
> May 09, 2014 12:01:57 AM org.jgroups.logging.JDKLogImpl debug
> FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187
> May 09, 2014 12:01:57 AM net.sf.hajdbc.logging.slf4j.SLF4JLogger log
> WARNING: timeout sending message to 10.0.0.187
> org.jgroups.TimeoutException: timeout sending message to 10.0.0.187
>         at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419)
>         at net.sf.hajdbc.distributed.jgroups.JGroupsCommandDispatcher.execute(JGroupsCommandDispatcher.java:177)
>         at net.sf.hajdbc.lock.distributed.DistributedLockManager$DistributedLock.lockCoordinator(DistributedLockManager.java:420)
>         at net.sf.hajdbc.lock.distributed.DistributedLockManager$DistributedLock.lockInterruptibly(DistributedLockManager.java:320)
>         at net.sf.hajdbc.sql.DatabaseClusterImpl.activate(DatabaseClusterImpl.java:864)
>         at net.sf.hajdbc.sql.DatabaseClusterImpl.activate(DatabaseClusterImpl.java:156)
> May 09, 2014 12:01:57 AM net.sf.hajdbc.logging.slf4j.SLF4JLogger log
> FINE: Received MemberAcquireLockCommand(writeLock()) from 10.0.0.187
> May 09, 2014 12:01:58 AM org.jgroups.logging.JDKLogImpl debug
> FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187
> May 09, 2014 12:02:01 AM org.jgroups.logging.JDKLogImpl debug
> FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187
> May 09, 2014 12:02:02 AM org.jgroups.logging.JDKLogImpl debug
> FINE: 10.0.0.188: sending are-you-alive msg to 10.0.0.187
> May 09, 2014 12:02:03 AM net.sf.hajdbc.logging.slf4j.SLF4JLogger log
> WARNING: timeout sending message to 10.0.0.187
> org.jgroups.TimeoutException: timeout sending message to 10.0.0.187
>         at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419)
>         at net.sf.hajdbc.distributed.jgroups.JGroupsCommandDispatcher.execute(JGroupsCommandDispatcher.java:177)
>         at net.sf.hajdbc.lock.distributed.DistributedLockManager$DistributedLock.lockCoordinator(DistributedLockManager.java:420)
>         at net.sf.hajdbc.lock.distributed.DistributedLockManager$DistributedLock.lockInterruptibly(DistributedLockManager.java:320)
>         at net.sf.hajdbc.sql.DatabaseClusterImpl.activate(DatabaseClusterImpl.java:864)
>         at net.sf.hajdbc.sql.DatabaseClusterImpl.activate(DatabaseClusterImpl.java:156)



--
This message was sent by Atlassian JIRA
(v6.2.6#6264)