[jbosscache-issues] [JBoss JIRA] Created: (JBCACHE-1531) Replication is allowed when buddy group state is inconsistent

[Brian Stansberry (JIRA)]

Replication is allowed when buddy group state is inconsistent
-------------------------------------------------------------

                 Key: JBCACHE-1531
                 URL: https://jira.jboss.org/jira/browse/JBCACHE-1531
             Project: JBoss Cache
          Issue Type: Bug
      Security Level: Public (Everyone can see)
            Reporter: Brian Stansberry
            Assignee: Manik Surtani
             Fix For: 3.2.0.BETA1


There's nothing preventing application threads sending replication messages during the buddy-group reformation process.  E.g. during this section of BuddyManager.reassignBuddies:

      if (!obsoleteBuddies.isEmpty())
      {
         removeFromGroup(obsoleteBuddies);
      }
      else
      {
         log.trace("No obsolete buddies found, nothing to announce.");
      }

      if (!uninitialisedBuddies.isEmpty())
      {
         addBuddies(newBuddies);
      }

in the middle a node might have no buddies. But application threads continue replicating -- to no one.


[JBoss] 21:45:19,482 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (AsyncViewChangeHandlerThread,10.34.32.156:33822) Destination list is empty: no need to send message
[JBoss] 21:45:19,495 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-194) Destination list is empty: no need to send message
[JBoss] 21:45:19,495 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-194) Destination list is empty: no need to send message
[JBoss] 21:45:19,496 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-194) Destination list is empty: no need to send message
[JBoss] 21:45:19,515 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-95) Destination list is empty: no need to send message
[JBoss] 21:45:19,537 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-64) Destination list is empty: no need to send message
[JBoss] 21:45:19,540 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-65) Destination list is empty: no need to send message
[JBoss] 21:45:19,540 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-65) Destination list is empty: no need to send message
[JBoss] 21:45:19,542 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-65) Destination list is empty: no need to send message
[JBoss] 21:45:19,564 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-53) Destination list is empty: no need to send message
[JBoss] 21:45:19,568 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-13) Destination list is empty: no need to send message
[JBoss] 21:45:19,569 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-185) Destination list is empty: no need to send message
[JBoss] 21:45:19,570 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-196) Destination list is empty: no need to send message
[JBoss] 21:45:19,605 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-171) Destination list is empty: no need to send message
[JBoss] 21:45:19,661 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-19) Destination list is empty: no need to send message
[JBoss] 21:45:19,667 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-51) Destination list is empty: no need to send message
[JBoss] 21:45:19,682 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-109) Destination list is empty: no need to send message
[JBoss] 21:45:19,682 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-152) Destination list is empty: no need to send message
[JBoss] 21:45:19,727 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-182) Destination list is empty: no need to send message
[JBoss] 21:45:19,772 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-20) Destination list is empty: no need to send message
[JBoss] 21:45:19,827 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-134) Destination list is empty: no need to send message
[JBoss] 21:45:19,845 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-171) Destination list is empty: no need to send message
[JBoss] 21:45:19,854 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-115) Destination list is empty: no need to send message
[JBoss] 21:45:19,893 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-148) Destination list is empty: no need to send message
[JBoss] 21:45:19,978 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-74) Destination list is empty: no need to send message

Effect is state on the new buddy backup nodes can be inconsistent, since messages sent after the state transfer to the new buddies but before the new buddy is added to the group will be lost.

Writing the above makes me realize the problem is probably just in assignBuddies -- the window between

            state = acquireState( ...);

....

           buddyGroup.addBuddies(buddies);

Ideally in that window, local application threads calling BuddyManager.getBuddyAddress(AsVector) should block.


-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira