[jbosscache-issues] [JBoss JIRA] Reopened: (JBCACHE-1531) Replication is allowed when buddy group state is inconsistent

[Brian Stansberry (JIRA)]

     [ https://jira.jboss.org/jira/browse/JBCACHE-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Brian Stansberry reopened JBCACHE-1531:
---------------------------------------



Testing is showing blocking the callers is creating more pain than the original issue. :)  Basically the state transfer can't lock because a caller has locks; callers can't clear locks because they can't acquire the latch to replicate.  This basically blocks everything until the state transfer timeout (60 secs).

Am about to attach a jawa03.log showing this happening. Problem begins at 23:21:53.

I think JBCACHE-1531 is an edge issue and I can put in some workarounds in the AS. IMHO we should defer fixing this until after 3.2.0.GA so we can think of a cleaner fix.

> Replication is allowed when buddy group state is inconsistent
> -------------------------------------------------------------
>
>                 Key: JBCACHE-1531
>                 URL: https://jira.jboss.org/jira/browse/JBCACHE-1531
>             Project: JBoss Cache
>          Issue Type: Bug
>      Security Level: Public(Everyone can see) 
>            Reporter: Brian Stansberry
>            Assignee: Manik Surtani
>             Fix For: 3.2.0.CR1, 3.2.0.GA
>
>         Attachments: jawa03.log
>
>
> There's nothing preventing application threads sending replication messages during the buddy-group reformation process.  E.g. during this section of BuddyManager.reassignBuddies:
>       if (!obsoleteBuddies.isEmpty())
>       {
>          removeFromGroup(obsoleteBuddies);
>       }
>       else
>       {
>          log.trace("No obsolete buddies found, nothing to announce.");
>       }
>       if (!uninitialisedBuddies.isEmpty())
>       {
>          addBuddies(newBuddies);
>       }
> in the middle a node might have no buddies. But application threads continue replicating -- to no one.
> [JBoss] 21:45:19,482 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (AsyncViewChangeHandlerThread,10.34.32.156:33822) Destination list is empty: no need to send message
> [JBoss] 21:45:19,495 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-194) Destination list is empty: no need to send message
> [JBoss] 21:45:19,495 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-194) Destination list is empty: no need to send message
> [JBoss] 21:45:19,496 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-194) Destination list is empty: no need to send message
> [JBoss] 21:45:19,515 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-95) Destination list is empty: no need to send message
> [JBoss] 21:45:19,537 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-64) Destination list is empty: no need to send message
> [JBoss] 21:45:19,540 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-65) Destination list is empty: no need to send message
> [JBoss] 21:45:19,540 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-65) Destination list is empty: no need to send message
> [JBoss] 21:45:19,542 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-65) Destination list is empty: no need to send message
> [JBoss] 21:45:19,564 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-53) Destination list is empty: no need to send message
> [JBoss] 21:45:19,568 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-13) Destination list is empty: no need to send message
> [JBoss] 21:45:19,569 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-185) Destination list is empty: no need to send message
> [JBoss] 21:45:19,570 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-196) Destination list is empty: no need to send message
> [JBoss] 21:45:19,605 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-171) Destination list is empty: no need to send message
> [JBoss] 21:45:19,661 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-19) Destination list is empty: no need to send message
> [JBoss] 21:45:19,667 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-51) Destination list is empty: no need to send message
> [JBoss] 21:45:19,682 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-109) Destination list is empty: no need to send message
> [JBoss] 21:45:19,682 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-152) Destination list is empty: no need to send message
> [JBoss] 21:45:19,727 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-182) Destination list is empty: no need to send message
> [JBoss] 21:45:19,772 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-20) Destination list is empty: no need to send message
> [JBoss] 21:45:19,827 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-134) Destination list is empty: no need to send message
> [JBoss] 21:45:19,845 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-171) Destination list is empty: no need to send message
> [JBoss] 21:45:19,854 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-115) Destination list is empty: no need to send message
> [JBoss] 21:45:19,893 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-148) Destination list is empty: no need to send message
> [JBoss] 21:45:19,978 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-74) Destination list is empty: no need to send message
> Effect is state on the new buddy backup nodes can be inconsistent, since messages sent after the state transfer to the new buddies but before the new buddy is added to the group will be lost.
> Writing the above makes me realize the problem is probably just in assignBuddies -- the window between
>             state = acquireState( ...);
> ....
>            buddyGroup.addBuddies(buddies);
> Ideally in that window, local application threads calling BuddyManager.getBuddyAddress(AsVector) should block.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira