Replication is allowed when buddy group state is inconsistent ------------------------------------------------------------- Key: JBCACHE-1531 URL: https://jira.jboss.org/jira/browse/JBCACHE-1531 Project: JBoss Cache Issue Type: Bug Security Level: Public (Everyone can see) Reporter: Brian Stansberry Assignee: Manik Surtani Fix For: 3.2.0.BETA1 There's nothing preventing application threads sending replication messages during the buddy-group reformation process. E.g. during this section of BuddyManager.reassignBuddies: if (!obsoleteBuddies.isEmpty()) { removeFromGroup(obsoleteBuddies); } else { log.trace("No obsolete buddies found, nothing to announce."); } if (!uninitialisedBuddies.isEmpty()) { addBuddies(newBuddies); } in the middle a node might have no buddies. But application threads continue replicating -- to no one. [JBoss] 21:45:19,482 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (AsyncViewChangeHandlerThread,10.34.32.156:33822) Destination list is empty: no need to send message [JBoss] 21:45:19,495 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-194) Destination list is empty: no need to send message [JBoss] 21:45:19,495 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-194) Destination list is empty: no need to send message [JBoss] 21:45:19,496 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-194) Destination list is empty: no need to send message [JBoss] 21:45:19,515 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-95) Destination list is empty: no need to send message [JBoss] 21:45:19,537 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-64) Destination list is empty: no need to send message [JBoss] 21:45:19,540 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-65) Destination list is empty: no need to send message [JBoss] 21:45:19,540 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-65) Destination list is empty: no need to send message [JBoss] 21:45:19,542 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-65) Destination list is empty: no need to send message [JBoss] 21:45:19,564 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-53) Destination list is empty: no need to send message [JBoss] 21:45:19,568 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-13) Destination list is empty: no need to send message [JBoss] 21:45:19,569 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-185) Destination list is empty: no need to send message [JBoss] 21:45:19,570 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-196) Destination list is empty: no need to send message [JBoss] 21:45:19,605 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-171) Destination list is empty: no need to send message [JBoss] 21:45:19,661 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-19) Destination list is empty: no need to send message [JBoss] 21:45:19,667 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-51) Destination list is empty: no need to send message [JBoss] 21:45:19,682 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-109) Destination list is empty: no need to send message [JBoss] 21:45:19,682 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-152) Destination list is empty: no need to send message [JBoss] 21:45:19,727 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-182) Destination list is empty: no need to send message [JBoss] 21:45:19,772 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-20) Destination list is empty: no need to send message [JBoss] 21:45:19,827 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-134) Destination list is empty: no need to send message [JBoss] 21:45:19,845 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-171) Destination list is empty: no need to send message [JBoss] 21:45:19,854 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-115) Destination list is empty: no need to send message [JBoss] 21:45:19,893 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-148) Destination list is empty: no need to send message [JBoss] 21:45:19,978 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-74) Destination list is empty: no need to send message Effect is state on the new buddy backup nodes can be inconsistent, since messages sent after the state transfer to the new buddies but before the new buddy is added to the group will be lost. Writing the above makes me realize the problem is probably just in assignBuddies -- the window between state = acquireState( ...); .... buddyGroup.addBuddies(buddies); Ideally in that window, local application threads calling BuddyManager.getBuddyAddress(AsVector) should block. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira