]
Brian Stansberry reopened JBCACHE-1531:
---------------------------------------
Testing is showing blocking the callers is creating more pain than the original issue. :)
Basically the state transfer can't lock because a caller has locks; callers can't
clear locks because they can't acquire the latch to replicate. This basically blocks
everything until the state transfer timeout (60 secs).
Am about to attach a jawa03.log showing this happening. Problem begins at 23:21:53.
I think JBCACHE-1531 is an edge issue and I can put in some workarounds in the AS. IMHO we
should defer fixing this until after 3.2.0.GA so we can think of a cleaner fix.
Replication is allowed when buddy group state is inconsistent
-------------------------------------------------------------
Key: JBCACHE-1531
URL:
https://jira.jboss.org/jira/browse/JBCACHE-1531
Project: JBoss Cache
Issue Type: Bug
Security Level: Public(Everyone can see)
Reporter: Brian Stansberry
Assignee: Manik Surtani
Fix For: 3.2.0.CR1, 3.2.0.GA
Attachments: jawa03.log
There's nothing preventing application threads sending replication messages during
the buddy-group reformation process. E.g. during this section of
BuddyManager.reassignBuddies:
if (!obsoleteBuddies.isEmpty())
{
removeFromGroup(obsoleteBuddies);
}
else
{
log.trace("No obsolete buddies found, nothing to announce.");
}
if (!uninitialisedBuddies.isEmpty())
{
addBuddies(newBuddies);
}
in the middle a node might have no buddies. But application threads continue replicating
-- to no one.
[JBoss] 21:45:19,482 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher]
(AsyncViewChangeHandlerThread,10.34.32.156:33822) Destination list is empty: no need to
send message
[JBoss] 21:45:19,495 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher]
(ajp-jawa04%2F10.34.32.156-8009-194) Destination list is empty: no need to send message
[JBoss] 21:45:19,495 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher]
(ajp-jawa04%2F10.34.32.156-8009-194) Destination list is empty: no need to send message
[JBoss] 21:45:19,496 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher]
(ajp-jawa04%2F10.34.32.156-8009-194) Destination list is empty: no need to send message
[JBoss] 21:45:19,515 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher]
(ajp-jawa04%2F10.34.32.156-8009-95) Destination list is empty: no need to send message
[JBoss] 21:45:19,537 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher]
(ajp-jawa04%2F10.34.32.156-8009-64) Destination list is empty: no need to send message
[JBoss] 21:45:19,540 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher]
(ajp-jawa04%2F10.34.32.156-8009-65) Destination list is empty: no need to send message
[JBoss] 21:45:19,540 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher]
(ajp-jawa04%2F10.34.32.156-8009-65) Destination list is empty: no need to send message
[JBoss] 21:45:19,542 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher]
(ajp-jawa04%2F10.34.32.156-8009-65) Destination list is empty: no need to send message
[JBoss] 21:45:19,564 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher]
(ajp-jawa04%2F10.34.32.156-8009-53) Destination list is empty: no need to send message
[JBoss] 21:45:19,568 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher]
(ajp-jawa04%2F10.34.32.156-8009-13) Destination list is empty: no need to send message
[JBoss] 21:45:19,569 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher]
(ajp-jawa04%2F10.34.32.156-8009-185) Destination list is empty: no need to send message
[JBoss] 21:45:19,570 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher]
(ajp-jawa04%2F10.34.32.156-8009-196) Destination list is empty: no need to send message
[JBoss] 21:45:19,605 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher]
(ajp-jawa04%2F10.34.32.156-8009-171) Destination list is empty: no need to send message
[JBoss] 21:45:19,661 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher]
(ajp-jawa04%2F10.34.32.156-8009-19) Destination list is empty: no need to send message
[JBoss] 21:45:19,667 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher]
(ajp-jawa04%2F10.34.32.156-8009-51) Destination list is empty: no need to send message
[JBoss] 21:45:19,682 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher]
(ajp-jawa04%2F10.34.32.156-8009-109) Destination list is empty: no need to send message
[JBoss] 21:45:19,682 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher]
(ajp-jawa04%2F10.34.32.156-8009-152) Destination list is empty: no need to send message
[JBoss] 21:45:19,727 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher]
(ajp-jawa04%2F10.34.32.156-8009-182) Destination list is empty: no need to send message
[JBoss] 21:45:19,772 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher]
(ajp-jawa04%2F10.34.32.156-8009-20) Destination list is empty: no need to send message
[JBoss] 21:45:19,827 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher]
(ajp-jawa04%2F10.34.32.156-8009-134) Destination list is empty: no need to send message
[JBoss] 21:45:19,845 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher]
(ajp-jawa04%2F10.34.32.156-8009-171) Destination list is empty: no need to send message
[JBoss] 21:45:19,854 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher]
(ajp-jawa04%2F10.34.32.156-8009-115) Destination list is empty: no need to send message
[JBoss] 21:45:19,893 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher]
(ajp-jawa04%2F10.34.32.156-8009-148) Destination list is empty: no need to send message
[JBoss] 21:45:19,978 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher]
(ajp-jawa04%2F10.34.32.156-8009-74) Destination list is empty: no need to send message
Effect is state on the new buddy backup nodes can be inconsistent, since messages sent
after the state transfer to the new buddies but before the new buddy is added to the group
will be lost.
Writing the above makes me realize the problem is probably just in assignBuddies -- the
window between
state = acquireState( ...);
....
buddyGroup.addBuddies(buddies);
Ideally in that window, local application threads calling
BuddyManager.getBuddyAddress(AsVector) should block.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: