[JBoss JIRA] Created: (JBCACHE-1445) Problem with cleanup after data gravitation
by Brian Stansberry (JIRA)
Problem with cleanup after data gravitation
-------------------------------------------
Key: JBCACHE-1445
URL: https://jira.jboss.org/jira/browse/JBCACHE-1445
Project: JBoss Cache
Issue Type: Bug
Security Level: Public (Everyone can see)
Components: Buddy Replication
Affects Versions: 3.0.0.GA
Reporter: Brian Stansberry
Assignee: Manik Surtani
JBoss AS web session replication soak testing is showing issues with buddy replication. One of the failure modes seems to show leftover data remaining in the main tree for a session's former owner after the session has failed over to another node.
I've taken this issue as a good chance to start filling in the test infrastructure in the org.jboss.cache.integration package. Test org.jboss.cache.integration.websession.BuddyReplicationFailoverTest.testFailoverAndFailBack() shows the issue.
I think some of the tests in the buddyreplication package should be checking for this; not sure why they pass. Likely some subtle variation in config or something.
The test uses a lot of infrastructure to mock what the AS does. But underneath it all, the commands to the cache come down to:
node0:
getData(fqn) w/ data gravitation option // nothing there
put(fqn, map) // establishes session
put(fqn, map) // updates session
node3
getData(fqn) w/ data gravitation option // gravitates session
put(fqn, map) // updates session
At this point the cache contents are examined, and the original node in node0's main tree is still present. A buddy backup node, data owner node3, is also present on node0, as it should be.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
16 years, 1 month
[JBoss JIRA] Created: (JBCACHE-1443) Async marshalling, async cache loading and async notification handling use unbounded queues
by Brian Stansberry (JIRA)
Async marshalling, async cache loading and async notification handling use unbounded queues
--------------------------------------------------------------------------------------------
Key: JBCACHE-1443
URL: https://jira.jboss.org/jira/browse/JBCACHE-1443
Project: JBoss Cache
Issue Type: Bug
Security Level: Public (Everyone can see)
Affects Versions: 3.0.0.CR3
Reporter: Brian Stansberry
Assignee: Manik Surtani
Priority: Critical
Fix For: 3.0.0.GA
The async handling in CommandAwareRpcDispatcher, AsyncCacheLoader and NotifierImpl all use Executors.newFixedThreadPool() so set up their ExecutorService. Per the method javadoc for that method "Creates a thread pool that reuses a fixed set of threads operating off a shared unbounded queue". Use of an unbounded queue is inherently unsafe, and particularly in these use cases where there could be a very large number of threads adding tasks to the queue, while the default config only creates a single thread to take tasks off the queue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
16 years, 1 month
[JBoss JIRA] Created: (JBCACHE-1441) DataContainerImpl.toString() can throw UnsupportedOperationException
by Elias Ross (JIRA)
DataContainerImpl.toString() can throw UnsupportedOperationException
--------------------------------------------------------------------
Key: JBCACHE-1441
URL: https://jira.jboss.org/jira/browse/JBCACHE-1441
Project: JBoss Cache
Issue Type: Bug
Security Level: Public (Everyone can see)
Reporter: Elias Ross
Assignee: Manik Surtani
When doing some testing with EasyMock, saw this:
Caused by: java.lang.UnsupportedOperationException: Not supported in UnversionedNode
at org.jboss.cache.AbstractNode.getChildrenDirect(AbstractNode.java:277)
at org.jboss.cache.mvcc.NodeReference.getChildrenDirect(NodeReference.java:267)
at org.jboss.cache.invocation.NodeInvocationDelegate.getChildrenDirect(NodeInvocationDelegate.java:163)
at org.jboss.cache.DataContainerImpl.numNodes(DataContainerImpl.java:450)
at org.jboss.cache.DataContainerImpl.getNumberOfNodes(DataContainerImpl.java:427)
at org.jboss.cache.DataContainerImpl.toString(DataContainerImpl.java:386)
at org.jboss.cache.DataContainerImpl.toString(DataContainerImpl.java:364)
at org.jboss.cache.invocation.CacheInvocationDelegate.toString(CacheInvocationDelegate.java:154)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
16 years, 1 month
[JBoss JIRA] Created: (JBCACHE-1439) CacheLoaderInterceptor calls both CacheLoader.exist() and get() for every load
by Elias Ross (JIRA)
CacheLoaderInterceptor calls both CacheLoader.exist() and get() for every load
------------------------------------------------------------------------------
Key: JBCACHE-1439
URL: https://jira.jboss.org/jira/browse/JBCACHE-1439
Project: JBoss Cache
Issue Type: Bug
Security Level: Public (Everyone can see)
Affects Versions: 3.0.0.CR3
Reporter: Elias Ross
Assignee: Manik Surtani
Every Cache.get() operation where there is data in the cache loader calls both CacheLoader.exist() followed by CacheLoader.get().
A single Cache.get() operation would be sufficient. Since exist() is non-trivial, perhaps requiring a network call, performance is hurt doing this extra operation.
I believe earlier versions did not do this.
if (n == null && loader.exists(fqn))
{
// just create a dummy node in memory
n = helper.wrapNodeForWriting(ctx, fqn, true, true, true, false, false);
n.setDataLoaded(false);
}
else
{
this.cacheMisses++;
}
if (!bypassLoadingData && n != null)
{
n = loadNode(ctx, fqn, n);
}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
16 years, 1 month
[JBoss JIRA] Updated: (JBCACHE-1240) Option to create another copy of backup data if data owner dies
by Manik Surtani (JIRA)
[ https://jira.jboss.org/jira/browse/JBCACHE-1240?page=com.atlassian.jira.p... ]
Manik Surtani updated JBCACHE-1240:
-----------------------------------
Fix Version/s: 3.1.0
> Option to create another copy of backup data if data owner dies
> ---------------------------------------------------------------
>
> Key: JBCACHE-1240
> URL: https://jira.jboss.org/jira/browse/JBCACHE-1240
> Project: JBoss Cache
> Issue Type: Feature Request
> Security Level: Public(Everyone can see)
> Components: Buddy Replication
> Reporter: Brian Stansberry
> Assignee: Manik Surtani
> Fix For: 3.1.0
>
>
> With BR, if a data owner dies, the number of copies of the data it owns is reduced by one. If the data is accessed, another node becomes owner and the number of backups returns to normal. But if the data isn't accessed, the number of backups remains reduced.
> Besides the general problem of lesser reliability for this data, this can be a problem with things like rolling upgrades, where all the backup nodes could be restarted before the data is accessed, leading to data loss.
> Downside to making another copy of the backup data when the owner dies is it adds stress to the system by replicating more data. So, this should be configurable.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
16 years, 1 month
[JBoss JIRA] Created: (JBCACHE-1437) IllegalMonitorStateException on shutdown "breaking locks"
by Elias Ross (JIRA)
IllegalMonitorStateException on shutdown "breaking locks"
---------------------------------------------------------
Key: JBCACHE-1437
URL: https://jira.jboss.org/jira/browse/JBCACHE-1437
Project: JBoss Cache
Issue Type: Bug
Security Level: Public (Everyone can see)
Reporter: Elias Ross
Assignee: Manik Surtani
Created two cache instances with MVCC and async replication, at shutdown when both are shutdown simultaneously I get this:
61015 INFO [main] RPCManagerImpl - Disconnecting and closing the Channel
61047 INFO [Incoming-1,udp,192.168.0.9:2164] RPCManagerImpl - Received new cluster view: [192.168.0.9:2162|1] [192.168.0.9:2164]
61078 ERROR [Incoming-1,udp,192.168.0.9:2164] LockUtil - Exception breaking locks held by GlobalTransaction:<192.168.0.9:2162>:391
java.lang.IllegalMonitorStateException: org.jboss.cache.util.concurrent.locks.OwnableReentrantLock@19bd8b4[State = 1, empty queue][Locked by GlobalTransaction:<192.168.0.9:2162>:391]
at org.jboss.cache.util.concurrent.locks.OwnableReentrantLock.tryRelease(OwnableReentrantLock.java:139)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1137)
at org.jboss.cache.util.concurrent.locks.OwnableReentrantLock.unlock(OwnableReentrantLock.java:105)
at org.jboss.cache.lock.MVCCLockManager.unlock(MVCCLockManager.java:142)
at org.jboss.cache.lock.LockUtil.breakTransactionLock(LockUtil.java:172)
at org.jboss.cache.lock.LockUtil.breakTransactionLock(LockUtil.java:64)
at org.jboss.cache.RPCManagerImpl.removeLocksForDeadMembers(RPCManagerImpl.java:406)
at org.jboss.cache.RPCManagerImpl.removeLocksForDeadMembers(RPCManagerImpl.java:412)
at org.jboss.cache.RPCManagerImpl.removeLocksForDeadMembers(RPCManagerImpl.java:412)
at org.jboss.cache.RPCManagerImpl.removeLocksForDeadMembers(RPCManagerImpl.java:412)
at org.jboss.cache.RPCManagerImpl.access$500(RPCManagerImpl.java:85)
at org.jboss.cache.RPCManagerImpl$MembershipListenerAdaptor.viewAccepted(RPCManagerImpl.java:685)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:722)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:765)
at org.jgroups.JChannel.up(JChannel.java:1270)
at org.jgroups.mux.Multiplexer.passToMuxChannel(Multiplexer.java:1088)
at org.jgroups.mux.Multiplexer.handleServiceDown(Multiplexer.java:877)
at org.jgroups.mux.Multiplexer.handleServiceMessage(Multiplexer.java:778)
at org.jgroups.mux.Multiplexer.up(Multiplexer.java:303)
at org.jgroups.JChannel.up(JChannel.java:1270)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:444)
at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:144)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:192)
at org.jgroups.protocols.FC.up(FC.java:468)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:787)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:233)
at org.jgroups.protocols.UNICAST.up(UNICAST.java:299)
at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:861)
at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:693)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:167)
at org.jgroups.protocols.FD.up(FD.java:284)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:308)
at org.jgroups.protocols.MERGE2.up(MERGE2.java:144)
at org.jgroups.protocols.Discovery.up(Discovery.java:263)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1275)
at org.jgroups.protocols.TP.access$100(TP.java:49)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1818)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1792)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
61078 ERROR [Incoming-1,udp,192.168.0.9:2164] Multiplexer - failure in handling service message SERVICE_DOWN(MSID,192.168.0.9:2162) from sender 192.168.0.9:2162
java.lang.IllegalMonitorStateException: org.jboss.cache.util.concurrent.locks.OwnableReentrantLock@19bd8b4[State = 1, empty queue][Locked by GlobalTransaction:<192.168.0.9:2162>:391]
at org.jboss.cache.util.concurrent.locks.OwnableReentrantLock.tryRelease(OwnableReentrantLock.java:139)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1137)
at org.jboss.cache.util.concurrent.locks.OwnableReentrantLock.unlock(OwnableReentrantLock.java:105)
at org.jboss.cache.lock.MVCCLockManager.unlock(MVCCLockManager.java:142)
at org.jboss.cache.lock.LockUtil.breakTransactionLock(LockUtil.java:211)
at org.jboss.cache.lock.LockUtil.breakTransactionLock(LockUtil.java:64)
at org.jboss.cache.RPCManagerImpl.removeLocksForDeadMembers(RPCManagerImpl.java:406)
at org.jboss.cache.RPCManagerImpl.removeLocksForDeadMembers(RPCManagerImpl.java:412)
at org.jboss.cache.RPCManagerImpl.removeLocksForDeadMembers(RPCManagerImpl.java:412)
at org.jboss.cache.RPCManagerImpl.removeLocksForDeadMembers(RPCManagerImpl.java:412)
at org.jboss.cache.RPCManagerImpl.access$500(RPCManagerImpl.java:85)
at org.jboss.cache.RPCManagerImpl$MembershipListenerAdaptor.viewAccepted(RPCManagerImpl.java:685)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:722)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:765)
at org.jgroups.JChannel.up(JChannel.java:1270)
at org.jgroups.mux.Multiplexer.passToMuxChannel(Multiplexer.java:1088)
at org.jgroups.mux.Multiplexer.handleServiceDown(Multiplexer.java:877)
at org.jgroups.mux.Multiplexer.handleServiceMessage(Multiplexer.java:778)
at org.jgroups.mux.Multiplexer.up(Multiplexer.java:303)
at org.jgroups.JChannel.up(JChannel.java:1270)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:444)
at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:144)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:192)
at org.jgroups.protocols.FC.up(FC.java:468)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:787)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:233)
at org.jgroups.protocols.UNICAST.up(UNICAST.java:299)
at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:861)
at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:693)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:167)
at org.jgroups.protocols.FD.up(FD.java:284)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:308)
at org.jgroups.protocols.MERGE2.up(MERGE2.java:144)
at org.jgroups.protocols.Discovery.up(Discovery.java:263)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1275)
at org.jgroups.protocols.TP.access$100(TP.java:49)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1818)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1792)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
63015 WARN [main] Multiplexer - failed to collect all service ACKs (2) for [dst: <null>, src: 192.168.0.9:2162 (3 headers), size=0 bytes] after 2000ms, missing ACKs from [192.168.0.9:2164] (received=[192.168.0.9:2162]), local_addr=192.168.0.9:2162
63047 INFO [main] RPCManagerImpl - Stopping the RpcDispatcher
63172 DEBUG [main] JBossCacheService - JBossCacheService#1 STOPPED
63172 INFO [main] RPCManagerImpl - Disconnecting and closing the Channel
63203 INFO [main] RPCManagerImpl - Stopping the RpcDispatcher
63297 DEBUG [main] JBossCacheService - JBossCacheService#3 STOPPED
May be harmless but scary looking.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
16 years, 1 month
[JBoss JIRA] Created: (JBCACHE-1435) Remove on evict policy attempts to prune buddy backup tree
by Elias Ross (JIRA)
Remove on evict policy attempts to prune buddy backup tree
----------------------------------------------------------
Key: JBCACHE-1435
URL: https://jira.jboss.org/jira/browse/JBCACHE-1435
Project: JBoss Cache
Issue Type: Bug
Security Level: Public (Everyone can see)
Affects Versions: 3.0.0.CR2
Reporter: Elias Ross
Assignee: Manik Surtani
9797 DEBUG [EvictionTimer-0] DefaultEvictionActionPolicy - Unable to evict /_BUDDY_BACKUP_/10.31.248.143_2116/min/5005001/5005001042
org.jboss.cache.CacheException: Cannot make a backup Fqn from a backup Fqn! Attempting to create a backup of /_BUDDY_BACKUP_/10.31.248.143_2116/min/5005001/5005001042
at org.jboss.cache.buddyreplication.BuddyFqnTransformer.getBackupFqn(BuddyFqnTransformer.java:64)
at org.jboss.cache.buddyreplication.Fqn2BuddyFqnVisitor.getBackupFqn(Fqn2BuddyFqnVisitor.java:242)
at org.jboss.cache.buddyreplication.Fqn2BuddyFqnVisitor.visitRemoveNodeCommand(Fqn2BuddyFqnVisitor.java:110)
at org.jboss.cache.commands.write.RemoveNodeCommand.acceptVisitor(RemoveNodeCommand.java:125)
at org.jboss.cache.buddyreplication.BuddyManager.transformFqns(BuddyManager.java:727)
at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:149)
at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:135)
at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:107)
at org.jboss.cache.interceptors.ReplicationInterceptor.handleCrudMethod(ReplicationInterceptor.java:160)
at org.jboss.cache.interceptors.ReplicationInterceptor.visitRemoveNodeCommand(ReplicationInterceptor.java:125)
at org.jboss.cache.commands.write.RemoveNodeCommand.acceptVisitor(RemoveNodeCommand.java:125)
at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
at org.jboss.cache.commands.AbstractVisitor.visitRemoveNodeCommand(AbstractVisitor.java:75)
at org.jboss.cache.commands.write.RemoveNodeCommand.acceptVisitor(RemoveNodeCommand.java:125)
at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
at org.jboss.cache.interceptors.TxInterceptor.attachGtxAndPassUpChain(TxInterceptor.java:284)
at org.jboss.cache.interceptors.TxInterceptor.handleDefault(TxInterceptor.java:271)
at org.jboss.cache.commands.AbstractVisitor.visitRemoveNodeCommand(AbstractVisitor.java:75)
at org.jboss.cache.commands.write.RemoveNodeCommand.acceptVisitor(RemoveNodeCommand.java:125)
at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
at org.jboss.cache.commands.AbstractVisitor.visitRemoveNodeCommand(AbstractVisitor.java:75)
at org.jboss.cache.commands.write.RemoveNodeCommand.acceptVisitor(RemoveNodeCommand.java:125)
at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
at org.jboss.cache.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:178)
at org.jboss.cache.interceptors.InvocationContextInterceptor.visitRemoveNodeCommand(InvocationContextInterceptor.java:88)
at org.jboss.cache.commands.write.RemoveNodeCommand.acceptVisitor(RemoveNodeCommand.java:125)
at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
at org.jboss.cache.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:71)
at org.jboss.cache.commands.AbstractVisitor.visitRemoveNodeCommand(AbstractVisitor.java:75)
at org.jboss.cache.commands.write.RemoveNodeCommand.acceptVisitor(RemoveNodeCommand.java:125)
at org.jboss.cache.interceptors.InterceptorChain.invoke(InterceptorChain.java:265)
at org.jboss.cache.invocation.CacheInvocationDelegate.removeNode(CacheInvocationDelegate.java:482)
at org.jboss.cache.eviction.RemoveOnEvictActionPolicy.evict(RemoveOnEvictActionPolicy.java:49)
at org.jboss.cache.eviction.BaseEvictionAlgorithm.evictCacheNode(BaseEvictionAlgorithm.java:264)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
16 years, 1 month