[JBoss JIRA] Created: (JBCACHE-1473) Cache Regions lose capacity - EvictCommand.perform() causes dead entries in EvictionQueue
by Roberto Tyley (JIRA)
Cache Regions lose capacity - EvictCommand.perform() causes dead entries in EvictionQueue
-----------------------------------------------------------------------------------------
Key: JBCACHE-1473
URL: https://jira.jboss.org/jira/browse/JBCACHE-1473
Project: JBoss Cache
Issue Type: Bug
Security Level: Public (Everyone can see)
Components: Eviction
Affects Versions: 3.0.2.GA
Reporter: Roberto Tyley
Assignee: Manik Surtani
There is a serious bug in EvictCommand.perform() which leads to cache regions declining in effective capacity over time. The syndrome is:
* Cache fails to fill - node count initially reaches around 90% of configured maxNodes capacity
* Cache capacity declines over time - in our case down to 70% of capacity after 36 hours.
In order to manifest this issue, you need to be removing as well as adding nodes to the cache. The bug is exposed by a race condition between the Client Code thread removing a node and the Eviction Thread processing the EvictionEventQueue, but the race condition is not the problem (the race condition just explains why you don't see this issue all the time), the problem is with the semantics of the boolean return value of EvictCommand.perform().
Failure Case: EvictCommand.perform() tries to evict a node X from the cache, when X is no longer in the cache:
* the EvictCommand.perform() method returns the boolean 'false', attempting to indicate: "the EvictCommand has done no work"
* the EvictionInterceptor interprets the return value of 'false' to mean: "only data has been removed from the node X, and the node still exists in the cache".
As a consequence the EvictionInterceptor creates a new ADD_NODE_EVENT for X, to ensure that X will be revisited by the eviction thread in future, and adds the event to theEvictionQueue.
The EvictionQueue now contains a NodeEntry for a Node which is not in the cache. The EvictionAlgorithm will evict nodes based on how many nodes it *thinks* are in the cache (the length of its EvictionQueue), rather than how many nodes are /actually/ in the cache, so the cache can not grow to it's full capacity. These dead NodeEntries build up over time and can take up a substantial portion of your cache if you're frequently performing node removals.
The fix for this issue is thankfully simple: As only EvictionInterceptor reads the boolean value returned by EvictCommand.perform(), the semantics of the return value should be aligned with those of the EvictionInterceptor, and should simply return whether the node STILL exists in the cache after the evict has been performed, as that is what the EvictionInterceptor is really interested in.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
15 years, 10 months
[JBoss JIRA] Created: (JBCACHE-1471) Slow cache.get for fqn with a large number of key/value entries.
by Alex Kluge (JIRA)
Slow cache.get for fqn with a large number of key/value entries.
----------------------------------------------------------------
Key: JBCACHE-1471
URL: https://jira.jboss.org/jira/browse/JBCACHE-1471
Project: JBoss Cache
Issue Type: Patch
Security Level: Public (Everyone can see)
Affects Versions: 3.0.2.GA
Environment: Encountered when using a TcpDelegatingCacheLoader
Reporter: Alex Kluge
Assignee: Manik Surtani
Priority: Minor
Stress tests of JBoss Cache with a far cache, and a large number (10K+) of key/value pairs in a node
did not perform as well as I would expect. On a read of the cache, cache.get(fqn, key), a large amount
of time was spent determining whether the data needed to be loaded from the far cache.
The big time sink is the n. getKeysDirect() call in org.jboss.cache.interceptors.CacheLoaderInterceptor,
which copies the keyset. As expected, introducing a containsKeyDirect(K key) method to NodeSPI, and
implementation classes where needed, allowed for a significant speedup when the new method is used
in place of n.getKeysDirect().contains(key). If this is a reasonable path, I have attached the patch file to this
issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
15 years, 10 months