[jbosscache-issues] [JBoss JIRA] Resolved: (JBCACHE-1473) Cache Regions lose capacity - EvictCommand.perform() causes dead entries in EvictionQueue

Cache Regions lose capacity - EvictCommand.perform() causes dead entries in EvictionQueue ----------------------------------------------------------------------------------------- Key: JBCACHE-1473 URL: https://jira.jboss.org/jira/browse/JBCACHE-1473 Project: JBoss Cache Issue Type: Bug Security Level: Public(Everyone can see) Components: Eviction Affects Versions: 3.0.2.GA Reporter: Roberto Tyley Assignee: Manik Surtani Priority: Critical Fix For: 3.1.0.GA Attachments: JBCACHE-1473.FIX.patch, JBCACHE-1473.INTEGRATION-TEST.zip, JBCACHE-1473.TEST.patch There is a serious bug in EvictCommand.perform() which leads to cache regions declining in effective capacity over time. The syndrome is: * Cache fails to fill - node count initially reaches around 90% of configured maxNodes capacity * Cache capacity declines over time - in our case down to 70% of capacity after 36 hours. In order to manifest this issue, you need to be removing as well as adding nodes to the cache. The bug is exposed by a race condition between the Client Code thread removing a node and the Eviction Thread processing the EvictionEventQueue, but the race condition is not the problem (the race condition just explains why you don't see this issue all the time), the problem is with the semantics of the boolean return value of EvictCommand.perform(). Failure Case: EvictCommand.perform() tries to evict a node X from the cache, when X is no longer in the cache: * the EvictCommand.perform() method returns the boolean 'false', attempting to indicate: "the EvictCommand has done no work" * the EvictionInterceptor interprets the return value of 'false' to mean: "only data has been removed from the node X, and the node still exists in the cache". As a consequence the EvictionInterceptor creates a new ADD_NODE_EVENT for X, to ensure that X will be revisited by the eviction thread in future, and adds the event to theEvictionQueue. The EvictionQueue now contains a NodeEntry for a Node which is not in the cache. The EvictionAlgorithm will evict nodes based on how many nodes it *thinks* are in the cache (the length of its EvictionQueue), rather than how many nodes are /actually/ in the cache, so the cache can not grow to it's full capacity. These dead NodeEntries build up over time and can take up a substantial portion of your cache if you're frequently performing node removals. The fix for this issue is thankfully simple: As only EvictionInterceptor reads the boolean value returned by EvictCommand.perform(), the semantics of the return value should be aligned with those of the EvictionInterceptor, and should simply return whether the node STILL exists in the cache after the evict has been performed, as that is what the EvictionInterceptor is really interested in.