[infinispan-issues] [JBoss JIRA] (ISPN-7133) NPE in BoundedEquivalentConcurrentHashMapV8 at load

[ https://issues.jboss.org/browse/ISPN-7133?page=com.atlassian.jira.plugin.... ] Gregory Ramsperger updated ISPN-7133: ------------------------------------- Description: During performance tests, we have encountered a small but consistent level of NullPointerExceptions in BoundedEquivalentConcurrentHashMapV8$LIRSEvictionPolicy.findIfEntriesNeedEvicting. In our system, we have multiple servers acting as a sort of registry. Infinispan is configured with a with a Redis cache store Items are added, removed, and read frequently. We were handling around 8k puts/min, 8k removes per minute, and about 1k gets per minute. We see 0-5 failures per minute on the cache.get calls with the stack below (mostly 0 with sporadic failures). The relevant code can be [seen on mainline|https://github.com/infinispan/infinispan/blob/master/commons/src...]. The node is null when `node.val` is called. {code:java|title=BoundedEquivalentConcurrentHashMapV8.java|linenumbers=true|firstline=1530} } else if (evict.state == Recency.HIR_NONRESIDENT) { Node<K, V> node = f.find(hash, evict.getKey()); V prevValue = node.val; if (prevValue != NULL_VALUE) { node.val = (V) NULL_VALUE; map.addCount(-1, -1); Node<K, V> nonResidentNode = new Node<K, V>(-1, null, evict.getKey(), prevValue, null); removedNodes.add(nonResidentNode); map.notifyListenerOfRemoval(nonResidentNode, true); } } {code} I believe this just needs a {{node != null}} check. h3. Backtrace {noformat} 2016-10-18 03:37:46,678 i-baa20229 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (EE-ManagedExecutorService-LechmereDefault-Thread-1) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.NullPointerException at org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8$LIRSEvictionPolicy.findIfEntriesNeedEvicting(BoundedEquivalentConcurrentHashMapV8.java:1532) [infinispan-commons-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8.compute(BoundedEquivalentConcurrentHashMapV8.java:3658) [infinispan-commons-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.container.DefaultDataContainer.compute(DefaultDataContainer.java:323) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.persistence.PersistenceUtil.loadAndStoreInDataContainer(PersistenceUtil.java:91) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.CacheLoaderInterceptor.loadInContext(CacheLoaderInterceptor.java:367) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.CacheLoaderInterceptor.loadIfNeeded(CacheLoaderInterceptor.java:362) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.CacheLoaderInterceptor.visitDataCommand(CacheLoaderInterceptor.java:183) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.CacheLoaderInterceptor.visitGetKeyValueCommand(CacheLoaderInterceptor.java:137) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.EntryWrappingInterceptor.visitDataReadCommand(EntryWrappingInterceptor.java:133) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.EntryWrappingInterceptor.visitGetKeyValueCommand(EntryWrappingInterceptor.java:123) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataReadCommand(NonTransactionalLockingInterceptor.java:31) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:77) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.CacheMgmtInterceptor.visitDataReadCommand(CacheMgmtInterceptor.java:103) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:91) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:76) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:411) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:403) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:286) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at our code {noformat} h3. Our cache config (boiled down): {code:java|title=Cache Config} ConfigurationBuilder configBuilder = new ConfigurationBuilder(); configBuilder .locking() .useLockStriping(false) .isolationLevel(IsolationLevel.READ_COMMITTED) .concurrencyLevel(12) .eviction() .strategy(EvictionStrategy.LIRS) .type(EvictionType.COUNT).size(10000) .clustering() .cacheMode(CacheMode.INVALIDATION_SYNC) .persistence() .passivation(false) .addStore(RedisStoreConfigurationBuilder.class) .connectionPool() .minIdle(6) .maxIdle(10) .maxTotal(20) .minEvictableIdleTime(30000) .timeBetweenEvictionRuns(30000) .topology(Topology.SERVER) .fetchPersistentState(true) .database(1) .addServer() .host("127.0.0.1") .port(REDIS_PORT) .socketTimeout(5000) .connectionTimeout(5000); {code} was: During performance tests, we have encountered a small but consistent level of NullPointerExceptions in BoundedEquivalentConcurrentHashMapV8$LIRSEvictionPolicy.findIfEntriesNeedEvicting. In our system, we have multiple servers acting as a sort of registry. Infinispan is configured with a with a Redis cache store Items are added, removed, and read frequently. We were handling around 8k puts/min, 8k removes per minute, and about 1k gets per minute. We see 0-5 failures per minute on the cache.get calls with the stack below (mostly 0 with sporadic failures). The relevant code can be [seen on mainline|https://github.com/infinispan/infinispan/blob/master/commons/src...]. The node is null when `node.val` is called. {code:java|title=BoundedEquivalentConcurrentHashMapV8.java|linenumbers=true|firstline=1530} } else if (evict.state == Recency.HIR_NONRESIDENT) { Node<K, V> node = f.find(hash, evict.getKey()); V prevValue = node.val; if (prevValue != NULL_VALUE) { node.val = (V) NULL_VALUE; map.addCount(-1, -1); Node<K, V> nonResidentNode = new Node<K, V>(-1, null, evict.getKey(), prevValue, null); removedNodes.add(nonResidentNode); map.notifyListenerOfRemoval(nonResidentNode, true); } } {code} I believe this just needs a null check. h3. Backtrace {noformat} 2016-10-18 03:37:46,678 i-baa20229 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (EE-ManagedExecutorService-LechmereDefault-Thread-1) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.NullPointerException at org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8$LIRSEvictionPolicy.findIfEntriesNeedEvicting(BoundedEquivalentConcurrentHashMapV8.java:1532) [infinispan-commons-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8.compute(BoundedEquivalentConcurrentHashMapV8.java:3658) [infinispan-commons-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.container.DefaultDataContainer.compute(DefaultDataContainer.java:323) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.persistence.PersistenceUtil.loadAndStoreInDataContainer(PersistenceUtil.java:91) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.CacheLoaderInterceptor.loadInContext(CacheLoaderInterceptor.java:367) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.CacheLoaderInterceptor.loadIfNeeded(CacheLoaderInterceptor.java:362) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.CacheLoaderInterceptor.visitDataCommand(CacheLoaderInterceptor.java:183) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.CacheLoaderInterceptor.visitGetKeyValueCommand(CacheLoaderInterceptor.java:137) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.EntryWrappingInterceptor.visitDataReadCommand(EntryWrappingInterceptor.java:133) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.EntryWrappingInterceptor.visitGetKeyValueCommand(EntryWrappingInterceptor.java:123) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataReadCommand(NonTransactionalLockingInterceptor.java:31) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:77) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.CacheMgmtInterceptor.visitDataReadCommand(CacheMgmtInterceptor.java:103) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:91) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:76) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:411) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:403) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:286) [infinispan-core-8.1.0.Final.jar:8.1.0.Final] at our code {noformat} h3. Our cache config (boiled down): {code:java|title=Cache Config} ConfigurationBuilder configBuilder = new ConfigurationBuilder(); configBuilder .locking() .useLockStriping(false) .isolationLevel(IsolationLevel.READ_COMMITTED) .concurrencyLevel(12) .eviction() .strategy(EvictionStrategy.LIRS) .type(EvictionType.COUNT).size(10000) .clustering() .cacheMode(CacheMode.INVALIDATION_SYNC) .persistence() .passivation(false) .addStore(RedisStoreConfigurationBuilder.class) .connectionPool() .minIdle(6) .maxIdle(10) .maxTotal(20) .minEvictableIdleTime(30000) .timeBetweenEvictionRuns(30000) .topology(Topology.SERVER) .fetchPersistentState(true) .database(1) .addServer() .host("127.0.0.1") .port(REDIS_PORT) .socketTimeout(5000) .connectionTimeout(5000); {code} -- This message was sent by Atlassian JIRA (v6.4.11#64026)