[infinispan-issues] [JBoss JIRA] (ISPN-2340) AsyncStore.remove() load the bucket from store inducing performance problem

AsyncStore.remove() load the bucket from store inducing performance problem --------------------------------------------------------------------------- Key: ISPN-2340 URL: https://issues.jboss.org/browse/ISPN-2340 Project: Infinispan Issue Type: Bug Components: Loaders and Stores Affects Versions: 5.2.0.Alpha4 Reporter: Christophe Domas Assignee: Galder Zamarreño Fix For: 5.2.0.CR1 I'm currently using infinispan-5.1.5 as a local persistent cache (like a persistent map) with no eviction and write-behind for maximum performance. I tried the 5.2.0alpha4 version and item processing mean decreased from 4ms to 60 ms. So I profiled the application and discovered that the Async.remove(Object) implementation differs from 5.2 to 5.1: +5.1+: {code} @Override public boolean remove(Object key) { enqueue(new Remove(key)); return true; } {code} +5.2+: {code} @Override public boolean remove(Object key) { try { InternalCacheEntry load = load(key); if (load != null) { enqueue(new Remove(key)); return true; } return false; } catch (CacheLoaderException e) { throw new CacheException("Could not load key/value entries from cacheloader", e); } } {code} So each time I remove an entry from my cache, I pay the cost of a bucket load from the store: {code} Thread [pool-9-thread-1] (Suspended) ExtendedRiverUnmarshaller(RiverUnmarshaller).doReadObject(boolean) line: 209 ExtendedRiverUnmarshaller(AbstractObjectInput).readObject() line: 37 JBossMarshaller(AbstractJBossMarshaller).objectFromObjectStream(ObjectInput) line: 163 VersionAwareMarshaller.objectFromObjectStream(ObjectInput) line: 190 CacheMarshaller(AbstractDelegatingMarshaller).objectFromObjectStream(ObjectInput) line: 79 FileCacheStore.objectFromInputStreamInReentrantMode(InputStream) line: 433 FileCacheStore.loadBucket(File) line: 306 FileCacheStore.loadBucket(Integer) line: 280 FileCacheStore(BucketBasedCacheStore).loadLockSafe(Object, Integer) line: 60 FileCacheStore(BucketBasedCacheStore).loadLockSafe(Object, Object) line: 49 FileCacheStore(LockSupportCacheStore<L>).load(Object) line: 142 AsyncStore(AbstractDelegatingStore).load(Object) line: 120 AsyncStore.remove(Object) line: 163 CacheStoreInterceptor.visitRemoveCommand(InvocationContext, RemoveCommand) line: 221 RemoveCommand.acceptVisitor(InvocationContext, Visitor) line: 72 CacheLoaderInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 118 CacheLoaderInterceptor.visitRemoveCommand(InvocationContext, RemoveCommand) line: 138 RemoveCommand.acceptVisitor(InvocationContext, Visitor) line: 72 EntryWrappingInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 118 EntryWrappingInterceptor.invokeNextAndApplyChanges(InvocationContext, FlagAffectedCommand) line: 211 EntryWrappingInterceptor.visitRemoveCommand(InvocationContext, RemoveCommand) line: 161 RemoveCommand.acceptVisitor(InvocationContext, Visitor) line: 72 NonTransactionalLockingInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 118 NonTransactionalLockingInterceptor.visitRemoveCommand(InvocationContext, RemoveCommand) line: 104 RemoveCommand.acceptVisitor(InvocationContext, Visitor) line: 72 NotificationInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 118 NotificationInterceptor(CommandInterceptor).handleDefault(InvocationContext, VisitableCommand) line: 132 NotificationInterceptor(AbstractVisitor).visitRemoveCommand(InvocationContext, RemoveCommand) line: 68 RemoveCommand.acceptVisitor(InvocationContext, Visitor) line: 72 IsMarshallableInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 118 IsMarshallableInterceptor(CommandInterceptor).handleDefault(InvocationContext, VisitableCommand) line: 132 IsMarshallableInterceptor(AbstractVisitor).visitRemoveCommand(InvocationContext, RemoveCommand) line: 68 IsMarshallableInterceptor.visitRemoveCommand(InvocationContext, RemoveCommand) line: 125 RemoveCommand.acceptVisitor(InvocationContext, Visitor) line: 72 InvocationContextInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 118 InvocationContextInterceptor.handleAll(InvocationContext, VisitableCommand) line: 129 InvocationContextInterceptor.handleDefault(InvocationContext, VisitableCommand) line: 93 InvocationContextInterceptor(AbstractVisitor).visitRemoveCommand(InvocationContext, RemoveCommand) line: 68 RemoveCommand.acceptVisitor(InvocationContext, Visitor) line: 72 InterceptorChain.invoke(InvocationContext, VisitableCommand) line: 347 CacheImpl<K,V>.executeCommandAndCommitIfNeeded(InvocationContext, VisitableCommand) line: 999 CacheImpl<K,V>.remove(Object, EnumSet<Flag>, ClassLoader) line: 290 DecoratedCache<K,V>.remove(Object) line: 324 CacheBidiMap<K,V>(AbstractDualBidiMap<K,V>).remove(Object) line: 199 OrderCache.removePendingByClientId(String) line: 168 OrderEventCacheManager(EventCacheManager).receiveAck(OrderId) line: 166 OrderEventCacheManager(EventCacheManager).cacheEventNotification(IamapiEvent, OrderId) line: 101 OrderMarketFixService.onOrderEvent(OrderEvent) line: 88 OrderEventDispatcher.fireEvent(OrderEvent) line: 52 OrderEventDispatcher.run() line: 67 ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1110 ThreadPoolExecutor$Worker.run() line: 603 Thread.run() line: 722 {code} My config file is: {code} <default> <loaders passivation="false" preload="true" shared="false"> <loader class="org.infinispan.loaders.file.FileCacheStore"> <properties> <property name="location" value="cache" /> <property name="fsyncMode" value="perWrite" /> </properties> <async enabled="true" /> </loader> </loaders> <expiration reaperEnabled="false" /> </default> {code} (I use perWrite for fsyncMode because we reached linux open files ulimit maximum) We cannot consider that as a bug, but this performance problem is a show-stopper for us to upgrade to 5.2. Best regards, Christophe