[infinispan-issues] [JBoss JIRA] (ISPN-2340) AsyncStore.remove() load the bucket from store inducing performance problem

[Christophe Domas (JIRA)]

Christophe Domas created ISPN-2340:
--------------------------------------

             Summary: AsyncStore.remove() load the bucket from store inducing performance problem
                 Key: ISPN-2340
                 URL: https://issues.jboss.org/browse/ISPN-2340
             Project: Infinispan
          Issue Type: Bug
          Components: Loaders and Stores
    Affects Versions: 5.2.0.Alpha4
            Reporter: Christophe Domas
            Assignee: Mircea Markus


I'm currently using infinispan-5.1.5 as a local persistent cache (like a persistent map) with no eviction and write-behind for maximum performance.

I tried the 5.2.0alpha4 version and item processing mean decreased from 4ms to 60 ms. 
So I profiled the application and discovered that the Async.remove(Object) implementation differs from 5.2 to 5.1:

+5.1+:
{code}
   @Override
   public boolean remove(Object key) {
      enqueue(new Remove(key));
      return true;
   }
{code}

+5.2+:
{code}
   @Override
   public boolean remove(Object key) {
      try {
         InternalCacheEntry load = load(key);
         if (load != null) {
            enqueue(new Remove(key));
            return true;
         }
         return false;
      } catch (CacheLoaderException e) {
         throw new CacheException("Could not load key/value entries from cacheloader", e);
      }
   }
{code}

So each time I remove an entry from my cache, I pay the cost of a bucket load from the store:

{code}
Thread [pool-9-thread-1] (Suspended)	
	ExtendedRiverUnmarshaller(RiverUnmarshaller).doReadObject(boolean) line: 209	
	ExtendedRiverUnmarshaller(AbstractObjectInput).readObject() line: 37	
	JBossMarshaller(AbstractJBossMarshaller).objectFromObjectStream(ObjectInput) line: 163	
	VersionAwareMarshaller.objectFromObjectStream(ObjectInput) line: 190	
	CacheMarshaller(AbstractDelegatingMarshaller).objectFromObjectStream(ObjectInput) line: 79	
	FileCacheStore.objectFromInputStreamInReentrantMode(InputStream) line: 433	
	FileCacheStore.loadBucket(File) line: 306	
	FileCacheStore.loadBucket(Integer) line: 280	
	FileCacheStore(BucketBasedCacheStore).loadLockSafe(Object, Integer) line: 60	
	FileCacheStore(BucketBasedCacheStore).loadLockSafe(Object, Object) line: 49	
	FileCacheStore(LockSupportCacheStore<L>).load(Object) line: 142	
	AsyncStore(AbstractDelegatingStore).load(Object) line: 120	
	AsyncStore.remove(Object) line: 163	
	CacheStoreInterceptor.visitRemoveCommand(InvocationContext, RemoveCommand) line: 221	
	RemoveCommand.acceptVisitor(InvocationContext, Visitor) line: 72	
	CacheLoaderInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 118	
	CacheLoaderInterceptor.visitRemoveCommand(InvocationContext, RemoveCommand) line: 138	
	RemoveCommand.acceptVisitor(InvocationContext, Visitor) line: 72	
	EntryWrappingInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 118	
	EntryWrappingInterceptor.invokeNextAndApplyChanges(InvocationContext, FlagAffectedCommand) line: 211	
	EntryWrappingInterceptor.visitRemoveCommand(InvocationContext, RemoveCommand) line: 161	
	RemoveCommand.acceptVisitor(InvocationContext, Visitor) line: 72	
	NonTransactionalLockingInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 118	
	NonTransactionalLockingInterceptor.visitRemoveCommand(InvocationContext, RemoveCommand) line: 104	
	RemoveCommand.acceptVisitor(InvocationContext, Visitor) line: 72	
	NotificationInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 118	
	NotificationInterceptor(CommandInterceptor).handleDefault(InvocationContext, VisitableCommand) line: 132	
	NotificationInterceptor(AbstractVisitor).visitRemoveCommand(InvocationContext, RemoveCommand) line: 68	
	RemoveCommand.acceptVisitor(InvocationContext, Visitor) line: 72	
	IsMarshallableInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 118	
	IsMarshallableInterceptor(CommandInterceptor).handleDefault(InvocationContext, VisitableCommand) line: 132	
	IsMarshallableInterceptor(AbstractVisitor).visitRemoveCommand(InvocationContext, RemoveCommand) line: 68	
	IsMarshallableInterceptor.visitRemoveCommand(InvocationContext, RemoveCommand) line: 125	
	RemoveCommand.acceptVisitor(InvocationContext, Visitor) line: 72	
	InvocationContextInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 118	
	InvocationContextInterceptor.handleAll(InvocationContext, VisitableCommand) line: 129	
	InvocationContextInterceptor.handleDefault(InvocationContext, VisitableCommand) line: 93	
	InvocationContextInterceptor(AbstractVisitor).visitRemoveCommand(InvocationContext, RemoveCommand) line: 68	
	RemoveCommand.acceptVisitor(InvocationContext, Visitor) line: 72	
	InterceptorChain.invoke(InvocationContext, VisitableCommand) line: 347	
	CacheImpl<K,V>.executeCommandAndCommitIfNeeded(InvocationContext, VisitableCommand) line: 999	
	CacheImpl<K,V>.remove(Object, EnumSet<Flag>, ClassLoader) line: 290	
	DecoratedCache<K,V>.remove(Object) line: 324	
	CacheBidiMap<K,V>(AbstractDualBidiMap<K,V>).remove(Object) line: 199	
	OrderCache.removePendingByClientId(String) line: 168	
	OrderEventCacheManager(EventCacheManager).receiveAck(OrderId) line: 166	
	OrderEventCacheManager(EventCacheManager).cacheEventNotification(IamapiEvent, OrderId) line: 101	
	OrderMarketFixService.onOrderEvent(OrderEvent) line: 88	
	OrderEventDispatcher.fireEvent(OrderEvent) line: 52	
	OrderEventDispatcher.run() line: 67	
	ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1110	
	ThreadPoolExecutor$Worker.run() line: 603	
	Thread.run() line: 722	

{code}

My config file is: 
{code}
	<default>
		<loaders passivation="false" preload="true" shared="false">
			<loader class="org.infinispan.loaders.file.FileCacheStore">
				<properties>
					<property name="location" value="cache" />
					<property name="fsyncMode" value="perWrite" />
				</properties>
				<async enabled="true" />
			</loader>
		</loaders>
		<expiration reaperEnabled="false" />
	</default>
{code}

(I use perWrite for fsyncMode because we reached linux open files ulimit maximum)

We cannot consider that as a bug, but this performance problem is a show-stopper for us to upgrade to 5.2.

Best regards,

Christophe




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira