[infinispan-dev] fixing eviction with transactions (critical for Hibernate Search)

[Mircea Markus]

On 7 Jun 2011, at 13:13, Sanne Grinovero wrote:

> Hello all,
> in this scenario we have the Infinispan Lucene Directory using
> batching (DummyTransaction), eviction and passivation to keep the
> amount of memory being used for the index under control; I'm using
> LIRS but experienced the same issue with all other strategies.
> 
> As you can see from the following stacktrace, the batching ends by
> sending a commit request, so the status of the transaction is 8
> (STATUS_COMMITTING) in this context.
> The new data is stored in the DataContainer, then the
> BoundedConcurrentHashMap notifies the EvictionManagerImpl as it has to
> evict some values, and this one attempts to acquire a lock on the
> to-be-evicted keys (which are obviously not the same I'm trying to
> store).
> Acquiring this lock is an invalid operation as the transaction is in
> commit state, and so this operation fails with an exception.
> 
> Thread [Hibernate Search: Directory writer-1] (Suspended (breakpoint
> at line 92 in LockManagerImpl))
> 	LockManagerImpl.lockAndRecord(Object, InvocationContext) line: 92	
> 	EvictionManagerImpl.acquireLock(InvocationContext, Object) line: 210	
> 	EvictionManagerImpl.onEntryEviction(Object, InternalCacheEntry) line: 170	
> 	EvictionManagerImpl.onEntryEviction(Map<Object,InternalCacheEntry>) line: 162	
> 	DefaultDataContainer$DefaultEvictionListener.onEntryEviction(Map<Object,InternalCacheEntry>)
> line: 201
> 	BoundedConcurrentHashMap$Segment<K,V>.notifyEvictionListener(Set<HashEntry<K,V>>)
> line: 1176
> 	BoundedConcurrentHashMap$Segment<K,V>.put(K, int, V, boolean) line: 1011	
> 	BoundedConcurrentHashMap<K,V>.put(K, V) line: 1556	
> 	DefaultDataContainer.put(Object, Object, long, long) line: 148	
> 	ReadCommittedEntry.commit(DataContainer) line: 177	
> 	LockingInterceptor.commitEntry(CacheEntry, boolean) line: 389	
> 	LockingInterceptor.cleanupLocks(InvocationContext, boolean) line: 367	
> 	LockingInterceptor.visitCommitCommand(TxInvocationContext,
> CommitCommand) line: 98
> 	CommitCommand.acceptVisitor(InvocationContext, Visitor) line: 60	
> 	CacheStoreInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext,
> VisitableCommand) line: 119
> 	CacheStoreInterceptor.visitCommitCommand(TxInvocationContext,
> CommitCommand) line: 148
> 	CommitCommand.acceptVisitor(InvocationContext, Visitor) line: 60	
> 	CacheLoaderInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext,
> VisitableCommand) line: 119
> 	CacheLoaderInterceptor(CommandInterceptor).handleDefault(InvocationContext,
> VisitableCommand) line: 133
> 	CacheLoaderInterceptor(AbstractVisitor).visitCommitCommand(TxInvocationContext,
> CommitCommand) line: 116
> 	CommitCommand.acceptVisitor(InvocationContext, Visitor) line: 60	
> 	NotificationInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext,
> VisitableCommand) line: 119
> 	NotificationInterceptor.visitCommitCommand(TxInvocationContext,
> CommitCommand) line: 56
> 	CommitCommand.acceptVisitor(InvocationContext, Visitor) line: 60	
> 	TxInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext,
> VisitableCommand) line: 119
> 	TxInterceptor.visitCommitCommand(TxInvocationContext, CommitCommand) line: 142	
> 	CommitCommand.acceptVisitor(InvocationContext, Visitor) line: 60	
> 	CacheMgmtInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext,
> VisitableCommand) line: 119
> 	CacheMgmtInterceptor(CommandInterceptor).handleDefault(InvocationContext,
> VisitableCommand) line: 133
> 	CacheMgmtInterceptor(AbstractVisitor).visitCommitCommand(TxInvocationContext,
> CommitCommand) line: 116
> 	CommitCommand.acceptVisitor(InvocationContext, Visitor) line: 60	
> 	InvocationContextInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext,
> VisitableCommand) line: 119
> 	InvocationContextInterceptor.handleAll(InvocationContext,
> VisitableCommand) line: 96
> 	InvocationContextInterceptor.handleDefault(InvocationContext,
> VisitableCommand) line: 63
> 	InvocationContextInterceptor(AbstractVisitor).visitCommitCommand(TxInvocationContext,
> CommitCommand) line: 116
> 	CommitCommand.acceptVisitor(InvocationContext, Visitor) line: 60	
> 	BatchingInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext,
> VisitableCommand) line: 119
> 	BatchingInterceptor.handleDefault(InvocationContext,
> VisitableCommand) line: 77
> 	BatchingInterceptor(AbstractVisitor).visitCommitCommand(TxInvocationContext,
> CommitCommand) line: 116
> 	CommitCommand.acceptVisitor(InvocationContext, Visitor) line: 60	
> 	InterceptorChain.invoke(InvocationContext, VisitableCommand) line: 274	
> 	TransactionCoordinator.commit(LocalTransaction, boolean) line: 136	
> 	TransactionXaAdapter.commit(Xid, boolean) line: 124	
> 	DummyTransaction.runCommitTx() line: 312	
> 	DummyTransaction.commit() line: 99	
> 	BatchModeTransactionManager(DummyBaseTransactionManager).commit() line: 97	
> 	BatchContainer.resolveTransaction(BatchContainer$BatchDetails,
> boolean) line: 131
> 	BatchContainer.endBatch(boolean, boolean) line: 108	
> 	BatchContainer.endBatch(boolean) line: 93	
> 	CacheDelegate<K,V>.endBatch(boolean) line: 436	
> 	InfinispanIndexOutput.close() line: 208	
> 	IOUtils.closeSafely(Closeable...) line: 80	
> 	FieldsWriter.close() line: 111	
> 	StoredFieldsWriter.flush(SegmentWriteState) line: 52	
> 	DocFieldProcessor.flush(Collection<DocConsumerPerThread>,
> SegmentWriteState) line: 58
> 
> I would like to remove the lock operation from the eviction listener,
> but I'm not understanding why this locking is needed there and would
> appreciate some explanations or help with this.
Looking at the code, my first thought was that it is needed for sync-ing cache store /passivator access on that key.
But afaik the cache store takes care of key locking at its own level[1], so I think that should be remove entirely.

[1] https://github.com/mmarkus/infinispan/blob/master/core/src/main/java/org/infinispan/loaders/LockSupportCacheStore.java
> Shouldn't an evict operation be a "best effort" operation in all
> cases,
+1. Even if my statement above doesn't stand and there's a reason to keep the lock for consistency, it should be a best effort: try to lock with 0 timeout, if fails just ignore it and move on. 
> or is the idea here that we want the evictable data to be
> consistently evicted on multiple nodes, or maybe even rollback an
> evict operation?
We don't advertise cluster-wise eviction consistency, so I doubt that is the case. 
> 
> Cheers,
> Sanne
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20110607/21883033/attachment-0001.html