[infinispan-dev] fixing eviction with transactions (critical for Hibernate Search)

[Sanne Grinovero]

Hello all,
in this scenario we have the Infinispan Lucene Directory using
batching (DummyTransaction), eviction and passivation to keep the
amount of memory being used for the index under control; I'm using
LIRS but experienced the same issue with all other strategies.

As you can see from the following stacktrace, the batching ends by
sending a commit request, so the status of the transaction is 8
(STATUS_COMMITTING) in this context.
The new data is stored in the DataContainer, then the
BoundedConcurrentHashMap notifies the EvictionManagerImpl as it has to
evict some values, and this one attempts to acquire a lock on the
to-be-evicted keys (which are obviously not the same I'm trying to
store).
Acquiring this lock is an invalid operation as the transaction is in
commit state, and so this operation fails with an exception.

Thread [Hibernate Search: Directory writer-1] (Suspended (breakpoint
at line 92 in LockManagerImpl))
	LockManagerImpl.lockAndRecord(Object, InvocationContext) line: 92	
	EvictionManagerImpl.acquireLock(InvocationContext, Object) line: 210	
	EvictionManagerImpl.onEntryEviction(Object, InternalCacheEntry) line: 170	
	EvictionManagerImpl.onEntryEviction(Map<Object,InternalCacheEntry>) line: 162	
	DefaultDataContainer$DefaultEvictionListener.onEntryEviction(Map<Object,InternalCacheEntry>)
line: 201
	BoundedConcurrentHashMap$Segment<K,V>.notifyEvictionListener(Set<HashEntry<K,V>>)
line: 1176
	BoundedConcurrentHashMap$Segment<K,V>.put(K, int, V, boolean) line: 1011	
	BoundedConcurrentHashMap<K,V>.put(K, V) line: 1556	
	DefaultDataContainer.put(Object, Object, long, long) line: 148	
	ReadCommittedEntry.commit(DataContainer) line: 177	
	LockingInterceptor.commitEntry(CacheEntry, boolean) line: 389	
	LockingInterceptor.cleanupLocks(InvocationContext, boolean) line: 367	
	LockingInterceptor.visitCommitCommand(TxInvocationContext,
CommitCommand) line: 98
	CommitCommand.acceptVisitor(InvocationContext, Visitor) line: 60	
	CacheStoreInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext,
VisitableCommand) line: 119
	CacheStoreInterceptor.visitCommitCommand(TxInvocationContext,
CommitCommand) line: 148
	CommitCommand.acceptVisitor(InvocationContext, Visitor) line: 60	
	CacheLoaderInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext,
VisitableCommand) line: 119
	CacheLoaderInterceptor(CommandInterceptor).handleDefault(InvocationContext,
VisitableCommand) line: 133
	CacheLoaderInterceptor(AbstractVisitor).visitCommitCommand(TxInvocationContext,
CommitCommand) line: 116
	CommitCommand.acceptVisitor(InvocationContext, Visitor) line: 60	
	NotificationInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext,
VisitableCommand) line: 119
	NotificationInterceptor.visitCommitCommand(TxInvocationContext,
CommitCommand) line: 56
	CommitCommand.acceptVisitor(InvocationContext, Visitor) line: 60	
	TxInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext,
VisitableCommand) line: 119
	TxInterceptor.visitCommitCommand(TxInvocationContext, CommitCommand) line: 142	
	CommitCommand.acceptVisitor(InvocationContext, Visitor) line: 60	
	CacheMgmtInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext,
VisitableCommand) line: 119
	CacheMgmtInterceptor(CommandInterceptor).handleDefault(InvocationContext,
VisitableCommand) line: 133
	CacheMgmtInterceptor(AbstractVisitor).visitCommitCommand(TxInvocationContext,
CommitCommand) line: 116
	CommitCommand.acceptVisitor(InvocationContext, Visitor) line: 60	
	InvocationContextInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext,
VisitableCommand) line: 119
	InvocationContextInterceptor.handleAll(InvocationContext,
VisitableCommand) line: 96
	InvocationContextInterceptor.handleDefault(InvocationContext,
VisitableCommand) line: 63
	InvocationContextInterceptor(AbstractVisitor).visitCommitCommand(TxInvocationContext,
CommitCommand) line: 116
	CommitCommand.acceptVisitor(InvocationContext, Visitor) line: 60	
	BatchingInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext,
VisitableCommand) line: 119
	BatchingInterceptor.handleDefault(InvocationContext,
VisitableCommand) line: 77
	BatchingInterceptor(AbstractVisitor).visitCommitCommand(TxInvocationContext,
CommitCommand) line: 116
	CommitCommand.acceptVisitor(InvocationContext, Visitor) line: 60	
	InterceptorChain.invoke(InvocationContext, VisitableCommand) line: 274	
	TransactionCoordinator.commit(LocalTransaction, boolean) line: 136	
	TransactionXaAdapter.commit(Xid, boolean) line: 124	
	DummyTransaction.runCommitTx() line: 312	
	DummyTransaction.commit() line: 99	
	BatchModeTransactionManager(DummyBaseTransactionManager).commit() line: 97	
	BatchContainer.resolveTransaction(BatchContainer$BatchDetails,
boolean) line: 131
	BatchContainer.endBatch(boolean, boolean) line: 108	
	BatchContainer.endBatch(boolean) line: 93	
	CacheDelegate<K,V>.endBatch(boolean) line: 436	
	InfinispanIndexOutput.close() line: 208	
	IOUtils.closeSafely(Closeable...) line: 80	
	FieldsWriter.close() line: 111	
	StoredFieldsWriter.flush(SegmentWriteState) line: 52	
	DocFieldProcessor.flush(Collection<DocConsumerPerThread>,
SegmentWriteState) line: 58

I would like to remove the lock operation from the eviction listener,
but I'm not understanding why this locking is needed there and would
appreciate some explanations or help with this.
Shouldn't an evict operation be a "best effort" operation in all
cases, or is the idea here that we want the evictable data to be
consistently evicted on multiple nodes, or maybe even rollback an
evict operation?

Cheers,
Sanne