TransactionTable does not shutdown gracefully --------------------------------------------- Key: ISPN-9701 URL: https://issues.jboss.org/browse/ISPN-9701 Project: Infinispan Issue Type: Bug Components: Transactions Affects Versions: 9.4.1.Final Reporter: Paul Ferraro Assignee: Dan Berindei Priority: Critical Here's a sample stacktrace during shutdown: {noformat} 16:54:15,033 WARN [org.wildfly.clustering.web.undertow] (default task-1) ISPN000472: Cache manager is stopping: org.infinispan.IllegalLifecycleStateException: ISPN000472: Cache manager is stopping at org.infinispan.marshall.core.GlobalMarshaller.getExternalizer(GlobalMarshaller.java:420) at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:400) at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355) at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183) at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176) at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1009) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1209) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performAsyncRemoteInvocation(JGroupsTransport.java:1105) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:246) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:291) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:323) at org.infinispan.transaction.impl.TransactionTable.removeTransactionInfoRemotely(TransactionTable.java:900) at org.infinispan.transaction.impl.TransactionTable.releaseLocksForCompletedTransaction(TransactionTable.java:886) at org.infinispan.transaction.xa.XaTransactionTable.forgetSuccessfullyCompletedTransaction(XaTransactionTable.java:195) at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:128) at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68) at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:419) at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:466) at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335) at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110) {noformat} The problem seems to be that shutDownGracefully() first waits for the localTransactions map to be empty. However, when the cache is clustered, releaseLocksForCompletedTransaction(...) removes the transaction from the localTransactions map *before* invoking removeTransactionInfoRemotely(...), which means that the subsequent TxCompletionNotificationCommand can fail to marshal (see above), or the transport might close before this command is sent. A naive fix would simply reorder the removeLocalTransaction(...) to happen after the call to removeTransactionInfoRemotely(...) within the releaseLocksForCompletedTransaction(...) method, but I'm sure there's more to it.