Thanks to latest fixes from Dan I'm now actually able to complete the test run; I've run it some times and it doesn't seem to hang anymore! Also the results are encouraging, so here I am with renewed motivation to stress you all to cover the last mile :) Still some tests are failing, I guess there are all well known and being tracked? I'll share the details, so you can have a look; besides the failing tests, we also still have a little variation in the number of tests being reported. commit id of tested code: b8ad1c72e6f5d2c59050299e3c4fa9b6c127d606 Results : Failed tests: testStateTransfer(org.infinispan.container.versioning.VersionedReplStateTransferTest): Could not commit implicit transaction Tests run: 2586, Failures: 1, Errors: 0, Skipped: 0 Results : Failed tests: testStateTransfer(org.infinispan.container.versioning.VersionedReplStateTransferTest): Could not commit implicit transaction Tests run: 2585, Failures: 1, Errors: 0, Skipped: 0 Results : Failed tests: testStateTransfer(org.infinispan.container.versioning.VersionedReplStateTransferTest): Could not commit implicit transaction testJoinAndLeave(org.infinispan.statetransfer.DataRehashedEventTest): expected [2] but found [1] Tests run: 2586, Failures: 2, Errors: 0, Skipped: 0 Results : Failed tests: testStateTransfer(org.infinispan.container.versioning.VersionedReplStateTransferTest): Could not commit implicit transaction Tests run: 2586, Failures: 1, Errors: 0, Skipped: 0 Results : Failed tests: testStateTransfer(org.infinispan.container.versioning.VersionedReplStateTransferTest): Could not commit implicit transaction Tests run: 2586, Failures: 1, Errors: 0, Skipped: 0 Results : Failed tests: testStateTransfer(org.infinispan.container.versioning.VersionedReplStateTransferTest): Could not commit implicit transaction Tests run: 2585, Failures: 1, Errors: 0, Skipped: 0 Results : Failed tests: testStateTransfer(org.infinispan.container.versioning.VersionedReplStateTransferTest): Could not commit implicit transaction Tests run: 2584, Failures: 1, Errors: 0, Skipped: 0 Results : Failed tests: testPutRemove(org.infinispan.loaders.decorators.AsyncStoreTest): testPutRemove-k-780 still in store testStateTransfer(org.infinispan.container.versioning.VersionedReplStateTransferTest): Could not commit implicit transaction Tests run: 2586, Failures: 2, Errors: 0, Skipped: 0 Results : Failed tests: testStateTransfer(org.infinispan.container.versioning.VersionedReplStateTransferTest): Could not commit implicit transaction Tests run: 2586, Failures: 1, Errors: 0, Skipped: 0 Results : Failed tests: testWriteSkewWithOnlyPut(org.infinispan.api.mvcc.repeatable_read.WriteSkewTest): org.infinispan.transaction.WriteSkewException: Detected write skew. Tests run: 2585, Failures: 1, Errors: 0, Skipped: 0 In one case I got: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test (default-test) on project infinispan-core: Execution default-test of goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test failed: java.lang.reflect.InvocationTargetException; nested exception is java.lang.reflect.InvocationTargetException: null: NullPointerException -> [Help 1] scrolling up I couldn't relate it to any failed test (there is no report output); the only interesting stacktrace is the following; do we really need to kill the Marshaller, preventing it to send any final commands (and I guess a clean shutdown) ? 2013-03-02 17:31:34,854 ERROR [OutboundTransferTask] (asyncTransportThread-1,MainOwnerChangesLockTest-NodeF) Failed to send entries to node MainOwnerChangesLockTest-NodeE-53378 : java.lang.RuntimeException: Failure to marshal argument(s) org.infinispan.CacheException: java.lang.RuntimeException: Failure to marshal argument(s) at org.infinispan.util.Util.rewrapAsCacheException(Util.java:542) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:186) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:175) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:197) at org.infinispan.statetransfer.OutboundTransferTask.sendEntries(OutboundTransferTask.java:257) at org.infinispan.statetransfer.OutboundTransferTask.run(OutboundTransferTask.java:187) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.RuntimeException: Failure to marshal argument(s) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:281) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:300) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179) ... 14 more Caused by: java.lang.InterruptedException: Cache manager is shutting down, so type write externalizer for type=java.lang.Integer cannot be resolved. Interruption being pushed up. at org.infinispan.marshall.jboss.ExternalizerTable.getObjectWriter(ExternalizerTable.java:185) at org.infinispan.marshall.jboss.JBossMarshaller$ExternalizerTableProxy.getObjectWriter(JBossMarshaller.java:159) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:139) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119) at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:87) at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:128) at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:112) at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:73) at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119) at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:96) at org.infinispan.marshall.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:92) at org.infinispan.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:64) at org.infinispan.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:109) at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:45) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:279) ... 16 more Caused by: an exception which occurred: in object java.lang.Integer@6 in object org.infinispan.statetransfer.StateResponseCommand@58e3da4e Also I occasionally saw this, which looks like weird: 2013-03-02 17:44:00,457 ERROR [InvocationContextInterceptor] (testng-LockOwnerCrashPessimisticTest) ISPN000136: Execution error org.infinispan.util.concurrent.TimeoutException: Could not acquire lock on MagicKey#null{167a468a@LockOwnerCrashPessimisticTest-NodeL-10361} on behalf of transaction GlobalTransaction:<LockOwnerCrashPessimisticTest-NodeJ-32491>:146134:local. Lock is being held by null at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.newTimeoutException(AbstractTxLockingInterceptor.java:217) at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.waitForTransactionsToComplete(AbstractTxLockingInterceptor.java:210) at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockKeyAndCheckOwnership(AbstractTxLockingInterceptor.java:174) at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPutKeyValueCommand(PessimisticLockingInterceptor.java:122) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:251) at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:191) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:211) at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194) at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) I'm assuming these failures are all well known so I didn't collect details; please let me know if you'd like me to try getting some of these to reproduce again and which details I'd need to share.. and of course trace logs are not an option ;-) [warn: since it always fails me at infinispan-core, I never tested the other modules] Cheers, Sanne