11:53 a.m.
Thanks to latest fixes from Dan I'm now actually able to complete the
test run; I've run it some times and it doesn't seem to hang anymore!
Also the results are encouraging, so here I am with renewed motivation
to stress you all to cover the last mile :)
Still some tests are failing, I guess there are all well known and
being tracked?
I'll share the details, so you can have a look; besides the failing
tests, we also still have a little variation in the number of tests
being reported.
commit id of tested code: b8ad1c72e6f5d2c59050299e3c4fa9b6c127d606
Results :
Failed tests:
testStateTransfer(org.infinispan.container.versioning.VersionedReplStateTransferTest):
Could not commit implicit transaction
Tests run: 2586, Failures: 1, Errors: 0, Skipped: 0
Results :
Failed tests:
testStateTransfer(org.infinispan.container.versioning.VersionedReplStateTransferTest):
Could not commit implicit transaction
Tests run: 2585, Failures: 1, Errors: 0, Skipped: 0
Results :
Failed tests:
testStateTransfer(org.infinispan.container.versioning.VersionedReplStateTransferTest):
Could not commit implicit transaction
testJoinAndLeave(org.infinispan.statetransfer.DataRehashedEventTest):
expected [2] but found [1]
Tests run: 2586, Failures: 2, Errors: 0, Skipped: 0
Results :
Failed tests:
testStateTransfer(org.infinispan.container.versioning.VersionedReplStateTransferTest):
Could not commit implicit transaction
Tests run: 2586, Failures: 1, Errors: 0, Skipped: 0
Results :
Failed tests:
testStateTransfer(org.infinispan.container.versioning.VersionedReplStateTransferTest):
Could not commit implicit transaction
Tests run: 2586, Failures: 1, Errors: 0, Skipped: 0
Results :
Failed tests:
testStateTransfer(org.infinispan.container.versioning.VersionedReplStateTransferTest):
Could not commit implicit transaction
Tests run: 2585, Failures: 1, Errors: 0, Skipped: 0
Results :
Failed tests:
testStateTransfer(org.infinispan.container.versioning.VersionedReplStateTransferTest):
Could not commit implicit transaction
Tests run: 2584, Failures: 1, Errors: 0, Skipped: 0
Results :
Failed tests:
testPutRemove(org.infinispan.loaders.decorators.AsyncStoreTest):
testPutRemove-k-780 still in store
testStateTransfer(org.infinispan.container.versioning.VersionedReplStateTransferTest):
Could not commit implicit transaction
Tests run: 2586, Failures: 2, Errors: 0, Skipped: 0
Results :
Failed tests:
testStateTransfer(org.infinispan.container.versioning.VersionedReplStateTransferTest):
Could not commit implicit transaction
Tests run: 2586, Failures: 1, Errors: 0, Skipped: 0
Results :
Failed tests:
testWriteSkewWithOnlyPut(org.infinispan.api.mvcc.repeatable_read.WriteSkewTest):
org.infinispan.transaction.WriteSkewException: Detected write skew.
Tests run: 2585, Failures: 1, Errors: 0, Skipped: 0
In one case I got:
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test
(default-test) on project infinispan-core: Execution default-test of
goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test
failed: java.lang.reflect.InvocationTargetException; nested exception
is java.lang.reflect.InvocationTargetException: null:
NullPointerException -> [Help 1]
scrolling up I couldn't relate it to any failed test (there is no
report output); the only interesting stacktrace is the following; do
we really need to kill the Marshaller, preventing it to send any final
commands (and I guess a clean shutdown) ?
2013-03-02 17:31:34,854 ERROR [OutboundTransferTask]
(asyncTransportThread-1,MainOwnerChangesLockTest-NodeF) Failed to send
entries to node MainOwnerChangesLockTest-NodeE-53378 :
java.lang.RuntimeException: Failure to marshal argument(s)
org.infinispan.CacheException: java.lang.RuntimeException: Failure to
marshal argument(s)
at org.infinispan.util.Util.rewrapAsCacheException(Util.java:542)
at
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:186)
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515)
at
org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:175)
at
org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:197)
at
org.infinispan.statetransfer.OutboundTransferTask.sendEntries(OutboundTransferTask.java:257)
at
org.infinispan.statetransfer.OutboundTransferTask.run(OutboundTransferTask.java:187)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: Failure to marshal argument(s)
at
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:281)
at
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:300)
at
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179)
... 14 more
Caused by: java.lang.InterruptedException: Cache manager is shutting
down, so type write externalizer for type=java.lang.Integer cannot be
resolved. Interruption being pushed up.
at
org.infinispan.marshall.jboss.ExternalizerTable.getObjectWriter(ExternalizerTable.java:185)
at
org.infinispan.marshall.jboss.JBossMarshaller$ExternalizerTableProxy.getObjectWriter(JBossMarshaller.java:159)
at
org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:139)
at
org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
at
org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
at
org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:87)
at
org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:128)
at
org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:112)
at
org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:73)
at
org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408)
at
org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
at
org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
at
org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
at
org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:96)
at
org.infinispan.marshall.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:92)
at
org.infinispan.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:64)
at
org.infinispan.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:109)
at
org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:45)
at
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:279)
... 16 more
Caused by: an exception which occurred:
in object java.lang.Integer@6
in object org.infinispan.statetransfer.StateResponseCommand@58e3da4e
Also I occasionally saw this, which looks like weird:
2013-03-02 17:44:00,457 ERROR [InvocationContextInterceptor]
(testng-LockOwnerCrashPessimisticTest) ISPN000136: Execution error
org.infinispan.util.concurrent.TimeoutException: Could not acquire
lock on MagicKey#null{167a468a@LockOwnerCrashPessimisticTest-NodeL-10361}
on behalf of transaction
GlobalTransaction:<LockOwnerCrashPessimisticTest-NodeJ-32491>:146134:local.
Lock is being held by null
at
org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.newTimeoutException(AbstractTxLockingInterceptor.java:217)
at
org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.waitForTransactionsToComplete(AbstractTxLockingInterceptor.java:210)
at
org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockKeyAndCheckOwnership(AbstractTxLockingInterceptor.java:174)
at
org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPutKeyValueCommand(PessimisticLockingInterceptor.java:122)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:251)
at
org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:191)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:211)
at
org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194)
at
org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
I'm assuming these failures are all well known so I didn't collect
details; please let me know if you'd like me to try getting some of
these to reproduce again and which details I'd need to share.. and of
course trace logs are not an option ;-)
[warn: since it always fails me at infinispan-core, I never tested the
other modules]
Cheers,
Sanne