ClusteredLockWith2NodesKillingOneTest random failures ----------------------------------------------------- Key: ISPN-8519 URL: https://issues.jboss.org/browse/ISPN-8519 Project: Infinispan Issue Type: Bug Components: Core, Test Suite - Core Reporter: Dan Berindei Assignee: Katia Aresti Labels: testsuite_stability Looks like the test should wait for the partitions to merge before destroying the lock? {noformat} java.lang.Error: java.util.concurrent.ExecutionException: org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available. Not all owners are in this partition at org.infinispan.functional.FunctionalTestUtils.await(FunctionalTestUtils.java:44) at org.infinispan.lock.ClusteredLockWith2NodesKillingOneTest.destroyLock(ClusteredLockWith2NodesKillingOneTest.java:36) Caused by: java.util.concurrent.ExecutionException: org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available. Not all owners are in this partition at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at org.infinispan.functional.FunctionalTestUtils.await(FunctionalTestUtils.java:42) ... 21 more Caused by: org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available. Not all owners are in this partition at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.doCheck(PartitionHandlingManagerImpl.java:256) at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.checkWrite(PartitionHandlingManagerImpl.java:103) at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.handleSingleWrite(PartitionHandlingInterceptor.java:94) at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.visitRemoveCommand(PartitionHandlingInterceptor.java:69) at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:63) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:187) at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:309) at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:252) at org.infinispan.statetransfer.StateTransferInterceptor.visitRemoveCommand(StateTransferInterceptor.java:108) at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:63) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58) at org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitRemoveCommand(CacheMgmtInterceptor.java:214) at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:63) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:127) at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:96) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:60) at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) at org.infinispan.interceptors.DDAsyncInterceptor.visitRemoveCommand(DDAsyncInterceptor.java:65) at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:63) at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50) at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:234) at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeededAsync(CacheImpl.java:1699) at org.infinispan.cache.impl.CacheImpl.removeAsync(CacheImpl.java:1565) at org.infinispan.cache.impl.DecoratedCache.removeAsync(DecoratedCache.java:400) at org.infinispan.cache.impl.AbstractDelegatingCache.removeAsync(AbstractDelegatingCache.java:229) at org.infinispan.cache.impl.EncoderCache.removeAsync(EncoderCache.java:371) at org.infinispan.lock.impl.manager.EmbeddedClusteredLockManager.remove(EmbeddedClusteredLockManager.java:109) ... 21 more ... Removed 16 stack frames {noformat} {noformat} java.lang.Error: java.util.concurrent.ExecutionException: java.lang.Error: java.util.concurrent.ExecutionException: org.infinispan.lock.exception.ClusteredLockException: org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available. Not all owners are in this partition at org.infinispan.functional.FunctionalTestUtils.await(FunctionalTestUtils.java:44) at org.infinispan.lock.ClusteredLockWith2NodesKillingOneTest.testLockWithAcquisitionAndKill(ClusteredLockWith2NodesKillingOneTest.java:45) Caused by: java.util.concurrent.ExecutionException: java.lang.Error: java.util.concurrent.ExecutionException: org.infinispan.lock.exception.ClusteredLockException: org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available. Not all owners are in this partition at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at org.infinispan.functional.FunctionalTestUtils.await(FunctionalTestUtils.java:42) ... 21 more Caused by: java.lang.Error: java.util.concurrent.ExecutionException: org.infinispan.lock.exception.ClusteredLockException: org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available. Not all owners are in this partition at org.infinispan.functional.FunctionalTestUtils.await(FunctionalTestUtils.java:44) at org.infinispan.lock.ClusteredLockWith2NodesKillingOneTest.lambda$testLockWithAcquisitionAndKill$1(ClusteredLockWith2NodesKillingOneTest.java:48) at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656) at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.lock.impl.lock.ClusteredLockImpl$TryLockRequestHolder.handle(ClusteredLockImpl.java:165) at org.infinispan.lock.impl.lock.ClusteredLockImpl$RequestHolder.handleLockResult(ClusteredLockImpl.java:106) at org.infinispan.lock.impl.lock.ClusteredLockImpl.lambda$tryLock$1(ClusteredLockImpl.java:294) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.invokeQueuedHandlers(QueueAsyncInvocationStage.java:106) at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:81) at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:30) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.AbstractRequest.complete(AbstractRequest.java:66) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:56) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onResponse(SingleTargetRequest.java:35) at org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:53) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1328) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1238) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:121) at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1366) at org.jgroups.JChannel.up(JChannel.java:819) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:893) at org.jgroups.protocols.RSVP.up(RSVP.java:163) at org.jgroups.protocols.FlowControl.up(FlowControl.java:343) at org.jgroups.protocols.tom.TOA.up(TOA.java:112) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:864) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240) at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002) at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728) at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383) at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600) at org.jgroups.protocols.Discovery.up(Discovery.java:262) at org.jgroups.protocols.TP.passMessageUp(TP.java:1229) at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87) ... 3 more Caused by: java.util.concurrent.ExecutionException: org.infinispan.lock.exception.ClusteredLockException: org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available. Not all owners are in this partition at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at org.infinispan.functional.FunctionalTestUtils.await(FunctionalTestUtils.java:42) ... 44 more Caused by: org.infinispan.lock.exception.ClusteredLockException: org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available. Not all owners are in this partition at org.infinispan.lock.impl.lock.ClusteredLockImpl.handleException(ClusteredLockImpl.java:359) at org.infinispan.lock.impl.lock.ClusteredLockImpl.access$100(ClusteredLockImpl.java:54) at org.infinispan.lock.impl.lock.ClusteredLockImpl$RequestHolder.handleLockResult(ClusteredLockImpl.java:96) at org.infinispan.lock.impl.lock.ClusteredLockImpl.lambda$tryLock$1(ClusteredLockImpl.java:294) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778) at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140) at org.infinispan.lock.impl.lock.ClusteredLockImpl.tryLock(ClusteredLockImpl.java:293) at org.infinispan.lock.impl.lock.ClusteredLockImpl.tryLock(ClusteredLockImpl.java:283) ... 44 more Caused by: org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available. Not all owners are in this partition at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.doCheck(PartitionHandlingManagerImpl.java:256) at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.checkWrite(PartitionHandlingManagerImpl.java:103) at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.handleSingleWrite(PartitionHandlingInterceptor.java:94) at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.visitReadWriteKeyCommand(PartitionHandlingInterceptor.java:89) at org.infinispan.commands.functional.ReadWriteKeyCommand.acceptVisitor(ReadWriteKeyCommand.java:102) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:187) at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:309) at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:252) at org.infinispan.statetransfer.StateTransferInterceptor.visitReadWriteKeyCommand(StateTransferInterceptor.java:162) at org.infinispan.commands.functional.ReadWriteKeyCommand.acceptVisitor(ReadWriteKeyCommand.java:102) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58) at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) at org.infinispan.interceptors.DDAsyncInterceptor.visitReadWriteKeyCommand(DDAsyncInterceptor.java:207) at org.infinispan.commands.functional.ReadWriteKeyCommand.acceptVisitor(ReadWriteKeyCommand.java:102) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:127) at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:96) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:60) at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) at org.infinispan.interceptors.DDAsyncInterceptor.visitReadWriteKeyCommand(DDAsyncInterceptor.java:207) at org.infinispan.commands.functional.ReadWriteKeyCommand.acceptVisitor(ReadWriteKeyCommand.java:102) at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50) at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:234) at org.infinispan.functional.impl.AbstractFunctionalMap.invokeAsync(AbstractFunctionalMap.java:126) at org.infinispan.functional.impl.ReadWriteMapImpl.eval(ReadWriteMapImpl.java:55) ... 46 more ... Removed 16 stack frames {noformat}