[infinispan-issues] [JBoss JIRA] (ISPN-8519) ClusteredLockWith2NodesKillingOneTest random failures
Dan Berindei (JIRA)
issues at jboss.org
Thu Nov 9 12:20:00 EST 2017
[ https://issues.jboss.org/browse/ISPN-8519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488336#comment-13488336 ]
Dan Berindei commented on ISPN-8519:
------------------------------------
Sorry, I thought the test was about cluster partitions, I see now you initially have 2 nodes and kill the coordinator. In some cases the new coordinator doesn't see the leave command sent by the previous coordinator and assumes it crashed. Since the lock cache had 2 members and it lost 1, it enters degraded mode.
I assume you don't actually want the lock cache to become degraded, there are at least 3 ways to avoid that:
# Don't use DENY_READ_WRITES
# Start 3 nodes, so that a majority stays alive after the coordinator is killed
# Kill the 2nd node instead of the coordinator, that way the cache doesn't enter degraded mode (because the coordinator always knows that the other node was shut down and is not running in a separate partition).
> ClusteredLockWith2NodesKillingOneTest random failures
> -----------------------------------------------------
>
> Key: ISPN-8519
> URL: https://issues.jboss.org/browse/ISPN-8519
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite - Core
> Reporter: Dan Berindei
> Assignee: Katia Aresti
> Labels: testsuite_stability
>
> Looks like the test should wait for the partitions to merge before destroying the lock?
> {noformat}
> java.lang.Error: java.util.concurrent.ExecutionException: org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available. Not all owners are in this partition
> at org.infinispan.functional.FunctionalTestUtils.await(FunctionalTestUtils.java:44)
> at org.infinispan.lock.ClusteredLockWith2NodesKillingOneTest.destroyLock(ClusteredLockWith2NodesKillingOneTest.java:36)
> Caused by: java.util.concurrent.ExecutionException: org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available. Not all owners are in this partition
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at org.infinispan.functional.FunctionalTestUtils.await(FunctionalTestUtils.java:42)
> ... 21 more
> Caused by: org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available. Not all owners are in this partition
> at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.doCheck(PartitionHandlingManagerImpl.java:256)
> at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.checkWrite(PartitionHandlingManagerImpl.java:103)
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.handleSingleWrite(PartitionHandlingInterceptor.java:94)
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.visitRemoveCommand(PartitionHandlingInterceptor.java:69)
> at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:63)
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:187)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:309)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:252)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitRemoveCommand(StateTransferInterceptor.java:108)
> at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:63)
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58)
> at org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitRemoveCommand(CacheMgmtInterceptor.java:214)
> at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:63)
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:127)
> at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:96)
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:60)
> at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> at org.infinispan.interceptors.DDAsyncInterceptor.visitRemoveCommand(DDAsyncInterceptor.java:65)
> at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:63)
> at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50)
> at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:234)
> at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeededAsync(CacheImpl.java:1699)
> at org.infinispan.cache.impl.CacheImpl.removeAsync(CacheImpl.java:1565)
> at org.infinispan.cache.impl.DecoratedCache.removeAsync(DecoratedCache.java:400)
> at org.infinispan.cache.impl.AbstractDelegatingCache.removeAsync(AbstractDelegatingCache.java:229)
> at org.infinispan.cache.impl.EncoderCache.removeAsync(EncoderCache.java:371)
> at org.infinispan.lock.impl.manager.EmbeddedClusteredLockManager.remove(EmbeddedClusteredLockManager.java:109)
> ... 21 more
> ... Removed 16 stack frames
> {noformat}
> {noformat}
> java.lang.Error: java.util.concurrent.ExecutionException: java.lang.Error: java.util.concurrent.ExecutionException: org.infinispan.lock.exception.ClusteredLockException: org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available. Not all owners are in this partition
> at org.infinispan.functional.FunctionalTestUtils.await(FunctionalTestUtils.java:44)
> at org.infinispan.lock.ClusteredLockWith2NodesKillingOneTest.testLockWithAcquisitionAndKill(ClusteredLockWith2NodesKillingOneTest.java:45)
> Caused by: java.util.concurrent.ExecutionException: java.lang.Error: java.util.concurrent.ExecutionException: org.infinispan.lock.exception.ClusteredLockException: org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available. Not all owners are in this partition
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at org.infinispan.functional.FunctionalTestUtils.await(FunctionalTestUtils.java:42)
> ... 21 more
> Caused by: java.lang.Error: java.util.concurrent.ExecutionException: org.infinispan.lock.exception.ClusteredLockException: org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available. Not all owners are in this partition
> at org.infinispan.functional.FunctionalTestUtils.await(FunctionalTestUtils.java:44)
> at org.infinispan.lock.ClusteredLockWith2NodesKillingOneTest.lambda$testLockWithAcquisitionAndKill$1(ClusteredLockWith2NodesKillingOneTest.java:48)
> at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
> at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.lock.impl.lock.ClusteredLockImpl$TryLockRequestHolder.handle(ClusteredLockImpl.java:165)
> at org.infinispan.lock.impl.lock.ClusteredLockImpl$RequestHolder.handleLockResult(ClusteredLockImpl.java:106)
> at org.infinispan.lock.impl.lock.ClusteredLockImpl.lambda$tryLock$1(ClusteredLockImpl.java:294)
> at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
> at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.invokeQueuedHandlers(QueueAsyncInvocationStage.java:106)
> at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:81)
> at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:30)
> at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
> at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.AbstractRequest.complete(AbstractRequest.java:66)
> at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:56)
> at org.infinispan.remoting.transport.impl.SingleTargetRequest.onResponse(SingleTargetRequest.java:35)
> at org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:53)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1328)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1238)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:121)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1366)
> at org.jgroups.JChannel.up(JChannel.java:819)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:893)
> at org.jgroups.protocols.RSVP.up(RSVP.java:163)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
> at org.jgroups.protocols.tom.TOA.up(TOA.java:112)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
> at org.jgroups.protocols.Discovery.up(Discovery.java:262)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1229)
> at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
> ... 3 more
> Caused by: java.util.concurrent.ExecutionException: org.infinispan.lock.exception.ClusteredLockException: org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available. Not all owners are in this partition
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at org.infinispan.functional.FunctionalTestUtils.await(FunctionalTestUtils.java:42)
> ... 44 more
> Caused by: org.infinispan.lock.exception.ClusteredLockException: org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available. Not all owners are in this partition
> at org.infinispan.lock.impl.lock.ClusteredLockImpl.handleException(ClusteredLockImpl.java:359)
> at org.infinispan.lock.impl.lock.ClusteredLockImpl.access$100(ClusteredLockImpl.java:54)
> at org.infinispan.lock.impl.lock.ClusteredLockImpl$RequestHolder.handleLockResult(ClusteredLockImpl.java:96)
> at org.infinispan.lock.impl.lock.ClusteredLockImpl.lambda$tryLock$1(ClusteredLockImpl.java:294)
> at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
> at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778)
> at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140)
> at org.infinispan.lock.impl.lock.ClusteredLockImpl.tryLock(ClusteredLockImpl.java:293)
> at org.infinispan.lock.impl.lock.ClusteredLockImpl.tryLock(ClusteredLockImpl.java:283)
> ... 44 more
> Caused by: org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available. Not all owners are in this partition
> at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.doCheck(PartitionHandlingManagerImpl.java:256)
> at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.checkWrite(PartitionHandlingManagerImpl.java:103)
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.handleSingleWrite(PartitionHandlingInterceptor.java:94)
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.visitReadWriteKeyCommand(PartitionHandlingInterceptor.java:89)
> at org.infinispan.commands.functional.ReadWriteKeyCommand.acceptVisitor(ReadWriteKeyCommand.java:102)
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:187)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:309)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:252)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitReadWriteKeyCommand(StateTransferInterceptor.java:162)
> at org.infinispan.commands.functional.ReadWriteKeyCommand.acceptVisitor(ReadWriteKeyCommand.java:102)
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58)
> at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> at org.infinispan.interceptors.DDAsyncInterceptor.visitReadWriteKeyCommand(DDAsyncInterceptor.java:207)
> at org.infinispan.commands.functional.ReadWriteKeyCommand.acceptVisitor(ReadWriteKeyCommand.java:102)
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:127)
> at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:96)
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:60)
> at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> at org.infinispan.interceptors.DDAsyncInterceptor.visitReadWriteKeyCommand(DDAsyncInterceptor.java:207)
> at org.infinispan.commands.functional.ReadWriteKeyCommand.acceptVisitor(ReadWriteKeyCommand.java:102)
> at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50)
> at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:234)
> at org.infinispan.functional.impl.AbstractFunctionalMap.invokeAsync(AbstractFunctionalMap.java:126)
> at org.infinispan.functional.impl.ReadWriteMapImpl.eval(ReadWriteMapImpl.java:55)
> ... 46 more
> ... Removed 16 stack frames
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
More information about the infinispan-issues
mailing list