11:20 a.m.
[
https://issues.jboss.org/browse/ISPN-8519?page=com.atlassian.jira.plugin....
]
Dan Berindei commented on ISPN-8519:
------------------------------------
Sorry, I thought the test was about cluster partitions, I see now you initially have 2
nodes and kill the coordinator. In some cases the new coordinator doesn't see the
leave command sent by the previous coordinator and assumes it crashed. Since the lock
cache had 2 members and it lost 1, it enters degraded mode.
I assume you don't actually want the lock cache to become degraded, there are at least
3 ways to avoid that:
# Don't use DENY_READ_WRITES
# Start 3 nodes, so that a majority stays alive after the coordinator is killed
# Kill the 2nd node instead of the coordinator, that way the cache doesn't enter
degraded mode (because the coordinator always knows that the other node was shut down and
is not running in a separate partition).
ClusteredLockWith2NodesKillingOneTest random failures
-----------------------------------------------------
Key: ISPN-8519
URL: https://issues.jboss.org/browse/ISPN-8519
Project: Infinispan
Issue Type: Bug
Components: Core, Test Suite - Core
Reporter: Dan Berindei
Assignee: Katia Aresti
Labels: testsuite_stability
Looks like the test should wait for the partitions to merge before destroying the lock?
{noformat}
java.lang.Error: java.util.concurrent.ExecutionException:
org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key
'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available.
Not all owners are in this partition
at org.infinispan.functional.FunctionalTestUtils.await(FunctionalTestUtils.java:44)
at
org.infinispan.lock.ClusteredLockWith2NodesKillingOneTest.destroyLock(ClusteredLockWith2NodesKillingOneTest.java:36)
Caused by: java.util.concurrent.ExecutionException:
org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key
'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available.
Not all owners are in this partition
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at org.infinispan.functional.FunctionalTestUtils.await(FunctionalTestUtils.java:42)
... 21 more
Caused by: org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key
'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available.
Not all owners are in this partition
at
org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.doCheck(PartitionHandlingManagerImpl.java:256)
at
org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.checkWrite(PartitionHandlingManagerImpl.java:103)
at
org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.handleSingleWrite(PartitionHandlingInterceptor.java:94)
at
org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.visitRemoveCommand(PartitionHandlingInterceptor.java:69)
at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:63)
at
org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:187)
at
org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:309)
at
org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:252)
at
org.infinispan.statetransfer.StateTransferInterceptor.visitRemoveCommand(StateTransferInterceptor.java:108)
at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:63)
at
org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58)
at
org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitRemoveCommand(CacheMgmtInterceptor.java:214)
at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:63)
at
org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:127)
at
org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:96)
at
org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:60)
at
org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
at
org.infinispan.interceptors.DDAsyncInterceptor.visitRemoveCommand(DDAsyncInterceptor.java:65)
at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:63)
at
org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50)
at
org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:234)
at
org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeededAsync(CacheImpl.java:1699)
at org.infinispan.cache.impl.CacheImpl.removeAsync(CacheImpl.java:1565)
at org.infinispan.cache.impl.DecoratedCache.removeAsync(DecoratedCache.java:400)
at
org.infinispan.cache.impl.AbstractDelegatingCache.removeAsync(AbstractDelegatingCache.java:229)
at org.infinispan.cache.impl.EncoderCache.removeAsync(EncoderCache.java:371)
at
org.infinispan.lock.impl.manager.EmbeddedClusteredLockManager.remove(EmbeddedClusteredLockManager.java:109)
... 21 more
... Removed 16 stack frames
{noformat}
{noformat}
java.lang.Error: java.util.concurrent.ExecutionException: java.lang.Error:
java.util.concurrent.ExecutionException:
org.infinispan.lock.exception.ClusteredLockException:
org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key
'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available.
Not all owners are in this partition
at org.infinispan.functional.FunctionalTestUtils.await(FunctionalTestUtils.java:44)
at
org.infinispan.lock.ClusteredLockWith2NodesKillingOneTest.testLockWithAcquisitionAndKill(ClusteredLockWith2NodesKillingOneTest.java:45)
Caused by: java.util.concurrent.ExecutionException: java.lang.Error:
java.util.concurrent.ExecutionException:
org.infinispan.lock.exception.ClusteredLockException:
org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key
'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available.
Not all owners are in this partition
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at org.infinispan.functional.FunctionalTestUtils.await(FunctionalTestUtils.java:42)
... 21 more
Caused by: java.lang.Error: java.util.concurrent.ExecutionException:
org.infinispan.lock.exception.ClusteredLockException:
org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key
'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available.
Not all owners are in this partition
at org.infinispan.functional.FunctionalTestUtils.await(FunctionalTestUtils.java:44)
at
org.infinispan.lock.ClusteredLockWith2NodesKillingOneTest.lambda$testLockWithAcquisitionAndKill$1(ClusteredLockWith2NodesKillingOneTest.java:48)
at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at
org.infinispan.lock.impl.lock.ClusteredLockImpl$TryLockRequestHolder.handle(ClusteredLockImpl.java:165)
at
org.infinispan.lock.impl.lock.ClusteredLockImpl$RequestHolder.handleLockResult(ClusteredLockImpl.java:106)
at
org.infinispan.lock.impl.lock.ClusteredLockImpl.lambda$tryLock$1(ClusteredLockImpl.java:294)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at
org.infinispan.interceptors.impl.QueueAsyncInvocationStage.invokeQueuedHandlers(QueueAsyncInvocationStage.java:106)
at
org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:81)
at
org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:30)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at org.infinispan.remoting.transport.AbstractRequest.complete(AbstractRequest.java:66)
at
org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:56)
at
org.infinispan.remoting.transport.impl.SingleTargetRequest.onResponse(SingleTargetRequest.java:35)
at
org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:53)
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1328)
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1238)
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:121)
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1366)
at org.jgroups.JChannel.up(JChannel.java:819)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:893)
at org.jgroups.protocols.RSVP.up(RSVP.java:163)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
at org.jgroups.protocols.tom.TOA.up(TOA.java:112)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
at org.jgroups.protocols.Discovery.up(Discovery.java:262)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1229)
at
org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
... 3 more
Caused by: java.util.concurrent.ExecutionException:
org.infinispan.lock.exception.ClusteredLockException:
org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key
'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available.
Not all owners are in this partition
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at org.infinispan.functional.FunctionalTestUtils.await(FunctionalTestUtils.java:42)
... 44 more
Caused by: org.infinispan.lock.exception.ClusteredLockException:
org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key
'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available.
Not all owners are in this partition
at
org.infinispan.lock.impl.lock.ClusteredLockImpl.handleException(ClusteredLockImpl.java:359)
at
org.infinispan.lock.impl.lock.ClusteredLockImpl.access$100(ClusteredLockImpl.java:54)
at
org.infinispan.lock.impl.lock.ClusteredLockImpl$RequestHolder.handleLockResult(ClusteredLockImpl.java:96)
at
org.infinispan.lock.impl.lock.ClusteredLockImpl.lambda$tryLock$1(ClusteredLockImpl.java:294)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
at
java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778)
at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140)
at org.infinispan.lock.impl.lock.ClusteredLockImpl.tryLock(ClusteredLockImpl.java:293)
at org.infinispan.lock.impl.lock.ClusteredLockImpl.tryLock(ClusteredLockImpl.java:283)
... 44 more
Caused by: org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key
'ClusteredLockKey{name=ClusteredLockWith2NodesKillingOneTest}' is not available.
Not all owners are in this partition
at
org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.doCheck(PartitionHandlingManagerImpl.java:256)
at
org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.checkWrite(PartitionHandlingManagerImpl.java:103)
at
org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.handleSingleWrite(PartitionHandlingInterceptor.java:94)
at
org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.visitReadWriteKeyCommand(PartitionHandlingInterceptor.java:89)
at
org.infinispan.commands.functional.ReadWriteKeyCommand.acceptVisitor(ReadWriteKeyCommand.java:102)
at
org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:187)
at
org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:309)
at
org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:252)
at
org.infinispan.statetransfer.StateTransferInterceptor.visitReadWriteKeyCommand(StateTransferInterceptor.java:162)
at
org.infinispan.commands.functional.ReadWriteKeyCommand.acceptVisitor(ReadWriteKeyCommand.java:102)
at
org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58)
at
org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
at
org.infinispan.interceptors.DDAsyncInterceptor.visitReadWriteKeyCommand(DDAsyncInterceptor.java:207)
at
org.infinispan.commands.functional.ReadWriteKeyCommand.acceptVisitor(ReadWriteKeyCommand.java:102)
at
org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:127)
at
org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:96)
at
org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:60)
at
org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
at
org.infinispan.interceptors.DDAsyncInterceptor.visitReadWriteKeyCommand(DDAsyncInterceptor.java:207)
at
org.infinispan.commands.functional.ReadWriteKeyCommand.acceptVisitor(ReadWriteKeyCommand.java:102)
at
org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50)
at
org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:234)
at
org.infinispan.functional.impl.AbstractFunctionalMap.invokeAsync(AbstractFunctionalMap.java:126)
at org.infinispan.functional.impl.ReadWriteMapImpl.eval(ReadWriteMapImpl.java:55)
... 46 more
... Removed 16 stack frames
{noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
2987
days inactive
2987
days old
infinispan-issues@lists.jboss.org
0 comments
1 participants
participants (1)
-
Dan Berindei (JIRA)