StreamDistPartitionHandlingTest.testUsingIteratorButPartitionOccursAfterRetrievingRemoteValues random failures -------------------------------------------------------------------------------------------------------------- Key: ISPN-11050 URL: https://issues.redhat.com/browse/ISPN-11050 Project: Infinispan Issue Type: Bug Components: Core, Test Suite Affects Versions: 10.1.0.Beta1 Reporter: Dan Berindei Assignee: Will Burns Priority: Major Labels: testsuite_stability Fix For: 11.0.0.Dev03, 10.1.6.Final Attachments: ISPN-6494_bundler_20191207-1109_StreamDistPartitionHandlingTest-infinispan-core.log.gz The test requests one entry from a stream iterator, then takes the cache to degraded mode, and assumes that the requesting the 2nd entry won't throw an {{AvailabilityException}} because the iterator already fetched the value in the background. However, even though the iterator requested all the segments in parallel, there is no guarantee that the originator also got the responses. If the test installs the partition views too soon, some segments will have to be retried, and they will fail: {noformat} 11:20:27,056 TRACE (testng-Test:[]) [JGroupsTransport] Test-NodeA-4361 sending single request 15 to Test-NodeC-45859: InitialPublisherCommand{cacheName='Test'} 11:20:27,058 TRACE (testng-Test:[]) [BasePartitionHandlingTest] Partition forming 11:20:27,066 TRACE (testng-Test:[]) [BasePartitionHandlingTest] Partition forming 11:20:27,066 TRACE (testng-Test:[]) [BasePartitionHandlingTest] Before installing new view... 11:20:27,068 TRACE (transport-thread-Test-NodeA-p48709-t6:[Topology-Test]) [CheckPoint] Waiting for event pre_released * 1 11:20:27,068 TRACE (jgroups-9,Test-NodeC-45859:[]) [JGroupsTransport] Test-NodeC-45859 sending response for request 15 to Test-NodeA-4361: SuccessfulResponse(PublisherResponse{size=0, completedSegments={6 12 15 31-33 36 40-41 44 50 53 56-57 75 78 85 89-92 97 103 113 122-123 127 130 135-136 148 173 188-189 192-193 221 224-226 238 241-242 253}, lostSegments=null, complete=true, segmentOffset=0}) 11:20:27,095 TRACE (testng-Test:[]) [BasePartitionHandlingTest] New views installed 11:20:27,096 TRACE (transport-thread-Test-NodeA-p48709-t6:[Topology-Test]) [CheckPoint] Received event post_released * 1 (available = 999999998, total = 999999999) 11:20:27,100 TRACE (transport-thread-Test-NodeA-p48709-t6:[Topology-Test]) [ClusterPublisherManagerImpl] Segments {3 6 11-12 15 22 26-29 31-37 39-44 47 50 52-57 62-63 75-78 81 85-86 89-92 97 103-105 113-115 120-125 127-128 130 135-136 141 148 155-156 172-174 188-194 197 200 205 209-210 221 224-226 229 234 237-238 241-243 245 248 253-254} not completed - retrying 11:20:27,102 TRACE (transport-thread-Test-NodeA-p48709-t6:[Topology-Test]) [JGroupsTransport] Test-NodeA-4361 sending single request 18 to Test-NodeC-45859: InitialPublisherCommand{cacheName='Test'} 11:20:27,100 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.partitionhandling.StreamDistPartitionHandlingTest.testUsingIteratorButPartitionOccursAfterRetrievingRemoteValues[DIST_SYNC, DENY_READ_WRITES] org.infinispan.partitionhandling.AvailabilityException: ISPN000305: Cluster is operating in degraded mode because of node failures. at org.infinispan.reactive.publisher.impl.PartitionAwareClusterPublisherManager$PartitionListener.onPartitionChange(PartitionAwareClusterPublisherManager.java:52) ~[classes/:?] at jdk.internal.reflect.GeneratedMethodAccessor336.invoke(Unknown Source) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.lambda$invoke$1(AbstractListenerImpl.java:426) ~[classes/:?] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:453) ~[classes/:?] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1854) ~[classes/:?] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1801) ~[classes/:?] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1754) ~[classes/:?] at org.infinispan.notifications.cachelistener.CacheNotifierImpl.doNotifyPartitionStatusChanged(CacheNotifierImpl.java:868) ~[classes/:?] at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyPartitionStatusChanged(CacheNotifierImpl.java:857) ~[classes/:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.mockito.internal.stubbing.defaultanswers.ForwardsInvocations.answer(ForwardsInvocations.java:48) ~[mockito-core-2.27.0.jar:?] at org.infinispan.test.Mocks.lambda$blockingAnswer$0(Mocks.java:156) ~[test-classes/:?] at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:39) ~[mockito-core-2.27.0.jar:?] at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:96) ~[mockito-core-2.27.0.jar:?] at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29) ~[mockito-core-2.27.0.jar:?] at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:35) ~[mockito-core-2.27.0.jar:?] at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:61) ~[mockito-core-2.27.0.jar:?] at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:49) ~[mockito-core-2.27.0.jar:?] at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor$DispatcherDefaultingToRealMethod.interceptAbstract(MockMethodInterceptor.java:126) ~[mockito-core-2.27.0.jar:?] at org.infinispan.notifications.cachelistener.CacheNotifier$MockitoMock$819492735.notifyPartitionStatusChanged(Unknown Source) ~[classes/:?] at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.lambda$setAvailabilityMode$0(PartitionHandlingManagerImpl.java:101) ~[classes/:?] at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106) ~[?:?] at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235) ~[?:?] at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:143) ~[?:?] at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.setAvailabilityMode(PartitionHandlingManagerImpl.java:99) ~[classes/:?] at org.infinispan.topology.LocalTopologyManagerImpl.doHandleTopologyUpdate(LocalTopologyManagerImpl.java:354) ~[classes/:?] at org.infinispan.topology.LocalTopologyManagerImpl.lambda$handleTopologyUpdate$1(LocalTopologyManagerImpl.java:286) ~[classes/:?] at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175) ~[classes/:?] at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:37) ~[classes/:?] at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:227) ~[classes/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] {noformat}