[JBoss JIRA] (ISPN-11489) TransactionCoordinator updated for non blocking
by Dan Berindei (Jira)
[ https://issues.redhat.com/browse/ISPN-11489?page=com.atlassian.jira.plugi... ]
Dan Berindei commented on ISPN-11489:
-------------------------------------
Random failures in {{StaleLocksWithCommitDuringStateTransferTest}} are related.
> TransactionCoordinator updated for non blocking
> -----------------------------------------------
>
> Key: ISPN-11489
> URL: https://issues.redhat.com/browse/ISPN-11489
> Project: Infinispan
> Issue Type: Sub-task
> Reporter: Will Burns
> Assignee: Will Burns
> Priority: Major
> Fix For: 11.0.0.Dev04
>
>
> TransactionCoordinator is utilized when working with transactions. Unfortunately due to transaction API being inherently blocking, we need to make some exceptions to how this is handled.
> # TransactionCoordinator should only be invokable by a blocking method, the inverse of the PersistenceManagerImpl
> # We should invoke all the non blocking code of TransactionCoordinator in a non blocking thread and then block on its completion in the blocking thread.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years
[JBoss JIRA] (ISPN-11569) DistributedStreamIteratorTest.testIterationDuringInitialTransfer random failures
by Dan Berindei (Jira)
Dan Berindei created ISPN-11569:
-----------------------------------
Summary: DistributedStreamIteratorTest.testIterationDuringInitialTransfer random failures
Key: ISPN-11569
URL: https://issues.redhat.com/browse/ISPN-11569
Project: Infinispan
Issue Type: Bug
Components: Core, Streams, Test Suite
Affects Versions: 11.0.0.Dev03
Reporter: Dan Berindei
Assignee: Dan Berindei
Fix For: 11.0.0.Dev04
Very similar to ISPN-11518, {{DistributedStreamIteratorTest.testIterationDuringInitialTransfer}} blocks {{StateProvider.startOutboundTransfer()}}, and with the recent non-blocking changes it also blocks the {{StateTransferStartCommand}} response, and the invocation of {{StateTransferLockImpl.transactionDataReceived()}} on the node that sent the {{StateTransferStartCommand}}.
{noformat}
14:44:11,141 TRACE (non-blocking-thread-Test-NodeA-p24343-t1:[]) [StateTransferLockImpl] Signalling topology 15 is installed
14:44:11,147 TRACE (non-blocking-thread-Test-NodeA-p24343-t1:[]) [TrianglePerCacheInboundInvocationHandler] Calling perform() on StateTransferStartCommand{topologyId=15, segments={0}, cacheName=testCache}
14:44:11,150 TRACE (non-blocking-thread-Test-NodeA-p24343-t1:[]) [CheckPoint] Triggering event pre_invoked * 1 (available = 1, total = 1)
14:44:11,150 TRACE (non-blocking-thread-Test-NodeA-p24343-t1:[]) [CheckPoint] Waiting for event pre_released * 1
14:44:11,163 TRACE (testng-Test:[]) [JGroupsTransport] Test-NodeD sending single request 9 to Test-NodeA: InitialPublisherCommand{cacheName='testCache'}
14:44:11,164 TRACE (jgroups-9,Test-NodeA:[]) [JGroupsTransport] Test-NodeA received request 9 from Test-NodeD: InitialPublisherCommand{cacheName='testCache'}
14:44:11,164 TRACE (jgroups-9,Test-NodeA:[]) [StateTransferLockImpl] Waiting for transaction data for topology 15, current topology is 14
14:44:31,155 WARN (non-blocking-thread-Test-NodeA-p24343-t1:[]) [CLUSTER] ISPN000071: Caught exception when handling command StateTransferStartCommand{topologyId=15, segments={0}, cacheName=testCache}
java.util.concurrent.TimeoutException: Timed out waiting for event pre_released
at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:50) ~[test-classes/:?]
at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:40) ~[test-classes/:?]
at org.infinispan.test.Mocks.lambda$blockingAnswer$0(Mocks.java:156) ~[test-classes/:?]
at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:39) ~[mockito-core-2.27.0.jar:?]
at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:96) ~[mockito-core-2.27.0.jar:?]
at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29) ~[mockito-core-2.27.0.jar:?]
at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:35) ~[mockito-core-2.27.0.jar:?]
at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:61) ~[mockito-core-2.27.0.jar:?]
at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:49) ~[mockito-core-2.27.0.jar:?]
at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor$DispatcherDefaultingToRealMethod.interceptAbstract(MockMethodInterceptor.java:126) ~[mockito-core-2.27.0.jar:?]
at org.infinispan.statetransfer.StateProvider$MockitoMock$193392642.startOutboundTransfer(Unknown Source) ~[classes/:?]
at org.infinispan.commands.statetransfer.StateTransferStartCommand.invokeAsync(StateTransferStartCommand.java:37) ~[classes/:?]
14:44:31,180 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.stream.DistributedStreamIteratorTest.testIterationDuringInitialTransfer[DIST_SYNC, tx=false]
java.lang.IllegalStateException: Thread already timed out waiting for event pre_released
at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:177) ~[test-classes/:?]
at org.infinispan.test.fwk.CheckPoint.triggerForever(CheckPoint.java:166) ~[test-classes/:?]
at org.infinispan.stream.DistributedStreamIteratorTest.testIterationDuringInitialTransfer(DistributedStreamIteratorTest.java:115) ~[test-classes/:?]
{noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years
[JBoss JIRA] (ISPN-11050) StreamDistPartitionHandlingTest.testUsingIteratorButPartitionOccursAfterRetrievingRemoteValues random failures
by Will Burns (Jira)
[ https://issues.redhat.com/browse/ISPN-11050?page=com.atlassian.jira.plugi... ]
Will Burns reopened ISPN-11050:
-------------------------------
> StreamDistPartitionHandlingTest.testUsingIteratorButPartitionOccursAfterRetrievingRemoteValues random failures
> --------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-11050
> URL: https://issues.redhat.com/browse/ISPN-11050
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite
> Affects Versions: 10.1.0.Beta1
> Reporter: Dan Berindei
> Assignee: Will Burns
> Priority: Major
> Labels: testsuite_stability
> Fix For: 11.0.0.Dev03, 10.1.6.Final
>
> Attachments: ISPN-6494_bundler_20191207-1109_StreamDistPartitionHandlingTest-infinispan-core.log.gz
>
>
> The test requests one entry from a stream iterator, then takes the cache to degraded mode, and assumes that the requesting the 2nd entry won't throw an {{AvailabilityException}} because the iterator already fetched the value in the background.
> However, even though the iterator requested all the segments in parallel, there is no guarantee that the originator also got the responses. If the test installs the partition views too soon, some segments will have to be retried, and they will fail:
> {noformat}
> 11:20:27,056 TRACE (testng-Test:[]) [JGroupsTransport] Test-NodeA-4361 sending single request 15 to Test-NodeC-45859: InitialPublisherCommand{cacheName='Test'}
> 11:20:27,058 TRACE (testng-Test:[]) [BasePartitionHandlingTest] Partition forming
> 11:20:27,066 TRACE (testng-Test:[]) [BasePartitionHandlingTest] Partition forming
> 11:20:27,066 TRACE (testng-Test:[]) [BasePartitionHandlingTest] Before installing new view...
> 11:20:27,068 TRACE (transport-thread-Test-NodeA-p48709-t6:[Topology-Test]) [CheckPoint] Waiting for event pre_released * 1
> 11:20:27,068 TRACE (jgroups-9,Test-NodeC-45859:[]) [JGroupsTransport] Test-NodeC-45859 sending response for request 15 to Test-NodeA-4361: SuccessfulResponse(PublisherResponse{size=0, completedSegments={6 12 15 31-33 36 40-41 44 50 53 56-57 75 78 85 89-92 97 103 113 122-123 127 130 135-136 148 173 188-189 192-193 221 224-226 238 241-242 253}, lostSegments=null, complete=true, segmentOffset=0})
> 11:20:27,095 TRACE (testng-Test:[]) [BasePartitionHandlingTest] New views installed
> 11:20:27,096 TRACE (transport-thread-Test-NodeA-p48709-t6:[Topology-Test]) [CheckPoint] Received event post_released * 1 (available = 999999998, total = 999999999)
> 11:20:27,100 TRACE (transport-thread-Test-NodeA-p48709-t6:[Topology-Test]) [ClusterPublisherManagerImpl] Segments {3 6 11-12 15 22 26-29 31-37 39-44 47 50 52-57 62-63 75-78 81 85-86 89-92 97 103-105 113-115 120-125 127-128 130 135-136 141 148 155-156 172-174 188-194 197 200 205 209-210 221 224-226 229 234 237-238 241-243 245 248 253-254} not completed - retrying
> 11:20:27,102 TRACE (transport-thread-Test-NodeA-p48709-t6:[Topology-Test]) [JGroupsTransport] Test-NodeA-4361 sending single request 18 to Test-NodeC-45859: InitialPublisherCommand{cacheName='Test'}
> 11:20:27,100 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.partitionhandling.StreamDistPartitionHandlingTest.testUsingIteratorButPartitionOccursAfterRetrievingRemoteValues[DIST_SYNC, DENY_READ_WRITES]
> org.infinispan.partitionhandling.AvailabilityException: ISPN000305: Cluster is operating in degraded mode because of node failures.
> at org.infinispan.reactive.publisher.impl.PartitionAwareClusterPublisherManager$PartitionListener.onPartitionChange(PartitionAwareClusterPublisherManager.java:52) ~[classes/:?]
> at jdk.internal.reflect.GeneratedMethodAccessor336.invoke(Unknown Source) ~[?:?]
> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.lambda$invoke$1(AbstractListenerImpl.java:426) ~[classes/:?]
> at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:453) ~[classes/:?]
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1854) ~[classes/:?]
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1801) ~[classes/:?]
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1754) ~[classes/:?]
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl.doNotifyPartitionStatusChanged(CacheNotifierImpl.java:868) ~[classes/:?]
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyPartitionStatusChanged(CacheNotifierImpl.java:857) ~[classes/:?]
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at org.mockito.internal.stubbing.defaultanswers.ForwardsInvocations.answer(ForwardsInvocations.java:48) ~[mockito-core-2.27.0.jar:?]
> at org.infinispan.test.Mocks.lambda$blockingAnswer$0(Mocks.java:156) ~[test-classes/:?]
> at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:39) ~[mockito-core-2.27.0.jar:?]
> at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:96) ~[mockito-core-2.27.0.jar:?]
> at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29) ~[mockito-core-2.27.0.jar:?]
> at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:35) ~[mockito-core-2.27.0.jar:?]
> at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:61) ~[mockito-core-2.27.0.jar:?]
> at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:49) ~[mockito-core-2.27.0.jar:?]
> at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor$DispatcherDefaultingToRealMethod.interceptAbstract(MockMethodInterceptor.java:126) ~[mockito-core-2.27.0.jar:?]
> at org.infinispan.notifications.cachelistener.CacheNotifier$MockitoMock$819492735.notifyPartitionStatusChanged(Unknown Source) ~[classes/:?]
> at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.lambda$setAvailabilityMode$0(PartitionHandlingManagerImpl.java:101) ~[classes/:?]
> at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106) ~[?:?]
> at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235) ~[?:?]
> at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:143) ~[?:?]
> at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.setAvailabilityMode(PartitionHandlingManagerImpl.java:99) ~[classes/:?]
> at org.infinispan.topology.LocalTopologyManagerImpl.doHandleTopologyUpdate(LocalTopologyManagerImpl.java:354) ~[classes/:?]
> at org.infinispan.topology.LocalTopologyManagerImpl.lambda$handleTopologyUpdate$1(LocalTopologyManagerImpl.java:286) ~[classes/:?]
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175) ~[classes/:?]
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:37) ~[classes/:?]
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:227) ~[classes/:?]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
> at java.lang.Thread.run(Thread.java:834) [?:?]
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years
[JBoss JIRA] (ISPN-11050) StreamDistPartitionHandlingTest.testUsingIteratorButPartitionOccursAfterRetrievingRemoteValues random failures
by Will Burns (Jira)
[ https://issues.redhat.com/browse/ISPN-11050?page=com.atlassian.jira.plugi... ]
Will Burns updated ISPN-11050:
------------------------------
Git Pull Request: https://github.com/infinispan/infinispan/pull/8026, https://github.com/infinispan/infinispan/pull/8131 (was: https://github.com/infinispan/infinispan/pull/8026)
> StreamDistPartitionHandlingTest.testUsingIteratorButPartitionOccursAfterRetrievingRemoteValues random failures
> --------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-11050
> URL: https://issues.redhat.com/browse/ISPN-11050
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite
> Affects Versions: 10.1.0.Beta1
> Reporter: Dan Berindei
> Assignee: Will Burns
> Priority: Major
> Labels: testsuite_stability
> Fix For: 11.0.0.Dev03, 10.1.6.Final
>
> Attachments: ISPN-6494_bundler_20191207-1109_StreamDistPartitionHandlingTest-infinispan-core.log.gz
>
>
> The test requests one entry from a stream iterator, then takes the cache to degraded mode, and assumes that the requesting the 2nd entry won't throw an {{AvailabilityException}} because the iterator already fetched the value in the background.
> However, even though the iterator requested all the segments in parallel, there is no guarantee that the originator also got the responses. If the test installs the partition views too soon, some segments will have to be retried, and they will fail:
> {noformat}
> 11:20:27,056 TRACE (testng-Test:[]) [JGroupsTransport] Test-NodeA-4361 sending single request 15 to Test-NodeC-45859: InitialPublisherCommand{cacheName='Test'}
> 11:20:27,058 TRACE (testng-Test:[]) [BasePartitionHandlingTest] Partition forming
> 11:20:27,066 TRACE (testng-Test:[]) [BasePartitionHandlingTest] Partition forming
> 11:20:27,066 TRACE (testng-Test:[]) [BasePartitionHandlingTest] Before installing new view...
> 11:20:27,068 TRACE (transport-thread-Test-NodeA-p48709-t6:[Topology-Test]) [CheckPoint] Waiting for event pre_released * 1
> 11:20:27,068 TRACE (jgroups-9,Test-NodeC-45859:[]) [JGroupsTransport] Test-NodeC-45859 sending response for request 15 to Test-NodeA-4361: SuccessfulResponse(PublisherResponse{size=0, completedSegments={6 12 15 31-33 36 40-41 44 50 53 56-57 75 78 85 89-92 97 103 113 122-123 127 130 135-136 148 173 188-189 192-193 221 224-226 238 241-242 253}, lostSegments=null, complete=true, segmentOffset=0})
> 11:20:27,095 TRACE (testng-Test:[]) [BasePartitionHandlingTest] New views installed
> 11:20:27,096 TRACE (transport-thread-Test-NodeA-p48709-t6:[Topology-Test]) [CheckPoint] Received event post_released * 1 (available = 999999998, total = 999999999)
> 11:20:27,100 TRACE (transport-thread-Test-NodeA-p48709-t6:[Topology-Test]) [ClusterPublisherManagerImpl] Segments {3 6 11-12 15 22 26-29 31-37 39-44 47 50 52-57 62-63 75-78 81 85-86 89-92 97 103-105 113-115 120-125 127-128 130 135-136 141 148 155-156 172-174 188-194 197 200 205 209-210 221 224-226 229 234 237-238 241-243 245 248 253-254} not completed - retrying
> 11:20:27,102 TRACE (transport-thread-Test-NodeA-p48709-t6:[Topology-Test]) [JGroupsTransport] Test-NodeA-4361 sending single request 18 to Test-NodeC-45859: InitialPublisherCommand{cacheName='Test'}
> 11:20:27,100 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.partitionhandling.StreamDistPartitionHandlingTest.testUsingIteratorButPartitionOccursAfterRetrievingRemoteValues[DIST_SYNC, DENY_READ_WRITES]
> org.infinispan.partitionhandling.AvailabilityException: ISPN000305: Cluster is operating in degraded mode because of node failures.
> at org.infinispan.reactive.publisher.impl.PartitionAwareClusterPublisherManager$PartitionListener.onPartitionChange(PartitionAwareClusterPublisherManager.java:52) ~[classes/:?]
> at jdk.internal.reflect.GeneratedMethodAccessor336.invoke(Unknown Source) ~[?:?]
> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.lambda$invoke$1(AbstractListenerImpl.java:426) ~[classes/:?]
> at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:453) ~[classes/:?]
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1854) ~[classes/:?]
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1801) ~[classes/:?]
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1754) ~[classes/:?]
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl.doNotifyPartitionStatusChanged(CacheNotifierImpl.java:868) ~[classes/:?]
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyPartitionStatusChanged(CacheNotifierImpl.java:857) ~[classes/:?]
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at org.mockito.internal.stubbing.defaultanswers.ForwardsInvocations.answer(ForwardsInvocations.java:48) ~[mockito-core-2.27.0.jar:?]
> at org.infinispan.test.Mocks.lambda$blockingAnswer$0(Mocks.java:156) ~[test-classes/:?]
> at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:39) ~[mockito-core-2.27.0.jar:?]
> at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:96) ~[mockito-core-2.27.0.jar:?]
> at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29) ~[mockito-core-2.27.0.jar:?]
> at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:35) ~[mockito-core-2.27.0.jar:?]
> at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:61) ~[mockito-core-2.27.0.jar:?]
> at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:49) ~[mockito-core-2.27.0.jar:?]
> at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor$DispatcherDefaultingToRealMethod.interceptAbstract(MockMethodInterceptor.java:126) ~[mockito-core-2.27.0.jar:?]
> at org.infinispan.notifications.cachelistener.CacheNotifier$MockitoMock$819492735.notifyPartitionStatusChanged(Unknown Source) ~[classes/:?]
> at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.lambda$setAvailabilityMode$0(PartitionHandlingManagerImpl.java:101) ~[classes/:?]
> at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106) ~[?:?]
> at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235) ~[?:?]
> at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:143) ~[?:?]
> at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.setAvailabilityMode(PartitionHandlingManagerImpl.java:99) ~[classes/:?]
> at org.infinispan.topology.LocalTopologyManagerImpl.doHandleTopologyUpdate(LocalTopologyManagerImpl.java:354) ~[classes/:?]
> at org.infinispan.topology.LocalTopologyManagerImpl.lambda$handleTopologyUpdate$1(LocalTopologyManagerImpl.java:286) ~[classes/:?]
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175) ~[classes/:?]
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:37) ~[classes/:?]
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:227) ~[classes/:?]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
> at java.lang.Thread.run(Thread.java:834) [?:?]
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years
[JBoss JIRA] (ISPN-11050) StreamDistPartitionHandlingTest.testUsingIteratorButPartitionOccursAfterRetrievingRemoteValues random failures
by Will Burns (Jira)
[ https://issues.redhat.com/browse/ISPN-11050?page=com.atlassian.jira.plugi... ]
Will Burns updated ISPN-11050:
------------------------------
Fix Version/s: 10.1.6.Final
> StreamDistPartitionHandlingTest.testUsingIteratorButPartitionOccursAfterRetrievingRemoteValues random failures
> --------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-11050
> URL: https://issues.redhat.com/browse/ISPN-11050
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite
> Affects Versions: 10.1.0.Beta1
> Reporter: Dan Berindei
> Assignee: Will Burns
> Priority: Major
> Labels: testsuite_stability
> Fix For: 11.0.0.Dev03, 10.1.6.Final
>
> Attachments: ISPN-6494_bundler_20191207-1109_StreamDistPartitionHandlingTest-infinispan-core.log.gz
>
>
> The test requests one entry from a stream iterator, then takes the cache to degraded mode, and assumes that the requesting the 2nd entry won't throw an {{AvailabilityException}} because the iterator already fetched the value in the background.
> However, even though the iterator requested all the segments in parallel, there is no guarantee that the originator also got the responses. If the test installs the partition views too soon, some segments will have to be retried, and they will fail:
> {noformat}
> 11:20:27,056 TRACE (testng-Test:[]) [JGroupsTransport] Test-NodeA-4361 sending single request 15 to Test-NodeC-45859: InitialPublisherCommand{cacheName='Test'}
> 11:20:27,058 TRACE (testng-Test:[]) [BasePartitionHandlingTest] Partition forming
> 11:20:27,066 TRACE (testng-Test:[]) [BasePartitionHandlingTest] Partition forming
> 11:20:27,066 TRACE (testng-Test:[]) [BasePartitionHandlingTest] Before installing new view...
> 11:20:27,068 TRACE (transport-thread-Test-NodeA-p48709-t6:[Topology-Test]) [CheckPoint] Waiting for event pre_released * 1
> 11:20:27,068 TRACE (jgroups-9,Test-NodeC-45859:[]) [JGroupsTransport] Test-NodeC-45859 sending response for request 15 to Test-NodeA-4361: SuccessfulResponse(PublisherResponse{size=0, completedSegments={6 12 15 31-33 36 40-41 44 50 53 56-57 75 78 85 89-92 97 103 113 122-123 127 130 135-136 148 173 188-189 192-193 221 224-226 238 241-242 253}, lostSegments=null, complete=true, segmentOffset=0})
> 11:20:27,095 TRACE (testng-Test:[]) [BasePartitionHandlingTest] New views installed
> 11:20:27,096 TRACE (transport-thread-Test-NodeA-p48709-t6:[Topology-Test]) [CheckPoint] Received event post_released * 1 (available = 999999998, total = 999999999)
> 11:20:27,100 TRACE (transport-thread-Test-NodeA-p48709-t6:[Topology-Test]) [ClusterPublisherManagerImpl] Segments {3 6 11-12 15 22 26-29 31-37 39-44 47 50 52-57 62-63 75-78 81 85-86 89-92 97 103-105 113-115 120-125 127-128 130 135-136 141 148 155-156 172-174 188-194 197 200 205 209-210 221 224-226 229 234 237-238 241-243 245 248 253-254} not completed - retrying
> 11:20:27,102 TRACE (transport-thread-Test-NodeA-p48709-t6:[Topology-Test]) [JGroupsTransport] Test-NodeA-4361 sending single request 18 to Test-NodeC-45859: InitialPublisherCommand{cacheName='Test'}
> 11:20:27,100 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.partitionhandling.StreamDistPartitionHandlingTest.testUsingIteratorButPartitionOccursAfterRetrievingRemoteValues[DIST_SYNC, DENY_READ_WRITES]
> org.infinispan.partitionhandling.AvailabilityException: ISPN000305: Cluster is operating in degraded mode because of node failures.
> at org.infinispan.reactive.publisher.impl.PartitionAwareClusterPublisherManager$PartitionListener.onPartitionChange(PartitionAwareClusterPublisherManager.java:52) ~[classes/:?]
> at jdk.internal.reflect.GeneratedMethodAccessor336.invoke(Unknown Source) ~[?:?]
> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.lambda$invoke$1(AbstractListenerImpl.java:426) ~[classes/:?]
> at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:453) ~[classes/:?]
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1854) ~[classes/:?]
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1801) ~[classes/:?]
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1754) ~[classes/:?]
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl.doNotifyPartitionStatusChanged(CacheNotifierImpl.java:868) ~[classes/:?]
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyPartitionStatusChanged(CacheNotifierImpl.java:857) ~[classes/:?]
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at org.mockito.internal.stubbing.defaultanswers.ForwardsInvocations.answer(ForwardsInvocations.java:48) ~[mockito-core-2.27.0.jar:?]
> at org.infinispan.test.Mocks.lambda$blockingAnswer$0(Mocks.java:156) ~[test-classes/:?]
> at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:39) ~[mockito-core-2.27.0.jar:?]
> at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:96) ~[mockito-core-2.27.0.jar:?]
> at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29) ~[mockito-core-2.27.0.jar:?]
> at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:35) ~[mockito-core-2.27.0.jar:?]
> at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:61) ~[mockito-core-2.27.0.jar:?]
> at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:49) ~[mockito-core-2.27.0.jar:?]
> at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor$DispatcherDefaultingToRealMethod.interceptAbstract(MockMethodInterceptor.java:126) ~[mockito-core-2.27.0.jar:?]
> at org.infinispan.notifications.cachelistener.CacheNotifier$MockitoMock$819492735.notifyPartitionStatusChanged(Unknown Source) ~[classes/:?]
> at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.lambda$setAvailabilityMode$0(PartitionHandlingManagerImpl.java:101) ~[classes/:?]
> at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106) ~[?:?]
> at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235) ~[?:?]
> at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:143) ~[?:?]
> at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.setAvailabilityMode(PartitionHandlingManagerImpl.java:99) ~[classes/:?]
> at org.infinispan.topology.LocalTopologyManagerImpl.doHandleTopologyUpdate(LocalTopologyManagerImpl.java:354) ~[classes/:?]
> at org.infinispan.topology.LocalTopologyManagerImpl.lambda$handleTopologyUpdate$1(LocalTopologyManagerImpl.java:286) ~[classes/:?]
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175) ~[classes/:?]
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:37) ~[classes/:?]
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:227) ~[classes/:?]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
> at java.lang.Thread.run(Thread.java:834) [?:?]
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years
[JBoss JIRA] (ISPN-11567) Scattered caches with a single node expire entries immediately
by Dan Berindei (Jira)
Dan Berindei created ISPN-11567:
-----------------------------------
Summary: Scattered caches with a single node expire entries immediately
Key: ISPN-11567
URL: https://issues.redhat.com/browse/ISPN-11567
Project: Infinispan
Issue Type: Bug
Components: Core
Affects Versions: 11.0.0.Dev03, 10.1.5.Final
Reporter: Dan Berindei
Assignee: Dan Berindei
Fix For: 11.0.0.Dev04
{{ClusterExpirationManager.checkExpiredMaxIdle()}} sends a {{TouchCommand}} to the other owners and expires the entry locally if the touch was unsuccessful.
{{ScatteredTouchResponseCollector}} is stateless, and reports that the entry has not been touched if it doesn't receive any {{true}} response. However, this is only correct if at least one backup entry existing on another node, and that is not the always the case: e.g. between the backup node leaving the cluster and another node becoming a backup, or when the cluster has a single node.
Since {{ClusterExpirationManager.checkExpiredMaxIdle()}} is called on every read, before the entry being expired on the local node, it means a transient entry in a scattered cache with a single node will expire on the first read, immediately after being inserted.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years