[JBoss JIRA] (ISPN-10073) Nested async operations with HotRod client hang
by Dejan Bosanac (Jira)
[ https://issues.jboss.org/browse/ISPN-10073?page=com.atlassian.jira.plugin... ]
Dejan Bosanac commented on ISPN-10073:
--------------------------------------
Thanks [~NadirX]! This might be a bit harder to do, as tests that are affected are part of the upstream project that work already with multiple implementations. We'll keep working around it.
> Nested async operations with HotRod client hang
> -----------------------------------------------
>
> Key: ISPN-10073
> URL: https://issues.jboss.org/browse/ISPN-10073
> Project: Infinispan
> Issue Type: Bug
> Components: Hot Rod
> Affects Versions: 9.4.10.Final
> Reporter: Dejan Bosanac
> Assignee: Will Burns
> Priority: Major
>
> We noticed some issued with the HotRod client, when trying to use nested async calls. Originally, this was discovered in the context of Vert.x tests, but below you can find the smallest reproducer code that I could create (without any Vert.x dependencies).
> {code:java}
> //DefaultCacheManager cm = new DefaultCacheManager();
> //Cache<String, String> cache = cm.createCache("default", new ConfigurationBuilder().build());
> RemoteCacheManager cm = new RemoteCacheManager();
> RemoteCache<String, String> cache = cm.getCache("default");
> final CompletableFuture put = new CompletableFuture();
> cache.putIfAbsentAsync("A", "A").thenAccept(resultA -> {
> final CompletableFuture nested = new CompletableFuture();
> cache.putIfAbsentAsync("B", "B").thenAccept(resultB -> {
> nested.complete("B");
> });
> try {
> nested.get();
> } catch (Exception e) {
> e.printStackTrace();
> }
> put.complete("B");
> });
> put.get();
> {code}
> This would basically hang and the nested operation would never receive response. What I noticed in the original test is that response would become available after the framework timeout the test.
> Everything works as expected with embedded cache.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years
[JBoss JIRA] (ISPN-9940) CoordinatorStopTest.testCoordinatorLeaving random failures
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-9940?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-9940:
----------------------------------
Status: Open (was: New)
> CoordinatorStopTest.testCoordinatorLeaving random failures
> ----------------------------------------------------------
>
> Key: ISPN-9940
> URL: https://issues.jboss.org/browse/ISPN-9940
> Project: Infinispan
> Issue Type: Bug
> Reporter: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Attachments: short_ISPN-9588_reproducer_20190206-1100_CoordinatorStopTest-infinispan-core.log
>
>
> There appear to be multiple failure modes:
> {noformat}
> 11:32:17,920 ERROR (jgroups-4,NodeB-49595:[]) [DelayedViewJGroupsTransport] Timed out waiting for view to be unblocked: [NodeB-49595|3] (2) [NodeB-49595, NodeC-2725]
> 11:32:17,925 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.scattered.statetransfer.CoordinatorStopTest.testCoordinatorLeaves[bias=NEVER]
> org.infinispan.test.TestException: java.util.concurrent.TimeoutException
> at org.infinispan.util.ControlledRpcManager.uncheckedGet(ControlledRpcManager.java:257) ~[test-classes/:?]
> at org.infinispan.util.ControlledRpcManager$SentRequest.expectAllResponses(ControlledRpcManager.java:580) ~[test-classes/:?]
> at org.infinispan.util.ControlledRpcManager$SentRequest.receiveAll(ControlledRpcManager.java:596) ~[test-classes/:?]
> at org.infinispan.scattered.statetransfer.CoordinatorStopTest.testCoordinatorLeaves(CoordinatorStopTest.java:162) ~[test-classes/:?]
> {noformat}
> {noformat}
> org.infinispan.util.concurrent.TimeoutException: Timed out waiting for a response for StateRequestCommand{cache=___defaultcache, origin=CoordinatorStopTest-NodeC-51924, type=CONFIRM_REVOKED_SEGMENTS, topologyId=9, segments={0}}
> at org.infinispan.util.ControlledRpcManager$InternalRequest.peekResponse(ControlledRpcManager.java:334)
> at org.infinispan.util.ControlledRpcManager$SentRequest.receiveAll(ControlledRpcManager.java:610)
> at org.infinispan.scattered.statetransfer.CoordinatorStopTest.testCoordinatorLeaves(CoordinatorStopTest.java:161)
> at org.infinispan.commons.test.TestNGLongTestsHook.run(TestNGLongTestsHook.java:24)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Suppressed: org.infinispan.util.logging.TraceException
> at org.infinispan.remoting.rpc.RpcManagerImpl.blocking(RpcManagerImpl.java:277)
> at org.infinispan.util.AbstractDelegatingRpcManager.blocking(AbstractDelegatingRpcManager.java:112)
> at org.infinispan.scattered.impl.ScatteredStateConsumerImpl.handleSegments(ScatteredStateConsumerImpl.java:159)
> at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:364)
> at org.infinispan.scattered.impl.ScatteredStateConsumerImpl.onTopologyUpdate(ScatteredStateConsumerImpl.java:101)
> at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:198)
> at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:57)
> at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:116)
> at org.infinispan.topology.LocalTopologyManagerImpl.doHandleRebalance(LocalTopologyManagerImpl.java:519)
> at org.infinispan.topology.LocalTopologyManagerImpl.lambda$handleRebalance$3(LocalTopologyManagerImpl.java:477)
> {noformat}
> Initially I thought it was related to ISPN-9588 and JGRP-2293, but the failure seems to be related to the way the test blocks the view installation and sometimes does not unblock it.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years
[JBoss JIRA] (ISPN-9940) CoordinatorStopTest.testCoordinatorLeaving random failures
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-9940?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-9940:
----------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/6810, https://github.com/infinispan/infinispan/pull/6811
> CoordinatorStopTest.testCoordinatorLeaving random failures
> ----------------------------------------------------------
>
> Key: ISPN-9940
> URL: https://issues.jboss.org/browse/ISPN-9940
> Project: Infinispan
> Issue Type: Bug
> Reporter: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Attachments: short_ISPN-9588_reproducer_20190206-1100_CoordinatorStopTest-infinispan-core.log
>
>
> There appear to be multiple failure modes:
> {noformat}
> 11:32:17,920 ERROR (jgroups-4,NodeB-49595:[]) [DelayedViewJGroupsTransport] Timed out waiting for view to be unblocked: [NodeB-49595|3] (2) [NodeB-49595, NodeC-2725]
> 11:32:17,925 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.scattered.statetransfer.CoordinatorStopTest.testCoordinatorLeaves[bias=NEVER]
> org.infinispan.test.TestException: java.util.concurrent.TimeoutException
> at org.infinispan.util.ControlledRpcManager.uncheckedGet(ControlledRpcManager.java:257) ~[test-classes/:?]
> at org.infinispan.util.ControlledRpcManager$SentRequest.expectAllResponses(ControlledRpcManager.java:580) ~[test-classes/:?]
> at org.infinispan.util.ControlledRpcManager$SentRequest.receiveAll(ControlledRpcManager.java:596) ~[test-classes/:?]
> at org.infinispan.scattered.statetransfer.CoordinatorStopTest.testCoordinatorLeaves(CoordinatorStopTest.java:162) ~[test-classes/:?]
> {noformat}
> {noformat}
> org.infinispan.util.concurrent.TimeoutException: Timed out waiting for a response for StateRequestCommand{cache=___defaultcache, origin=CoordinatorStopTest-NodeC-51924, type=CONFIRM_REVOKED_SEGMENTS, topologyId=9, segments={0}}
> at org.infinispan.util.ControlledRpcManager$InternalRequest.peekResponse(ControlledRpcManager.java:334)
> at org.infinispan.util.ControlledRpcManager$SentRequest.receiveAll(ControlledRpcManager.java:610)
> at org.infinispan.scattered.statetransfer.CoordinatorStopTest.testCoordinatorLeaves(CoordinatorStopTest.java:161)
> at org.infinispan.commons.test.TestNGLongTestsHook.run(TestNGLongTestsHook.java:24)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Suppressed: org.infinispan.util.logging.TraceException
> at org.infinispan.remoting.rpc.RpcManagerImpl.blocking(RpcManagerImpl.java:277)
> at org.infinispan.util.AbstractDelegatingRpcManager.blocking(AbstractDelegatingRpcManager.java:112)
> at org.infinispan.scattered.impl.ScatteredStateConsumerImpl.handleSegments(ScatteredStateConsumerImpl.java:159)
> at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:364)
> at org.infinispan.scattered.impl.ScatteredStateConsumerImpl.onTopologyUpdate(ScatteredStateConsumerImpl.java:101)
> at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:198)
> at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:57)
> at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:116)
> at org.infinispan.topology.LocalTopologyManagerImpl.doHandleRebalance(LocalTopologyManagerImpl.java:519)
> at org.infinispan.topology.LocalTopologyManagerImpl.lambda$handleRebalance$3(LocalTopologyManagerImpl.java:477)
> {noformat}
> Initially I thought it was related to ISPN-9588 and JGRP-2293, but the failure seems to be related to the way the test blocks the view installation and sometimes does not unblock it.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years
[JBoss JIRA] (ISPN-10086) AllClusterExecutorTest.testExecutorTriConsumerTimeoutException random failures
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-10086?page=com.atlassian.jira.plugin... ]
Dan Berindei updated ISPN-10086:
--------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/6808, https://github.com/infinispan/infinispan/pull/6809
> AllClusterExecutorTest.testExecutorTriConsumerTimeoutException random failures
> ------------------------------------------------------------------------------
>
> Key: ISPN-10086
> URL: https://issues.jboss.org/browse/ISPN-10086
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 10.0.0.Beta3, 9.4.11.Final
> Reporter: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Fix For: 10.0.0.Beta4, 9.4.12.Final
>
>
> When running tests in parallel, the remote request may complete before the thread that sent the request resumes. Because the scheduled executor is mocked to return {{null}}, this leads to a {{NullPointerException}}:
> {noformat}
> 09:43:41,995 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.manager.AllClusterExecutorTest.testExecutorTriConsumerTimeoutException
> java.lang.NullPointerException: null
> at org.infinispan.remoting.transport.AbstractRequest.setTimeoutFuture(AbstractRequest.java:94) ~[classes/:?]
> at org.infinispan.remoting.transport.AbstractRequest.setTimeout(AbstractRequest.java:55) ~[classes/:?]
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:831) ~[classes/:?]
> at org.infinispan.manager.impl.AllClusterExecutor.submitConsumer(AllClusterExecutor.java:182) ~[classes/:?]
> at org.infinispan.manager.ClusterExecutor.submitConsumer(ClusterExecutor.java:116) ~[classes/:?]
> at org.infinispan.manager.Test$17.call(AllClusterExecutorTest.java:402) ~[test-classes/:?]
> at org.infinispan.test.TestingUtil.withCacheManagers(TestingUtil.java:1601) ~[test-classes/:?]
> at org.infinispan.manager.AllClusterExecutorTest.testExecutorTriConsumerTimeoutException(AllClusterExecutorTest.java:382) ~[test-classes/:?]
> {noformat}
> The problem may be that the test calls {{Mockito.verify()}} from the consumer twice, once for each node, instead of doing it only on the remote node.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 1 month
[JBoss JIRA] (ISPN-10086) AllClusterExecutorTest.testExecutorTriConsumerTimeoutException random failures
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-10086?page=com.atlassian.jira.plugin... ]
Dan Berindei updated ISPN-10086:
--------------------------------
Status: Open (was: New)
> AllClusterExecutorTest.testExecutorTriConsumerTimeoutException random failures
> ------------------------------------------------------------------------------
>
> Key: ISPN-10086
> URL: https://issues.jboss.org/browse/ISPN-10086
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 10.0.0.Beta3, 9.4.11.Final
> Reporter: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Fix For: 10.0.0.Beta4, 9.4.12.Final
>
>
> When running tests in parallel, the remote request may complete before the thread that sent the request resumes. Because the scheduled executor is mocked to return {{null}}, this leads to a {{NullPointerException}}:
> {noformat}
> 09:43:41,995 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.manager.AllClusterExecutorTest.testExecutorTriConsumerTimeoutException
> java.lang.NullPointerException: null
> at org.infinispan.remoting.transport.AbstractRequest.setTimeoutFuture(AbstractRequest.java:94) ~[classes/:?]
> at org.infinispan.remoting.transport.AbstractRequest.setTimeout(AbstractRequest.java:55) ~[classes/:?]
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:831) ~[classes/:?]
> at org.infinispan.manager.impl.AllClusterExecutor.submitConsumer(AllClusterExecutor.java:182) ~[classes/:?]
> at org.infinispan.manager.ClusterExecutor.submitConsumer(ClusterExecutor.java:116) ~[classes/:?]
> at org.infinispan.manager.Test$17.call(AllClusterExecutorTest.java:402) ~[test-classes/:?]
> at org.infinispan.test.TestingUtil.withCacheManagers(TestingUtil.java:1601) ~[test-classes/:?]
> at org.infinispan.manager.AllClusterExecutorTest.testExecutorTriConsumerTimeoutException(AllClusterExecutorTest.java:382) ~[test-classes/:?]
> {noformat}
> The problem may be that the test calls {{Mockito.verify()}} from the consumer twice, once for each node, instead of doing it only on the remote node.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 1 month