[JBoss JIRA] (ISPN-8706) HotRodMergeTest Timed out waiting for rebalancing to complete
by Ryan Emerson (JIRA)
[ https://issues.jboss.org/browse/ISPN-8706?page=com.atlassian.jira.plugin.... ]
Ryan Emerson updated ISPN-8706:
-------------------------------
Status: Pull Request Sent (was: Coding In Progress)
Git Pull Request: https://github.com/infinispan/infinispan/pull/5806 (was: https://github.com/infinispan/infinispan/pull/5715)
> HotRodMergeTest Timed out waiting for rebalancing to complete
> -------------------------------------------------------------
>
> Key: ISPN-8706
> URL: https://issues.jboss.org/browse/ISPN-8706
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Server
> Affects Versions: 9.2.0.Alpha2
> Reporter: Ryan Emerson
> Assignee: Ryan Emerson
> Attachments: HotRodMergeTest_20180129.log.gz
>
>
> The HotRodMergeTest is now consistently failing with the following error:
> {code}
> [OK: 321, KO: 1, SKIP: 0] Test failed: org.infinispan.server.hotrod.HotRodMergeTest.testNewTopologySentAfterCleanMerge[DIST_SYNC]
> java.lang.RuntimeException: Timed out waiting for rebalancing to complete on node HotRodMergeTest[DIST_SYNC]-NodeA-31095, expected member list is [HotRodMergeTest[DIST_SYNC]-NodeA-31095, HotRodMergeTest[DIST_SYNC]-NodeB-33866], current member list is [HotRodMergeTest[DIST_SYNC]-NodeA-31095]!
> at org.infinispan.test.TestingUtil.waitForNoRebalance(TestingUtil.java:388)
> at org.infinispan.test.TestingUtil.waitForNoRebalance(TestingUtil.java:424)
> at org.infinispan.partitionhandling.BasePartitionHandlingTest$Partition.waitForPartitionToForm(BasePartitionHandlingTest.java:282)
> at org.infinispan.partitionhandling.BasePartitionHandlingTest$Partition.merge(BasePartitionHandlingTest.java:258)
> at org.infinispan.partitionhandling.BasePartitionHandlingTest$Partition.merge(BasePartitionHandlingTest.java:246)
> at org.infinispan.server.hotrod.HotRodMergeTest.testNewTopologySentAfterCleanMerge(HotRodMergeTest.java:87)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:38)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:382)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (ISPN-8925) Conflict Resolution can exhaust the ASYNC_TRANSPORT_EXECUTOR
by Ryan Emerson (JIRA)
[ https://issues.jboss.org/browse/ISPN-8925?page=com.atlassian.jira.plugin.... ]
Ryan Emerson commented on ISPN-8925:
------------------------------------
ISPN-8706 was caused by this issue as it was loading in many caches, such as counters, counter-config etc
> Conflict Resolution can exhaust the ASYNC_TRANSPORT_EXECUTOR
> -------------------------------------------------------------
>
> Key: ISPN-8925
> URL: https://issues.jboss.org/browse/ISPN-8925
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.2.0.Final
> Reporter: Ryan Emerson
> Assignee: Ryan Emerson
> Fix For: 9.2.1.Final
>
>
> During the execution of AvailabilityStrategyContext#updateTopologiesAfterMerge, it's necessary for a topology update to be sent to the cluster via ClusterTopologyManagerImpl#executeOnClusterAsync, which utilises the ASYNC_TRANSPORT_EXECUTOR, before a call is made to ConflictManager#resolveConflicts. This topology update is vital as it contains the topologyId which all of the conflict resolution RPCs depend on. If this topology update is not sent, then ConflictManager#resolveConflicts will eventually timeout as no progress can be made.
> The problem is that during the entire execution of AvailabilityStrategyContext#doMergePartitions, an ASYNC_TRANSPORT_EXECUTOR thread is occupied. Therefore, when AvailabilityStrategyContext#updateTopologiesAfterMerge is called prior to conflict resolution it's possible that ALL threads are executing runnables that are waiting indefinitely on ConflictManager#resolveConflicts and therefore it's not possible to send the topology update.
> As the number of caches increase the number of doMergePartition runnables on the ASYNC_TRANSPORT_EXECUTOR increases, consequently so does the likelihood of the executor's resources becoming exhausted.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (ISPN-8925) Conflict Resolution can exhaust the ASYNC_TRANSPORT_EXECUTOR
by Ryan Emerson (JIRA)
[ https://issues.jboss.org/browse/ISPN-8925?page=com.atlassian.jira.plugin.... ]
Ryan Emerson updated ISPN-8925:
-------------------------------
Status: Open (was: New)
> Conflict Resolution can exhaust the ASYNC_TRANSPORT_EXECUTOR
> -------------------------------------------------------------
>
> Key: ISPN-8925
> URL: https://issues.jboss.org/browse/ISPN-8925
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.2.0.Final
> Reporter: Ryan Emerson
> Assignee: Ryan Emerson
> Fix For: 9.2.1.Final
>
>
> During the execution of AvailabilityStrategyContext#updateTopologiesAfterMerge, it's necessary for a topology update to be sent to the cluster via ClusterTopologyManagerImpl#executeOnClusterAsync, which utilises the ASYNC_TRANSPORT_EXECUTOR, before a call is made to ConflictManager#resolveConflicts. This topology update is vital as it contains the topologyId which all of the conflict resolution RPCs depend on. If this topology update is not sent, then ConflictManager#resolveConflicts will eventually timeout as no progress can be made.
> The problem is that during the entire execution of AvailabilityStrategyContext#doMergePartitions, an ASYNC_TRANSPORT_EXECUTOR thread is occupied. Therefore, when AvailabilityStrategyContext#updateTopologiesAfterMerge is called prior to conflict resolution it's possible that ALL threads are executing runnables that are waiting indefinitely on ConflictManager#resolveConflicts and therefore it's not possible to send the topology update.
> As the number of caches increase the number of doMergePartition runnables on the ASYNC_TRANSPORT_EXECUTOR increases, consequently so does the likelihood of the executor's resources becoming exhausted.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (ISPN-8925) Conflict Resolution can exhaust the ASYNC_TRANSPORT_EXECUTOR
by Ryan Emerson (JIRA)
[ https://issues.jboss.org/browse/ISPN-8925?page=com.atlassian.jira.plugin.... ]
Ryan Emerson updated ISPN-8925:
-------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/5806
> Conflict Resolution can exhaust the ASYNC_TRANSPORT_EXECUTOR
> -------------------------------------------------------------
>
> Key: ISPN-8925
> URL: https://issues.jboss.org/browse/ISPN-8925
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.2.0.Final
> Reporter: Ryan Emerson
> Assignee: Ryan Emerson
> Fix For: 9.2.1.Final
>
>
> During the execution of AvailabilityStrategyContext#updateTopologiesAfterMerge, it's necessary for a topology update to be sent to the cluster via ClusterTopologyManagerImpl#executeOnClusterAsync, which utilises the ASYNC_TRANSPORT_EXECUTOR, before a call is made to ConflictManager#resolveConflicts. This topology update is vital as it contains the topologyId which all of the conflict resolution RPCs depend on. If this topology update is not sent, then ConflictManager#resolveConflicts will eventually timeout as no progress can be made.
> The problem is that during the entire execution of AvailabilityStrategyContext#doMergePartitions, an ASYNC_TRANSPORT_EXECUTOR thread is occupied. Therefore, when AvailabilityStrategyContext#updateTopologiesAfterMerge is called prior to conflict resolution it's possible that ALL threads are executing runnables that are waiting indefinitely on ConflictManager#resolveConflicts and therefore it's not possible to send the topology update.
> As the number of caches increase the number of doMergePartition runnables on the ASYNC_TRANSPORT_EXECUTOR increases, consequently so does the likelihood of the executor's resources becoming exhausted.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (ISPN-8925) Conflict Resolution can exhaust the ASYNC_TRANSPORT_EXECUTOR
by Ryan Emerson (JIRA)
Ryan Emerson created ISPN-8925:
----------------------------------
Summary: Conflict Resolution can exhaust the ASYNC_TRANSPORT_EXECUTOR
Key: ISPN-8925
URL: https://issues.jboss.org/browse/ISPN-8925
Project: Infinispan
Issue Type: Bug
Components: Core
Affects Versions: 9.2.0.Final
Reporter: Ryan Emerson
Assignee: Ryan Emerson
Fix For: 9.2.1.Final
During the execution of AvailabilityStrategyContext#updateTopologiesAfterMerge, it's necessary for a topology update to be sent to the cluster via ClusterTopologyManagerImpl#executeOnClusterAsync, which utilises the ASYNC_TRANSPORT_EXECUTOR, before a call is made to ConflictManager#resolveConflicts. This topology update is vital as it contains the topologyId which all of the conflict resolution RPCs depend on. If this topology update is not sent, then ConflictManager#resolveConflicts will eventually timeout as no progress can be made.
The problem is that during the entire execution of AvailabilityStrategyContext#doMergePartitions, an ASYNC_TRANSPORT_EXECUTOR thread is occupied. Therefore, when AvailabilityStrategyContext#updateTopologiesAfterMerge is called prior to conflict resolution it's possible that ALL threads are executing runnables that are waiting indefinitely on ConflictManager#resolveConflicts and therefore it's not possible to send the topology update.
As the number of caches increase the number of doMergePartition runnables on the ASYNC_TRANSPORT_EXECUTOR increases, consequently so does the likelihood of the executor's resources becoming exhausted.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (ISPN-8858) BulkOperationsTest parameter values change in every run
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-8858?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo updated ISPN-8858:
------------------------------
Fix Version/s: 9.3.0.Alpha1
> BulkOperationsTest parameter values change in every run
> -------------------------------------------------------
>
> Key: ISPN-8858
> URL: https://issues.jboss.org/browse/ISPN-8858
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Server
> Affects Versions: 9.2.0.CR2
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Minor
> Fix For: 9.3.0.Final, 9.3.0.Alpha1
>
>
> The {{BulkOperationsTest}} methods use a data provider, but the parameter values are lambdas, so they are logged with {{Object.toString()}}:
> {noformat}
> [OK: 3222, KO: 2, SKIP: 0] Test starting: org.infinispan.client.hotrod.BulkOperationsTest.testClear[LOCAL]([org.infinispan.client.hotrod.BulkOperationsTest$$Lambda$1387/864766760@5e4517de])
> [OK: 3223, KO: 2, SKIP: 0] Test succeeded: org.infinispan.client.hotrod.BulkOperationsTest.testClear[LOCAL]([org.infinispan.client.hotrod.BulkOperationsTest$$Lambda$1387/864766760@5e4517de])
> [OK: 3223, KO: 2, SKIP: 0] Test starting: org.infinispan.client.hotrod.BulkOperationsTest.testClear[LOCAL]([org.infinispan.client.hotrod.BulkOperationsTest$$Lambda$1388/1656036295@52bab332])
> [OK: 3224, KO: 2, SKIP: 0] Test succeeded: org.infinispan.client.hotrod.BulkOperationsTest.testClear[LOCAL]([org.infinispan.client.hotrod.BulkOperationsTest$$Lambda$1388/1656036295@52bab332])
> [OK: 3224, KO: 2, SKIP: 0] Test starting: org.infinispan.client.hotrod.BulkOperationsTest.testClear[LOCAL]([org.infinispan.client.hotrod.BulkOperationsTest$$Lambda$1389/1747688893@311c80b2])
> [OK: 3225, KO: 2, SKIP: 0] Test succeeded: org.infinispan.client.hotrod.BulkOperationsTest.testClear[LOCAL]([org.infinispan.client.hotrod.BulkOperationsTest$$Lambda$1389/1747688893@311c80b2])
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (ISPN-8858) BulkOperationsTest parameter values change in every run
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-8858?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo updated ISPN-8858:
------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
> BulkOperationsTest parameter values change in every run
> -------------------------------------------------------
>
> Key: ISPN-8858
> URL: https://issues.jboss.org/browse/ISPN-8858
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Server
> Affects Versions: 9.2.0.CR2
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Minor
> Fix For: 9.3.0.Final
>
>
> The {{BulkOperationsTest}} methods use a data provider, but the parameter values are lambdas, so they are logged with {{Object.toString()}}:
> {noformat}
> [OK: 3222, KO: 2, SKIP: 0] Test starting: org.infinispan.client.hotrod.BulkOperationsTest.testClear[LOCAL]([org.infinispan.client.hotrod.BulkOperationsTest$$Lambda$1387/864766760@5e4517de])
> [OK: 3223, KO: 2, SKIP: 0] Test succeeded: org.infinispan.client.hotrod.BulkOperationsTest.testClear[LOCAL]([org.infinispan.client.hotrod.BulkOperationsTest$$Lambda$1387/864766760@5e4517de])
> [OK: 3223, KO: 2, SKIP: 0] Test starting: org.infinispan.client.hotrod.BulkOperationsTest.testClear[LOCAL]([org.infinispan.client.hotrod.BulkOperationsTest$$Lambda$1388/1656036295@52bab332])
> [OK: 3224, KO: 2, SKIP: 0] Test succeeded: org.infinispan.client.hotrod.BulkOperationsTest.testClear[LOCAL]([org.infinispan.client.hotrod.BulkOperationsTest$$Lambda$1388/1656036295@52bab332])
> [OK: 3224, KO: 2, SKIP: 0] Test starting: org.infinispan.client.hotrod.BulkOperationsTest.testClear[LOCAL]([org.infinispan.client.hotrod.BulkOperationsTest$$Lambda$1389/1747688893@311c80b2])
> [OK: 3225, KO: 2, SKIP: 0] Test succeeded: org.infinispan.client.hotrod.BulkOperationsTest.testClear[LOCAL]([org.infinispan.client.hotrod.BulkOperationsTest$$Lambda$1389/1747688893@311c80b2])
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years