[infinispan-issues] [JBoss JIRA] (ISPN-11679) Bad uneven segment distributions can happen after the rebalancing with SyncConsistentHashFactory
Masafumi Miura (Jira)
issues at jboss.org
Mon Apr 20 16:14:24 EDT 2020
[ https://issues.redhat.com/browse/ISPN-11679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14047285#comment-14047285 ]
Masafumi Miura commented on ISPN-11679:
---------------------------------------
Here is the example result of the unit test with 22 nodes:
{code}
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.infinispan.distribution.ch.SyncConsistentHashFactoryKeyDistributionTest
[OK: 0, KO: 0, SKIP: 0] Test starting: org.infinispan.distribution.ch.SyncConsistentHashFactoryKeyDistributionTest.testRebalanceDistribution
Distribution for 22 nodes (relative to the average)
===
Segments = 200 220 400 440 600 660 720 800 880 1000 1100
Any owner: % < 0.60 = 1.46% 0.00% 0.12% 0.00% 0.03% 0.02% 0.01% 0.00% 0.00% 0.00% 0.00%
Any owner: % < 0.70 = 4.54% 0.00% 0.91% 0.00% 0.33% 0.13% 0.12% 0.07% 0.07% 0.03% 0.01%
Any owner: % < 0.80 = 11.69% 0.02% 3.12% 0.00% 2.85% 1.73% 1.66% 1.19% 0.81% 1.03% 0.70%
Any owner: % < 0.90 = 23.57% 3.80% 11.91% 0.45% 12.28% 10.06% 8.65% 8.26% 6.84% 10.01% 8.64%
Any owner: % > 1.10 = 42.02% 0.72% 7.25% 0.15% 7.82% 0.43% 1.80% 1.55% 0.06% 2.04% 0.17%
Any owner: % > 1.15 = 14.77% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
Any owner: % > 1.30 = 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
Any owner: % > 1.50 = 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
Any owner: 99.90% percentile = 1.167 1.150 1.139 1.125 1.130 1.117 1.123 1.125 1.100 1.122 1.110
Any owner: max = 1.167 1.150 1.139 1.125 1.130 1.117 1.123 1.125 1.113 1.122 1.110
Any owner: min = 0.222 0.800 0.444 0.850 0.500 0.550 0.523 0.639 0.650 0.622 0.700
Primary: % < 0.60 = 1.68% 0.00% 0.10% 0.00% 0.03% 0.02% 0.01% 0.00% 0.00% 0.00% 0.00%
Primary: % < 0.70 = 5.32% 0.00% 0.70% 0.00% 0.23% 0.17% 0.11% 0.10% 0.05% 0.04% 0.01%
Primary: % < 0.80 = 13.24% 0.00% 3.56% 0.00% 2.42% 1.96% 1.02% 1.01% 0.89% 1.20% 0.75%
Primary: % < 0.90 = 26.48% 0.01% 13.25% 0.01% 13.74% 11.12% 5.67% 8.96% 7.39% 9.02% 9.20%
Primary: % > 1.10 = 56.32% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
Primary: % > 1.15 = 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
Primary: % > 1.30 = 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
Primary: % > 1.50 = 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
Primary: 99.90% percentile = 1.111 1.000 1.056 1.000 1.074 1.033 1.063 1.056 1.025 1.067 1.040
Primary: max = 1.111 1.000 1.056 1.000 1.074 1.033 1.063 1.056 1.025 1.067 1.040
Primary: min = 0.222 1.000 0.444 1.000 0.519 0.567 0.531 0.611 0.675 0.622 0.700
Segments per node - max/min ratio = 5.250 1.438 2.500 1.294 2.222 1.970 2.088 1.717 1.673 1.804 1.549
[OK: 1, KO: 0, SKIP: 0] Test succeeded: org.infinispan.distribution.ch.SyncConsistentHashFactoryKeyDistributionTest.testRebalanceDistribution
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 71.962 s - in org.infinispan.distribution.ch.SyncConsistentHashFactoryKeyDistributionTest
{code}
If the number of the segment is configured as the exact same to "20 * number of nodes" (= 440 in the above example), the test result looks slightly better than other cases. Is this an expected result?
> Bad uneven segment distributions can happen after the rebalancing with SyncConsistentHashFactory
> ------------------------------------------------------------------------------------------------
>
> Key: ISPN-11679
> URL: https://issues.redhat.com/browse/ISPN-11679
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.4.19.Final, 10.1.6.Final
> Reporter: Masafumi Miura
> Priority: Major
>
> Uneven segment distribution can happen after rebalancing with SyncConsistentHashFactory.
> Even if a larger number of segments than "20 * number of nodes" is configured, some nodes have only around 50-60% number of segments than other nodes after the rebalance in the worst case.
> I understand the documentation (like the following) states about the possibility of slight uneven distribution as a potential drawback of SyncConsistentHashFactory.
> [Infinispan 10 configuration guide|https://infinispan.org/docs/stable/titles/configuring/configuring.html#key_ownership] says:
> {quote}
> This implementation does have some negative points in that the load distribution is slightly uneven. It also moves more segments than strictly necessary on a join or leave.
> {quote}
> [JDG 7 Developer Guide|https://access.redhat.com/documentation/en-us/red_hat_data_grid/7.2/html/developer_guide/high_availability_using_server_hinting#consistenthashfactories-1] says:
> {quote}
> It has a potential drawback in that it can move a greater number of segments than necessary during re-balancing. This can be mitigated by using a larger number of segments.
> Another potential drawback is that the segments are not distributed as evenly as possible, and actually using a very large number of segments can make the distribution of segments worse.
> {quote}
> However, only half segments than others is not a slight uneven.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
More information about the infinispan-issues
mailing list