[jboss-jira] [JBoss JIRA] (WFLY-9913) Balancer fails to balance requests according to load provided by custom load metric

[Jan Kašík (JIRA)]

     [ https://issues.jboss.org/browse/WFLY-9913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jan Kašík updated WFLY-9913:
----------------------------
    Description: 
When balancer uses load provided by custom load metric, it intermittently fails to redirect requests to expected workers. We use following scenario in our test with EAP 7.1.1 nodes:

# Prepare one balancer and three workers with custom load metric which has these attribute values:
{{history: 0, decay: 0, capacity: 1000}} and uses implementation from \[1\].
# The implementation takes dummy load values from files in local filesystem. So the values in files are set then.
# The verification, that expected load factor is loaded is made by running {{"/subsystem=undertow/configuration=filter/mod-cluster=modcluster:read-resource(include-runtime=true, recursive=true, recursive-depth=100)}}.
# 1000+ requests is made on balancer and for each request, the JVM route of handling node is noted.
# It is verified, that expected amount of request went to each node according to load factor.
# Load values in files are changes and verification, that they were loaded is made again.
# 1000+ requests is made on balancer and for each request, the JVM route of handling node is noted.
# After the requests were made, the verification, that all went according to expectations is made. In this step, the verification sometimes fails, because there were unexpected count of requests on each node.

We suspect, that this is an error in mod_cluster implementation. The problem is, that sometimes more requests than expected ends up on the worker with lowest load:
{{java.lang.AssertionError: Assert #5, Request distribution according to the load was supposed to be [jboss-eap-7.2-1:106, jboss-eap-7.2-2:106, jboss-eap-7.2-3:789] with tolerance +/-10, but was: [jboss-eap-7.2-1:36, jboss-eap-7.2-2:35, jboss-eap-7.2-3:930]}}

[1]: https://github.com/Karm/mod_cluster-custom-load-metric


  was:
When balancer uses load provided by custom load metric, it intermittently fails to redirect requests to expected workers. We use following scenario in our test with EAP 7.1.1 nodes:

# Prepare one balancer and three workers with custom load metric which has these attribute values:
{{history: 0, decay: 0, capacity: 1000}} and uses implementation from \[1\].
# The implementation takes dummy load values from files in local filesystem. So the values in files are set then.
# The verification, that expected load factor is loaded is made by running {{"/subsystem=undertow/configuration=filter/mod-cluster=modcluster:read-resource(include-runtime=true, recursive=true, recursive-depth=100)}}.
# 1000+ requests is made on balancer and for each request, the JVM route of handling node is noted.
# It is verified, that expected amount of request went to each node according to load factor.
# Load values in files are changes and verification, that they were loaded is made again.
# 1000+ requests is made on balancer and for each request, the JVM route of handling node is noted.
# After the requests were made, the verification, that all went according to expectations is made. In this step, the verification sometimes fails, because there were unexpected count of requests on each node.

This occurs only on EAP/Wildlfly balancer and we suspect, that this is an error in Undertow mod_cluster implementation. The problem is, that sometimes more requests than expected ends up on the worker with lowest load:
{{java.lang.AssertionError: Assert #5, Request distribution according to the load was supposed to be [jboss-eap-7.2-1:106, jboss-eap-7.2-2:106, jboss-eap-7.2-3:789] with tolerance +/-10, but was: [jboss-eap-7.2-1:36, jboss-eap-7.2-2:35, jboss-eap-7.2-3:930]}}

[1]: https://github.com/Karm/mod_cluster-custom-load-metric




> Balancer fails to balance requests according to load provided by custom load metric
> -----------------------------------------------------------------------------------
>
>                 Key: WFLY-9913
>                 URL: https://issues.jboss.org/browse/WFLY-9913
>             Project: WildFly
>          Issue Type: Bug
>          Components: mod_cluster, Web (Undertow)
>    Affects Versions: 12.0.0.Beta1
>            Reporter: Jan Kašík
>            Assignee: Radoslav Husar
>
> When balancer uses load provided by custom load metric, it intermittently fails to redirect requests to expected workers. We use following scenario in our test with EAP 7.1.1 nodes:
> # Prepare one balancer and three workers with custom load metric which has these attribute values:
> {{history: 0, decay: 0, capacity: 1000}} and uses implementation from \[1\].
> # The implementation takes dummy load values from files in local filesystem. So the values in files are set then.
> # The verification, that expected load factor is loaded is made by running {{"/subsystem=undertow/configuration=filter/mod-cluster=modcluster:read-resource(include-runtime=true, recursive=true, recursive-depth=100)}}.
> # 1000+ requests is made on balancer and for each request, the JVM route of handling node is noted.
> # It is verified, that expected amount of request went to each node according to load factor.
> # Load values in files are changes and verification, that they were loaded is made again.
> # 1000+ requests is made on balancer and for each request, the JVM route of handling node is noted.
> # After the requests were made, the verification, that all went according to expectations is made. In this step, the verification sometimes fails, because there were unexpected count of requests on each node.
> We suspect, that this is an error in mod_cluster implementation. The problem is, that sometimes more requests than expected ends up on the worker with lowest load:
> {{java.lang.AssertionError: Assert #5, Request distribution according to the load was supposed to be [jboss-eap-7.2-1:106, jboss-eap-7.2-2:106, jboss-eap-7.2-3:789] with tolerance +/-10, but was: [jboss-eap-7.2-1:36, jboss-eap-7.2-2:35, jboss-eap-7.2-3:930]}}
> [1]: https://github.com/Karm/mod_cluster-custom-load-metric



--
This message was sent by Atlassian JIRA
(v7.5.0#75005)