Radoslav Husar updated MODCLUSTER-527:
Tester: Michal Karm Babacek
Load Balancing logic can fail production environments
Issue Type: Bug
Components: Native (httpd modules)
Reporter: Sean Cavanagh
Assignee: Jean-Frederic Clere
We suffered the same problem as MODCLUSTER-100 in our production environment.
If any balance pool sees 0 requests during 2*LBstatusRecalTime the next request will go
to the first worker in the pool. If an application consistently sends requests at a rate
less than 2*LBstatusRecalTime, then all requests will be sent to the same node, and
_balancing will be completely broken_
Less worrisome, but still not ideal, if one worker is more heavily balanced than the
others (e.g. it's load factor has been manually set to '1' in order to offload
it; a somewhat common practice), that worker will still get a session once every
I know it seems odd to hear about an application which can load a server by sending fewer
than 1 request every 10 seconds, however, if one is using sticky sessions, one is not
balancing _requests_, one is balancing _logins_. The usage pattern for our enterprise
application sees our users login relatively slowly, and once a user has their JSESSIONID,
those http requests never see the mod_proxy_cluster balancing algorithm again.
It's nice that there is a config parameter now, which will allow us to balance
properly again, but I would offer that since the behaviour of external users can
completely break the algorithm this bug requires a more sophisticated fix.
If authoritative parties agree it's a good idea, I'm happy to write a patch that
does the following:
1) improves the documentation, more clearly explaining how the algorithm functions and
alerting users to this parameter
2) clean up a couple of straight-up documentation errors
3) Modifies the internal_find_best_byrequest formula from:
status = lbstatus + (elected - oldelected) * 1000)/lbfactor;
status = lbstatus + (elected - oldelected + 1) * 1000)/lbfactor
It seems to me that the current formula assumes that each node would be winning at least
some elections during the LBstatusRecalTime window, so by adding 1 we can ensure that
assumption is always true, and no worker ever gets a score of 0.
I did a bit of algebra and I'm pretty confident that this change won't influence
the outcome of any elections. That could only happen if:
lfc - lfw > lfc*lsw - lfw*lsc
where lfc is the client load factor, lsc is the client's lbstatus, and simlar for lfw
Since the election numbers are multiplied by 1000, and the max difference between two
load factors is 99, I conclude it is impossible for the addition of a single election to
both workers to change the outcome of the election.
So yeah, let me know.