2:15 a.m.
To Sanne’s point, I think HTTP(/2) would be a better longer term path if we think we can
make it as efficient as current HR. But let’s evaluate the numbers of cycles to reach that
point. Doing Seb’s approach might be a good first step.
Speaking of Sebastian, I have been discussing with Burr, Edson on the idea of a *node*
sidecar (as opposed to a *pod* sidecar). To your problem, could you use Daemonset to
enforce one Load Balancer per node or at least per project instead of one per pod deployed
with Infinispan in it?
WDYT, is it possible?
On 30 May 2017, at 20:43, Sebastian Laskawiec
<slaskawi(a)redhat.com> wrote:
Hey guys!
Over past few weeks I've been working on accessing Infinispan cluster deployed inside
Kubernetes from the outside world. The POC diagram looks like the following:
<pasted1.png>
As a reminder, the easiest (though not the most effective) way to do it is to expose a
load balancer Service (or a Node Port Service) and access it using a client with basic
intelligence (so that it doesn't try to update server list based on topology
information). As you might expect, this won't give you much performance but at least
you could access the cluster. Another approach is to use TLS/SNI but again, the
performance would be even worse.
During the research I tried to answer this problem and created "External IP
Controller" [1] (and corresponding Pull Request for mapping internal/external
addresses [2]). The main idea is to create a controller deployed inside Kubernetes which
will create (and destroy if not needed) a load balancer per Infinispan Pod. Additionally
the controller exposes mapping between internal and external addresses which allows the
client to properly update server list as well as consistent hash information. A full
working example is located here [3].
The biggest question is whether it's worth it? The short answer is yes. Here are some
benchmark results of performing 10k puts and 10k puts&gets (please take them with a
big grain of salt, I didn't optimize any server settings):
Benchmark app deployed inside Kuberenetes and using internal addresses (baseline):
10k puts: 674.244 ± 16.654
10k puts&gets: 1288.437 ± 136.207
Benchamrking app deployed in a VM outside of Kubernetes with basic intelligence:
10k puts: 1465.567 ± 176.349
10k puts&gets: 2684.984 ± 114.993
Benchamrking app deployed in a VM outside of Kubernetes with address mapping and topology
aware hashing:
10k puts: 1052.891 ± 31.218
10k puts&gets: 2465.586 ± 85.034
Note that benchmarking Infinispan from a VM might be very misleading since it depends on
data center configuration. Benchmarks above definitely contain some delay between Google
Compute Engine VM and a Kubernetes cluster deployed in Google Container Engine. How big is
the delay? Hard to tell. What counts is the difference between client using basic
intelligence and topology aware intelligence. And as you can see it's not that small.
So the bottom line - if you can, deploy your application along with Infinispan cluster
inside Kubernetes. That's the fastest configuration since only iptables are involved.
Otherwise use a load balancer per pod with External IP Controller. If you don't care
about performance, just use basic client intelligence and expose everything using single
load balancer.
Thanks,
Sebastian
[1] https://github.com/slaskawi/external-ip-proxy
<https://github.com/slaskawi/external-ip-proxy>
[2] https://github.com/infinispan/infinispan/pull/5164
<https://github.com/infinispan/infinispan/pull/5164>
[3] https://github.com/slaskawi/external-ip-proxy/tree/master/benchmark
<https://github.com/slaskawi/external-ip-proxy/tree/master/benchmark>__...
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev