<div dir="ltr">Hey Emmanuel,<div><br></div><div>Comments inlined.</div><div><br></div><div>Thanks,</div><div>Sebastian<br><br><div class="gmail_quote"><div dir="ltr">On Wed, May 31, 2017 at 2:56 PM Emmanuel Bernard <<a href="mailto:emmanuel@hibernate.org">emmanuel@hibernate.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">To Sanne’s point, I think HTTP(/2) would be a better longer term path if we think we can make it as efficient as current HR. But let’s evaluate the numbers of cycles to reach that point. Doing Seb’s approach might be a good first step.</div></blockquote><div><br></div><div>I will be looking into HTTP/2 implementation starting from today/tomorrow. So it should be there soon. And of course I will do some benchmarks (or even help Jiri to upgrade perfCheck to do benchmarks using HTTP/1.1 and HTTP/2).</div><div><br></div><div>Also please bear in mind that there will probably be two ways of switching protocols - using HTTP/1.1 Upgrade header and TLS/ALPN negotiation. As you might expect, the latter enforces TLS (and therefore the throughput will be lower). </div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>Speaking of Sebastian, I have been discussing with Burr, Edson on the idea of a *node* sidecar (as opposed to a *pod* sidecar). To your problem, could you use Daemonset to enforce one Load Balancer per node or at least per project instead of one per pod deployed with Infinispan in it?</div></div></blockquote><div><br></div><div>Unless I missed anything, it won't buy us anything. The idea behind the POC is to make all Infinispan nodes directly accessible from the outside world. The client must be able to access whichever node it wishes. This is achieved by creating a load balancer per Infinispan pod. So the load balancer works more like an external IP rather than a "real" load balancer. </div><div><br></div><div>Just FYI, another round of comments on L3/L4 TCP Ingress has just started: <a href="https://github.com/kubernetes/kubernetes/issues/23291">https://github.com/kubernetes/kubernetes/issues/23291</a></div><div>The rough estimate is to get it in Kube 1.8. Once this is implemented, we could use a TCP Ingress per pod (instead of load balancer per pod). The main difference will probably be in $$$. Load balancers are pretty expensive.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><br></div><div>WDYT, is it possible?</div><div><br><div><blockquote type="cite"></blockquote></div></div></div><div style="word-wrap:break-word"><div><div><blockquote type="cite"><div>On 30 May 2017, at 20:43, Sebastian Laskawiec <<a href="mailto:slaskawi@redhat.com" target="_blank">slaskawi@redhat.com</a>> wrote:</div><br class="m_-5137476386842397598Apple-interchange-newline"></blockquote></div></div></div><div style="word-wrap:break-word"><div><div><blockquote type="cite"><div><div dir="ltr">Hey guys!<div><br></div><div>Over past few weeks I've been working on accessing Infinispan cluster deployed inside Kubernetes from the outside world. The POC diagram looks like the following:</div><div><br></div></div></div></blockquote></div></div></div><div style="word-wrap:break-word"><div><div><blockquote type="cite"><div><div dir="ltr"><div><span id="m_-5137476386842397598cid:15c58f5a1c0d0cad2201"><pasted1.png></span><br></div></div></div></blockquote></div></div></div><div style="word-wrap:break-word"><div><div><blockquote type="cite"><div><div dir="ltr"><div><br></div><div>As a reminder, the easiest (though not the most effective) way to do it is to expose a load balancer Service (or a Node Port Service) and access it using a client with basic intelligence (so that it doesn't try to update server list based on topology information). As you might expect, this won't give you much performance but at least you could access the cluster. Another approach is to use TLS/SNI but again, the performance would be even worse.</div><div><br></div><div>During the research I tried to answer this problem and created "External IP Controller" [1] (and corresponding Pull Request for mapping internal/external addresses [2]). The main idea is to create a controller deployed inside Kubernetes which will create (and destroy if not needed) a load balancer per Infinispan Pod. Additionally the controller exposes mapping between internal and external addresses which allows the client to properly update server list as well as consistent hash information. A full working example is located here [3].</div><div><br></div><div>The biggest question is whether it's worth it? The short answer is yes. Here are some benchmark results of performing 10k puts and 10k puts&gets (please take them with a big grain of salt, I didn't optimize any server settings):</div><div><ul><li>Benchmark app deployed inside Kuberenetes and using internal addresses (baseline):</li><ul><li>10k puts: 674.244 ± 16.654</li><li>10k puts&gets: 1288.437 ± 136.207</li></ul><li>Benchamrking app deployed in a VM outside of Kubernetes with basic intelligence:</li><ul><li><b><font color="#990000">10k puts: 1465.567 ± 176.349</font></b></li><li><b><font color="#990000">10k puts&gets: 2684.984 ± 114.993</font></b></li></ul><li>Benchamrking app deployed in a VM outside of Kubernetes with address mapping and topology aware hashing:</li><ul><li><b><font color="#38761d">10k puts: 1052.891 ± 31.218</font></b></li><li><b><font color="#38761d">10k puts&gets: 2465.586 ± 85.034</font></b></li></ul></ul></div><div>Note that benchmarking Infinispan from a VM might be very misleading since it depends on data center configuration. Benchmarks above definitely contain some delay between Google Compute Engine VM and a Kubernetes cluster deployed in Google Container Engine. How big is the delay? Hard to tell. What counts is the difference between client using basic intelligence and topology aware intelligence. And as you can see it's not that small.</div><div><br></div><div>So the bottom line - if you can, deploy your application along with Infinispan cluster inside Kubernetes. That's the fastest configuration since only iptables are involved. Otherwise use a load balancer per pod with External IP Controller. If you don't care about performance, just use basic client intelligence and expose everything using single load balancer.</div><div><br></div><div>Thanks,</div><div>Sebastian </div><div><br></div><div>[1] <a href="https://github.com/slaskawi/external-ip-proxy" target="_blank">https://github.com/slaskawi/external-ip-proxy</a></div><div>[2] <a href="https://github.com/infinispan/infinispan/pull/5164" target="_blank">https://github.com/infinispan/infinispan/pull/5164</a></div><div>[3] <a href="https://github.com/slaskawi/external-ip-proxy/tree/master/benchmark" target="_blank">https://github.com/slaskawi/external-ip-proxy/tree/master/benchmark</a></div></div></div></blockquote></div></div></div><div style="word-wrap:break-word"><div><div><blockquote type="cite"><div>
_______________________________________________<br>infinispan-dev mailing list<br><a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br><a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></div></blockquote></div></div></div>_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></blockquote></div></div></div><div dir="ltr">-- <br></div><div data-smartmail="gmail_signature"><div dir="ltr"><p class="inbox-inbox-fullname-container" style="box-sizing:border-box;color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><span class="inbox-inbox-firstname-container" style="box-sizing:border-box">SEBASTIAN</span><span class="inbox-inbox-Apple-converted-space"> </span><span class="inbox-inbox-lastname-container" style="box-sizing:border-box">ŁASKAWIEC</span></p><p class="inbox-inbox-position-container" style="box-sizing:border-box;color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><span class="inbox-inbox-position" style="box-sizing:border-box">INFINISPAN DEVELOPER</span></p><p class="inbox-inbox-legal-container" style="box-sizing:border-box;font-family:overpass,sans-serif;margin:0px;font-size:10px;color:rgb(153,153,153)"><a class="inbox-inbox-redhat-anchor" href="https://www.redhat.com/" target="_blank" style="box-sizing:border-box;color:rgb(0,136,206);margin:0px;text-decoration:none">Red Hat<span class="inbox-inbox-Apple-converted-space"> </span><span style="box-sizing:border-box">EMEA</span></a></p><table border="0" style="box-sizing:border-box;color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium"><tbody style="box-sizing:border-box"><tr style="box-sizing:border-box"><td width="100px" style="box-sizing:border-box"><a href="https://red.ht/sig" style="box-sizing:border-box"><img width="90" height="auto" style="box-sizing: border-box;" src="https://www.redhat.com/files/brand/email/sig-redhat.png"></a></td></tr></tbody></table></div></div>