<div dir="ltr">Hey Radim,<div><br></div><div>Moving to dev mailing list.</div><div><br></div><div>Comments inlined.</div><div><br></div><div>Thanks,</div><div>Sebastian</div><br><div class="gmail_quote"><div dir="ltr">On Tue, May 2, 2017 at 5:28 PM Radim Vansa <<a href="mailto:rvansa@redhat.com">rvansa@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Sebastian,<br>
<br>
I am currently getting acquainted with OpenShift so I have been reading<br>
your blogposts about that. Couple of questions:<br>
<br>
<a href="http://blog.infinispan.org/2016/10/openshift-and-node-affinity.html" rel="noreferrer" target="_blank">http://blog.infinispan.org/2016/10/openshift-and-node-affinity.html</a><br>
<br>
- so you need to have different deployment config for each rack/site?<br></blockquote><div><br></div><div>Yes. A while ago I read an article about managing scheduler using labels:</div><div><a href="https://blog.openshift.com/deploying-applications-to-specific-nodes/">https://blog.openshift.com/deploying-applications-to-specific-nodes/</a><br></div><div><br></div><div>So I think it can be optimized to 1 DeploymentConfig + some magic in spec.template. But that's only my intuition. I haven't played with this yet.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><a href="http://blog.infinispan.org/2017/03/checking-infinispan-cluster-health-and.html" rel="noreferrer" target="_blank">http://blog.infinispan.org/2017/03/checking-infinispan-cluster-health-and.html</a><br>
<br>
maxUnavailable: 1 and maxSurge: 1 don't sound too good to me - if you<br>
can't fit all the data into single pod, you need to set maxUnavailable:<br>
0 (to not bring any nodes down before the rolling upgrade completes) and<br>
maxSurge: 100% to have enough nodes started. + Some post-hook to make<br>
sure all data are in new cluster before you bring down the old one. Am I<br>
missing something?<br></blockquote><div><br></div><div>Before answering those questions, let me show you two examples:</div><div><ul><li>maxUnavailable: 1, maxSurge 1</li><ul><li></li><li>oc logs transactions-repository-2-deploy -f</li><ol><li>--> Scaling up transactions-repository-2 from 0 to 3, scaling down transactions-repository-1 from 3 to 0 (keep 2 pods available, don't exceed 4 pods)</li><li><b> Scaling transactions-repository-2 up to 1</b></li><li><b> Scaling transactions-repository-1 down to 2</b></li><li> Scaling transactions-repository-2 up to 2</li><li> Scaling transactions-repository-1 down to 1</li><li> Scaling transactions-repository-2 up to 3</li><li> Scaling transactions-repository-1 down to 0</li><li>--> Success</li></ol></ul><li>maxUnavailable: 0, maxSurge 100%</li><ul><li>oc logs transactions-repository-3-deploy -f</li><ul></ul><ol><li>--> Scaling up transactions-repository-3 from 0 to 3, scaling down transactions-repository-2 from 3 to 0 (keep 3 pods available, don't exceed 6 pods)<br></li><li> Scaling transactions-repository-3 up to 3</li><li><b> Scaling transactions-repository-2 down to 1<br></b></li><li><b> Scaling transactions-repository-2 down to 0</b><br></li><li>--> Success</li></ol></ul></ul><div>So we are talking about Kubernetes Rolling Update here. You have a new version of your deployment (e.g. with updated parameters, labels etc) and you want update your deployment in Kubernetes (do not mess it up with Infinispan Rolling Upgrade where the intention is to roll out a new Infinispan cluster).</div></div><div><br></div><div>The former approach (maxUnavailable: 1, maxSurge 1) allocates additional Infinispan node for greater cluster capacity. Then it scales the old cluster down. This results in sending KILL [1] signal to the Pod so it gets a chance to shut down gracefully. As a side effect, this also triggers cluster rebalance (since 1 node leaves the cluster). And we go like this on and on until we replace old cluster with new one.</div><div><br></div><div>The latter approach spins a new cluster up. Then Kubernetes sends KILL signal too <b><u><i>all</i></u></b> old cluster members.</div><div><br></div><div>Both approaches should work if configured correctly (the former relies heavily on readiness probes and the latter on moving data off the node after receiving KILL signal). However I would assume the latter generates much more network traffic in a short period of time which I consider a bit more risky. </div><div><br></div><div>Regarding to to a hook which ensures all data has been migrated - I'm not sure how to build such a hook. The main idea is to keep cluster in operational state so that none of the client would notice the rollout. It works like a charm with the former approach. </div><div><br></div><div>[1] <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods">https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods</a></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Radim<br>
<br>
--<br>
Radim Vansa <<a href="mailto:rvansa@redhat.com" target="_blank">rvansa@redhat.com</a>><br>
JBoss Performance Team<br>
<br>
</blockquote></div></div><div dir="ltr">-- <br></div><div data-smartmail="gmail_signature"><div dir="ltr"><p class="inbox-inbox-fullname-container" style="box-sizing:border-box;color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><span class="inbox-inbox-firstname-container" style="box-sizing:border-box">SEBASTIAN</span><span class="inbox-inbox-Apple-converted-space"> </span><span class="inbox-inbox-lastname-container" style="box-sizing:border-box">ŁASKAWIEC</span></p><p class="inbox-inbox-position-container" style="box-sizing:border-box;color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><span class="inbox-inbox-position" style="box-sizing:border-box">INFINISPAN DEVELOPER</span></p><p class="inbox-inbox-legal-container" style="box-sizing:border-box;font-family:overpass,sans-serif;margin:0px;font-size:10px;color:rgb(153,153,153)"><a class="inbox-inbox-redhat-anchor" href="https://www.redhat.com/" target="_blank" style="box-sizing:border-box;color:rgb(0,136,206);margin:0px;text-decoration:none">Red Hat<span class="inbox-inbox-Apple-converted-space"> </span><span style="box-sizing:border-box">EMEA</span></a></p><table border="0" style="box-sizing:border-box;color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium"><tbody style="box-sizing:border-box"><tr style="box-sizing:border-box"><td width="100px" style="box-sizing:border-box"><a href="https://red.ht/sig" style="box-sizing:border-box"><img width="90" height="auto" style="box-sizing: border-box;" src="https://www.redhat.com/files/brand/email/sig-redhat.png"></a></td></tr></tbody></table></div></div>