<div dir="ltr">Hey Sanne!<div><br></div><div>I would treat both those things (upgrading Infinispan from version X to Y and changing configuration) separate. </div><div><br></div><div>Thanks</div><div>Sebastian</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 19, 2016 at 1:06 PM, Sanne Grinovero <span dir="ltr"><<a href="mailto:sanne@infinispan.org" target="_blank">sanne@infinispan.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Just wondering, we're looking into how to compare configuration<br>
settings just to try validate the user isn't attempting an insane<br>
upgrade right?<br>
<br>
Or is there an actual need to want to change Infinispan version AND<br>
switch the essential grid configuration settings at the same time?<br>
(that seems quite insane to me, and unnecessary as one could do it in<br>
two steps..)<br>
<br>
Thanks<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
On 19 July 2016 at 10:06, Sebastian Laskawiec <<a href="mailto:slaskawi@redhat.com">slaskawi@redhat.com</a>> wrote:<br>
> Hey Tristan, Emmanuel!<br>
><br>
> Comments inlined.<br>
><br>
> Thanks<br>
> Sebastian<br>
><br>
> On Tue, Jul 19, 2016 at 10:08 AM, Emmanuel Bernard <<a href="mailto:emmanuel@hibernate.org">emmanuel@hibernate.org</a>><br>
> wrote:<br>
>><br>
>> Considering very few options can be changed at runtime safely, should we<br>
>> rather focus of a strategy where we start a new grid and populate it<br>
>> with the old grid before flipping the proxy to the new one?<br>
><br>
><br>
> +1, that's exactly what the Infinispan Rolling Upgrade does.<br>
><br>
>><br>
>><br>
>> On Mon 2016-07-18 17:12, Tristan Tarrant wrote:<br>
>> > On 14/07/16 12:17, Sebastian Laskawiec wrote:<br>
>> > > Hey!<br>
>> > ><br>
>> > > I've been thinking about potential use of Kubernetes/OpenShift<br>
>> > > (OpenShift = Kubernetes + additional features) Rolling Update<br>
>> > > mechanism for updating configuration of Hot Rod servers. You might<br>
>> > > find some more information about the rolling updates here [1][2] but<br>
>> > > putting it simply, Kubernetes replaces nodes in the cluster one at a<br>
>> > > time. What's worth mentioning, Kubernetes ensures that the newly<br>
>> > > created replica is fully operational before taking down another one.<br>
>> > ><br>
>> > > There are two things that make me scratching my head...<br>
>> > ><br>
>> > > #1 - What type of configuration changes can we introduce using rolling<br>
>> > > updates?<br>
>> > ><br>
>> > > I'm pretty sure introducing a new cache definition won't do any harm.<br>
>> > > But what if we change a cache type from Distributed to Replicated? Do<br>
>> > > you have any idea which configuration changes are safe and which are<br>
>> > > not? Could come up with such list?<br>
>> > Very few changes are safe, but obviously this would need to be verified<br>
>> > on a per-attribute basis. All of the attributes which can be changed at<br>
>> > runtime (timeouts, eviction size) are safe.<br>
>> ><br>
>> > ><br>
>> > > #2 - How to prevent loosing data during the rolling update process?<br>
>> > I believe you want to write losing :)<br>
><br>
><br>
> Good one :)<br>
><br>
>><br>
>> > > In Kubernetes we have a mechanism called lifecycle hooks [3] (we can<br>
>> > > invoke a script during container startup/shutdown). The problem with<br>
>> > > shutdown script is that it's time constrained (if it won't end up<br>
>> > > within certain amount of time, Kubernetes will simply kill the<br>
>> > > container). Fortunately this time is configurable.<br>
>> > ><br>
>> > > The idea to prevent from loosing data would be to invoke (enquque and<br>
>> > > wait for finish) state transfer process triggered by the shutdown hook<br>
>> > > (with timeout set to maximum value). If for some reason this won't<br>
>> > > work (e.g. a user has so much data that migrating it this way would<br>
>> > > take ages), there is a backup plan - Infinispan Rolling Upgrades [4].<br>
>> > The thing that concerns me here is the amount of churn involved: the<br>
>> > safest bet for us is that the net topology doesn't change, i.e. you end<br>
>> > up with the exact number of nodes you started with<br>
><br>
><br>
> Yes, Kubernetes Rolling Update works this way. The number of nodes at the<br>
> end of the process is equal to the number you started with.<br>
><br>
>><br>
>> and they are replaced<br>
>> > one by one in a way that the replacement assumes the identity of the<br>
>> > replaced (both as persistent uuid, owned segments and data in a<br>
>> > persistent store).<br>
>><br>
>> Other types could be supported but they will<br>
>> > definitely have a level of risk.<br>
>> > Also we don't have any guarantees that a newer version will be able to<br>
>> > cluster with an older one...<br>
><br>
><br>
> I'm not sure we can ensure the same identity of the replaced node. If we<br>
> consider configuration changes, a user can change anything...<br>
><br>
> I think I'm convinced that the Infinispan Rolling Upgrade procedure is the<br>
> only proper solution at this stage. Other ways (although much simpler) must<br>
> be treated as - 'do it at your own risk'.<br>
><br>
>><br>
>> ><br>
>> > Tristan<br>
>> > _______________________________________________<br>
>> > infinispan-dev mailing list<br>
>> > <a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
>> > <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
>> _______________________________________________<br>
>> infinispan-dev mailing list<br>
>> <a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
>> <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> infinispan-dev mailing list<br>
> <a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
</div></div></blockquote></div><br></div>