[infinispan-dev] Kubernetes/OpenShift Rolling updates and configuration changes

[Sebastian Laskawiec]

Hey Tristan, Emmanuel!

Comments inlined.

Thanks
Sebastian

On Tue, Jul 19, 2016 at 10:08 AM, Emmanuel Bernard <emmanuel at hibernate.org>
wrote:

> Considering very few options can be changed at runtime safely, should we
> rather focus of a strategy where we start a new grid and populate it
> with the old grid before flipping the proxy to the new one?
>

+1, that's exactly what the Infinispan Rolling Upgrade does.


>
> On Mon 2016-07-18 17:12, Tristan Tarrant wrote:
> > On 14/07/16 12:17, Sebastian Laskawiec wrote:
> > > Hey!
> > >
> > > I've been thinking about potential use of Kubernetes/OpenShift
> > > (OpenShift = Kubernetes + additional features) Rolling Update
> > > mechanism for updating configuration of Hot Rod servers. You might
> > > find some more information about the rolling updates here [1][2] but
> > > putting it simply, Kubernetes replaces nodes in the cluster one at a
> > > time. What's worth mentioning, Kubernetes ensures that the newly
> > > created replica is fully operational before taking down another one.
> > >
> > > There are two things that make me scratching my head...
> > >
> > > #1 - What type of configuration changes can we introduce using rolling
> > > updates?
> > >
> > > I'm pretty sure introducing a new cache definition won't do any harm.
> > > But what if we change a cache type from Distributed to Replicated? Do
> > > you have any idea which configuration changes are safe and which are
> > > not? Could come up with such list?
> > Very few changes are safe, but obviously this would need to be verified
> > on a per-attribute basis. All of the attributes which can be changed at
> > runtime (timeouts, eviction size) are safe.
> >
> > >
> > > #2 - How to prevent loosing data during the rolling update process?
> > I believe you want to write losing :)
>

Good one :)


> > > In Kubernetes we have a mechanism called lifecycle hooks [3] (we can
> > > invoke a script during container startup/shutdown). The problem with
> > > shutdown script is that it's time constrained (if it won't end up
> > > within certain amount of time, Kubernetes will simply kill the
> > > container). Fortunately this time is configurable.
> > >
> > > The idea to prevent from loosing data would be to invoke (enquque and
> > > wait for finish) state transfer process triggered by the shutdown hook
> > > (with timeout set to maximum value). If for some reason this won't
> > > work (e.g. a user has so much data that migrating it this way would
> > > take ages), there is a backup plan - Infinispan Rolling Upgrades [4].
> > The thing that concerns me here is the amount of churn involved: the
> > safest bet for us is that the net topology doesn't change, i.e. you end
> > up with the exact number of nodes you started with
>

Yes, Kubernetes Rolling Update works this way. The number of nodes at the
end of the process is equal to the number you started with.


> and they are replaced
> > one by one in a way that the replacement assumes the identity of the
> > replaced (both as persistent uuid, owned segments and data in a
> > persistent store).
>
Other types could be supported but they will
> > definitely have a level of risk.
> > Also we don't have any guarantees that a newer version will be able to
> > cluster with an older one...
>

I'm not sure we can ensure the same identity of the replaced node. If we
consider configuration changes, a user can change anything...

I think I'm convinced that the Infinispan Rolling Upgrade procedure is the
only proper solution at this stage. Other ways (although much simpler) must
be treated as - 'do it at your own risk'.


> >
> > Tristan
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160719/28a97dc9/attachment.html