[infinispan-dev] Kubernetes/OpenShift Rolling updates and configuration changes

[Sebastian Laskawiec]

Hey!

I've been thinking about potential use of Kubernetes/OpenShift (OpenShift =
Kubernetes + additional features) Rolling Update mechanism for updating
configuration of Hot Rod servers. You might find some more information
about the rolling updates here [1][2] but putting it simply, Kubernetes
replaces nodes in the cluster one at a time. What's worth mentioning,
Kubernetes ensures that the newly created replica is fully operational
before taking down another one.

There are two things that make me scratching my head...

#1 - What type of configuration changes can we introduce using rolling
updates?

I'm pretty sure introducing a new cache definition won't do any harm. But
what if we change a cache type from Distributed to Replicated? Do you have
any idea which configuration changes are safe and which are not? Could come
up with such list?

#2 - How to prevent loosing data during the rolling update process?

In Kubernetes we have a mechanism called lifecycle hooks [3] (we can invoke
a script during container startup/shutdown). The problem with shutdown
script is that it's time constrained (if it won't end up within certain
amount of time, Kubernetes will simply kill the container). Fortunately
this time is configurable.

The idea to prevent from loosing data would be to invoke (enquque and wait
for finish) state transfer process triggered by the shutdown hook (with
timeout set to maximum value). If for some reason this won't work (e.g. a
user has so much data that migrating it this way would take ages), there is
a backup plan - Infinispan Rolling Upgrades [4].

What do you think about this?

Thanks
Sebastian

[1] https://www.youtube.com/watch?v=9C6YeyyUUmI
[2] http://kubernetes.io/docs/user-guide/rolling-updates/
[3]
http://kubernetes.io/docs/user-guide/container-environment/#container-hooks
[4]
http://infinispan.org/docs/stable/user_guide/user_guide.html#_Rolling_chapter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160714/482a578f/attachment-0001.html