[infinispan-dev] Infinispan and OpenShift/Kubernetes PetSets

[Sebastian Laskawiec]

Hey!

I've been playing with Kubernetes PetSets [1] for a while and I'd like to
share some thoughts. Before I dig in, let me give you some PetSets
highlights:

   - PetSets are alpha resources for managing stateful apps in Kubernetes
   1.3 (and OpenShift Origin 1.3).
   - Since this is an alpha resource, there are no guarantees about
   backwards compatibility. Alpha resources can also be disabled in some
   public cloud providers (you can control which API versions are accessible
   [2]).
   - PetSets allows starting pods in sequence (not relevant for us, but
   this is a killer feature for master-slave systems).
   - Each Pod has it's own unique entry in DNS, which makes discovery very
   simple (I'll dig into that a bit later)
   - Volumes are always mounted to the same Pods, which is very important
   in Cache Store scenarios when we restart pods (e.g. Rolling Upgrades [3]).

Thoughts and ideas after spending some time playing with this feature:

   - PetSets make discovery a lot easier. It's a combination of two things
   - Headless Services [4] which create multiple A records in DNS and
   predictable host names. Each Pod has it's own unique DNS entry following
   pattern: {PetSetName}-{PodIndex}.{ServiceName} [5]. Here's an example of an
   Infinispan PetSet deployed on my local cluster [6]. As you can see we have
   all domain names and IPs from a single DNS query.
   - Maybe we could perform discovery using this mechanism? I'm aware of
   DNS discovery implemented in KUBE_PING [7][8] but the code looks trivial
   [9] so maybe it should be implement inside JGroups? @Bela - WDYT?
   - PetSets do not integrate well with OpenShift 'new-app' command. In
   other words, our users will need to use provided yaml (or json) files to
   create Infinispan cluster. It's not a show-stopper but it's a bit less
   convenient than 'oc new-app'.
   - Since PetSets are alpha resources they need to be considered as
   secondary way to deploy Infinispan on Kubernetes and OpenShift.
   - Finally, the persistent volumes - since a Pod always gets the same
   volume, it would be safe to use any file-based cache store.

If you'd like to play with PetSets on your local environment, here are
necessary yaml files [10].

Thanks
Sebastian


[1] http://kubernetes.io/docs/user-guide/petset/
[2] For checking which APIs are accessible, use 'kubectl api-versions'
[3]
http://infinispan.org/docs/stable/user_guide/user_guide.html#_Rolling_chapter
[4] http://kubernetes.io/docs/user-guide/services/#headless-services
[5] http://kubernetes.io/docs/user-guide/petset/#peer-discovery
[6] https://gist.github.com/slaskawi/0866e63a39276f8ab66376229716a676
[7] https://github.com/jboss-openshift/openshift-ping/tree/master/dns
[8] https://github.com/jgroups-extras/jgroups-kubernetes/tree/master/dns
[9] http://stackoverflow.com/a/12405896/562699
[10] You might need to adjust ImageStream.
https://gist.github.com/slaskawi/7cffb5588dabb770f654557579c5f2d0
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160819/4b0cfef0/attachment.html