[infinispan-dev] Infinispan and OpenShift/Kubernetes PetSets

[Bela Ban]

On 19/08/16 11:27, Gustavo Fernandes wrote:
> On Fri, Aug 19, 2016 at 10:18 AM, Bela Ban <bban at redhat.com
> <mailto:bban at redhat.com>> wrote:
>
>     Hi Sebastian
>
>     the usual restrictions apply: if DNS discovery depends on external libs,
>     then it should be hosted in jgroups-extras, otherwise we can add it to
>     JGroups itself.
>
>
>
> Bela, you forgot to mention an extra restriction: even if it has no
> external deps, it should
> not be too big in terms of LOC (what is the threshold?)  ;)


You mean because people copy&paste the deps into one class? Yes, that 
happened before but since the resulting class was 17K, I rejected it. 
The largest class in JGroups (Util) has 4K, so that's a ballpark figure.

However, with jgroups-extras, we now have the possibility of adding deps.


> Gustavo
>
>
>     On 19/08/16 11:00, Sebastian Laskawiec wrote:
>     > Hey!
>     >
>     > I've been playing with Kubernetes PetSets [1] for a while and I'd like
>     > to share some thoughts. Before I dig in, let me give you some PetSets
>     > highlights:
>     >
>      >   * PetSets are alpha resources for managing stateful apps in
>     Kubernetes
>     >     1.3 (and OpenShift Origin 1.3).
>      >   * Since this is an alpha resource, there are no guarantees about
>     >     backwards compatibility. Alpha resources can also be disabled in
>     >     some public cloud providers (you can control which API versions are
>     >     accessible [2]).
>      >   * PetSets allows starting pods in sequence (not relevant for
>     us, but
>     >     this is a killer feature for master-slave systems).
>      >   * Each Pod has it's own unique entry in DNS, which makes discovery
>     >     very simple (I'll dig into that a bit later)
>      >   * Volumes are always mounted to the same Pods, which is very
>     important
>     >     in Cache Store scenarios when we restart pods (e.g. Rolling Upgrades
>     >     [3]).
>     >
>     > Thoughts and ideas after spending some time playing with this feature:
>     >
>      >   * PetSets make discovery a lot easier. It's a combination of two
>     >     things - Headless Services [4] which create multiple A records in
>     >     DNS and predictable host names. Each Pod has it's own unique DNS
>     >     entry following pattern: {PetSetName}-{PodIndex}.{ServiceName} [5].
>     >     Here's an example of an Infinispan PetSet deployed on my local
>     >     cluster [6]. As you can see we have all domain names and IPs from a
>     >     single DNS query.
>      >   * Maybe we could perform discovery using this mechanism? I'm
>     aware of
>     >     DNS discovery implemented in KUBE_PING [7][8] but the code looks
>     >     trivial [9] so maybe it should be implement inside JGroups? @Bela -
>     >     WDYT?
>      >   * PetSets do not integrate well with OpenShift 'new-app'
>     command. In
>     >     other words, our users will need to use provided yaml (or json)
>     >     files to create Infinispan cluster. It's not a show-stopper but it's
>     >     a bit less convenient than 'oc new-app'.
>      >   * Since PetSets are alpha resources they need to be considered as
>     >     secondary way to deploy Infinispan on Kubernetes and OpenShift.
>      >   * Finally, the persistent volumes - since a Pod always gets the
>     same
>      >     volume, it would be safe to use any file-based cache store.
>      >
>      > If you'd like to play with PetSets on your local environment,
>     here are
>      > necessary yaml files [10].
>      >
>      > Thanks
>      > Sebastian
>      >
>      >
>      > [1] http://kubernetes.io/docs/user-guide/petset/
>     <http://kubernetes.io/docs/user-guide/petset/>
>      > [2] For checking which APIs are accessible, use 'kubectl
>     api-versions'
>      > [3]
>      >
>     http://infinispan.org/docs/stable/user_guide/user_guide.html#_Rolling_chapter
>     <http://infinispan.org/docs/stable/user_guide/user_guide.html#_Rolling_chapter>
>      > [4]
>     http://kubernetes.io/docs/user-guide/services/#headless-services
>     <http://kubernetes.io/docs/user-guide/services/#headless-services>
>      > [5] http://kubernetes.io/docs/user-guide/petset/#peer-discovery
>     <http://kubernetes.io/docs/user-guide/petset/#peer-discovery>
>      > [6]
>     https://gist.github.com/slaskawi/0866e63a39276f8ab66376229716a676
>     <https://gist.github.com/slaskawi/0866e63a39276f8ab66376229716a676>
>      > [7]
>     https://github.com/jboss-openshift/openshift-ping/tree/master/dns
>     <https://github.com/jboss-openshift/openshift-ping/tree/master/dns>
>      > [8]
>     https://github.com/jgroups-extras/jgroups-kubernetes/tree/master/dns
>     <https://github.com/jgroups-extras/jgroups-kubernetes/tree/master/dns>
>      > [9] http://stackoverflow.com/a/12405896/562699
>     <http://stackoverflow.com/a/12405896/562699>
>      > [10] You might need to adjust ImageStream.
>      > https://gist.github.com/slaskawi/7cffb5588dabb770f654557579c5f2d0
>     <https://gist.github.com/slaskawi/7cffb5588dabb770f654557579c5f2d0>
>
>     --
>     Bela Ban, JGroups lead (http://www.jgroups.org)
>
>     _______________________________________________
>     infinispan-dev mailing list
>     infinispan-dev at lists.jboss.org <mailto:infinispan-dev at lists.jboss.org>
>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
>     <https://lists.jboss.org/mailman/listinfo/infinispan-dev>
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)