Hey Kevin!
The timing for looking into PetSets is perfect I think. Kubernetes upstream
folks are thinking about Sticky IPs [16] (which are not essential for
Infinispan but other projects might be really interested in this) and I
asked some time ago about exposing PetSets to the outside world [17] (which
is essential for accessing the cluster using Hot Rod client).
Please keep me in the loop. I'm really interested in this.
Thanks
Sebastian
[16]
https://github.com/kubernetes/kubernetes/issues/28969
[17]
https://groups.google.com/forum/#!topic/kubernetes-dev/K-9KA_wMbmk
On Sun, Aug 21, 2016 at 7:05 PM, Kevin Conner <kconner(a)redhat.com> wrote:
Apologies, I’ve been travelling to London for the JDG meeting.
On 19 Aug 2016, at 10:00, Sebastian Laskawiec <slaskawi(a)redhat.com> wrote:
> I've been playing with Kubernetes PetSets [1] for a while and I'd like
to share some thoughts. Before I dig in, let me give you some PetSets
highlights:
> • PetSets are alpha resources for managing stateful apps in
Kubernetes 1.3 (and OpenShift Origin 1.3).
> • Since this is an alpha resource, there are no guarantees about
backwards compatibility. Alpha resources can also be disabled in some
public cloud providers (you can control which API versions are accessible
[2]).
> • PetSets allows starting pods in sequence (not relevant for us,
but this is a killer feature for master-slave systems).
> • Each Pod has it's own unique entry in DNS, which makes discovery
very simple (I'll dig into that a bit later)
> • Volumes are always mounted to the same Pods, which is very
important in Cache Store scenarios when we restart pods (e.g. Rolling
Upgrades [3]).
> Thoughts and ideas after spending some time playing with this feature:
> • PetSets make discovery a lot easier. It's a combination of two
things - Headless Services [4] which create multiple A records in DNS and
predictable host names. Each Pod has it's own unique DNS entry following
pattern: {PetSetName}-{PodIndex}.{ServiceName} [5]. Here's an example of
an Infinispan PetSet deployed on my local cluster [6]. As you can see we
have all domain names and IPs from a single DNS query.
> • Maybe we could perform discovery using this mechanism? I'm aware
of DNS discovery implemented in KUBE_PING [7][8] but the code looks trivial
[9] so maybe it should be implement inside JGroups? @Bela - WDYT?
> • PetSets do not integrate well with OpenShift 'new-app' command.
In other words, our users will need to use provided yaml (or json) files to
create Infinispan cluster. It's not a show-stopper but it's a bit less
convenient than 'oc new-app'.
> • Since PetSets are alpha resources they need to be considered as
secondary way to deploy Infinispan on Kubernetes and OpenShift.
> • Finally, the persistent volumes - since a Pod always gets the
same volume, it would be safe to use any file-based cache store.
> If you'd like to play with PetSets on your local environment, here are
necessary yaml files [10].
PetSets are still in extremely early stages and I am part of a working
group that has recently formed to cover the introduction of this within the
products, including driving any necessary changes back upstream. There
will be Cloud Enablement involvement in this effort and any adoption of
petsets will likely go through the existing project.
Kev
--
JBoss by Red Hat