Hi Sebastian the usual restrictions apply: if DNS discovery depends on external libs, then it should be hosted in jgroups-extras, otherwise we can add it to JGroups itself. On 19/08/16 11:00, Sebastian Laskawiec wrote: > Hey! > > I've been playing with Kubernetes PetSets [1] for a while and I'd like > to share some thoughts. Before I dig in, let me give you some PetSets > highlights: > > * PetSets are alpha resources for managing stateful apps in Kubernetes > 1.3 (and OpenShift Origin 1.3). > * Since this is an alpha resource, there are no guarantees about > backwards compatibility. Alpha resources can also be disabled in > some public cloud providers (you can control which API versions are > accessible [2]). > * PetSets allows starting pods in sequence (not relevant for us, but > this is a killer feature for master-slave systems). > * Each Pod has it's own unique entry in DNS, which makes discovery > very simple (I'll dig into that a bit later) > * Volumes are always mounted to the same Pods, which is very important > in Cache Store scenarios when we restart pods (e.g. Rolling Upgrades > [3]). > > Thoughts and ideas after spending some time playing with this feature: > > * PetSets make discovery a lot easier. It's a combination of two > things - Headless Services [4] which create multiple A records in > DNS and predictable host names. Each Pod has it's own unique DNS > entry following pattern: {PetSetName}-{PodIndex}.{ServiceName} [5]. > Here's an example of an Infinispan PetSet deployed on my local > cluster [6]. As you can see we have all domain names and IPs from a > single DNS query. > * Maybe we could perform discovery using this mechanism? I'm aware of > DNS discovery implemented in KUBE_PING [7][8] but the code looks > trivial [9] so maybe it should be implement inside JGroups? @Bela - > WDYT? > * PetSets do not integrate well with OpenShift 'new-app' command. In > other words, our users will need to use provided yaml (or json) > files to create Infinispan cluster. It's not a show-stopper but it's > a bit less convenient than 'oc new-app'. > * Since PetSets are alpha resources they need to be considered as > secondary way to deploy Infinispan on Kubernetes and OpenShift. > * Finally, the persistent volumes - since a Pod always gets the same > volume, it would be safe to use any file-based cache store. > > If you'd like to play with PetSets on your local environment, here are > necessary yaml files [10]. > > Thanks > Sebastian > > > [1] http://kubernetes.io/docs/user-guide/petset/ > [2] For checking which APIs are accessible, use 'kubectl api-versions' > [3] > http://infinispan.org/docs/stable/user_guide/user_guide.html > #_Rolling_chapter > [4] http://kubernetes.io/docs/user-guide/services/#headless-services > [5] http://kubernetes.io/docs/user-guide/petset/#peer-discovery > [6] https://gist.github.com/slaskawi/0866e63a39276f8ab66376229716a676 > [7] https://github.com/jboss-openshift/openshift-ping/tree/master/dns > [8] https://github.com/jgroups-extras/jgroups-kubernetes/tree/master/dns > [9] http://stackoverflow.com/a/12405896/562699 > [10] You might need to adjust ImageStream. > https://gist.github.com/slaskawi/7cffb5588dabb770f654557579c5f2d0 > -- Bela Ban, JGroups lead (http://www.jgroups.org)

On Fri, Aug 19, 2016 at 10:18 AM, Bela Ban <bban(a)redhat.com <mailto:bban@redhat.com>> wrote: Hi Sebastian the usual restrictions apply: if DNS discovery depends on external libs, then it should be hosted in jgroups-extras, otherwise we can add it to JGroups itself. Bela, you forgot to mention an extra restriction: even if it has no external deps, it should not be too big in terms of LOC (what is the threshold?) ;)

Gustavo On 19/08/16 11:00, Sebastian Laskawiec wrote: > Hey! > > I've been playing with Kubernetes PetSets [1] for a while and I'd like > to share some thoughts. Before I dig in, let me give you some PetSets > highlights: > > * PetSets are alpha resources for managing stateful apps in Kubernetes > 1.3 (and OpenShift Origin 1.3). > * Since this is an alpha resource, there are no guarantees about > backwards compatibility. Alpha resources can also be disabled in > some public cloud providers (you can control which API versions are > accessible [2]). > * PetSets allows starting pods in sequence (not relevant for us, but > this is a killer feature for master-slave systems). > * Each Pod has it's own unique entry in DNS, which makes discovery > very simple (I'll dig into that a bit later) > * Volumes are always mounted to the same Pods, which is very important > in Cache Store scenarios when we restart pods (e.g. Rolling Upgrades > [3]). > > Thoughts and ideas after spending some time playing with this feature: > > * PetSets make discovery a lot easier. It's a combination of two > things - Headless Services [4] which create multiple A records in > DNS and predictable host names. Each Pod has it's own unique DNS > entry following pattern: {PetSetName}-{PodIndex}.{ServiceName} [5]. > Here's an example of an Infinispan PetSet deployed on my local > cluster [6]. As you can see we have all domain names and IPs from a > single DNS query. > * Maybe we could perform discovery using this mechanism? I'm aware of > DNS discovery implemented in KUBE_PING [7][8] but the code looks > trivial [9] so maybe it should be implement inside JGroups? @Bela - > WDYT? > * PetSets do not integrate well with OpenShift 'new-app' command. In > other words, our users will need to use provided yaml (or json) > files to create Infinispan cluster. It's not a show-stopper but it's > a bit less convenient than 'oc new-app'. > * Since PetSets are alpha resources they need to be considered as > secondary way to deploy Infinispan on Kubernetes and OpenShift. > * Finally, the persistent volumes - since a Pod always gets the same > volume, it would be safe to use any file-based cache store. > > If you'd like to play with PetSets on your local environment, here are > necessary yaml files [10]. > > Thanks > Sebastian > > > [1] http://kubernetes.io/docs/user-guide/petset/ <http://kubernetes.io/docs/user-guide/petset/> > [2] For checking which APIs are accessible, use 'kubectl api-versions' > [3] > http://infinispan.org/docs/stable/user_guide/user_guide.html#_Rolling_cha... <http://infinispan.org/docs/stable/user_guide/user_guide.html#_Rolling_cha... > [4] http://kubernetes.io/docs/user-guide/services/#headless-services <http://kubernetes.io/docs/user-guide/services/#headless-services> > [5] http://kubernetes.io/docs/user-guide/petset/#peer-discovery <http://kubernetes.io/docs/user-guide/petset/#peer-discovery> > [6] https://gist.github.com/slaskawi/0866e63a39276f8ab66376229716a676 <https://gist.github.com/slaskawi/0866e63a39276f8ab66376229716a676> > [7] https://github.com/jboss-openshift/openshift-ping/tree/master/dns <https://github.com/jboss-openshift/openshift-ping/tree/master/dns> > [8] https://github.com/jgroups-extras/jgroups-kubernetes/tree/master/dns <https://github.com/jgroups-extras/jgroups-kubernetes/tree/master/dns> > [9] http://stackoverflow.com/a/12405896/562699 <http://stackoverflow.com/a/12405896/562699> > [10] You might need to adjust ImageStream. > https://gist.github.com/slaskawi/7cffb5588dabb770f654557579c5f2d0 <https://gist.github.com/slaskawi/7cffb5588dabb770f654557579c5f2d0> -- Bela Ban, JGroups lead (http://www.jgroups.org) _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org <mailto:infinispan-dev@lists.jboss.org> https://lists.jboss.org/mailman/listinfo/infinispan-dev <https://lists.jboss.org/mailman/listinfo/infinispan-dev> _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

A little late to the discussion but if all of the pods share the same set of volumes, couldn't you also use the FILE_PING discovery protocol between the pods? Thanks, Alan ------------------------------ *From: *"Sebastian Laskawiec" <slaskawi(a)redhat.com&gt; *To: *"infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt;, "Bela Ban" <bban(a)redhat.com&gt;, "Kevin Conner" <kconner(a)redhat.com&gt;, "Rob Cernich" < rcernich(a)redhat.com&gt; *Sent: *Friday, August 19, 2016 5:00:12 AM *Subject: *[infinispan-dev] Infinispan and OpenShift/Kubernetes PetSets Hey! I've been playing with Kubernetes PetSets [1] for a while and I'd like to share some thoughts. Before I dig in, let me give you some PetSets highlights: - PetSets are alpha resources for managing stateful apps in Kubernetes 1.3 (and OpenShift Origin 1.3). - Since this is an alpha resource, there are no guarantees about backwards compatibility. Alpha resources can also be disabled in some public cloud providers (you can control which API versions are accessible [2]). - PetSets allows starting pods in sequence (not relevant for us, but this is a killer feature for master-slave systems). - Each Pod has it's own unique entry in DNS, which makes discovery very simple (I'll dig into that a bit later) - Volumes are always mounted to the same Pods, which is very important in Cache Store scenarios when we restart pods (e.g. Rolling Upgrades [3]). Thoughts and ideas after spending some time playing with this feature: - PetSets make discovery a lot easier. It's a combination of two things - Headless Services [4] which create multiple A records in DNS and predictable host names. Each Pod has it's own unique DNS entry following pattern: {PetSetName}-{PodIndex}.{ServiceName} [5]. Here's an example of an Infinispan PetSet deployed on my local cluster [6]. As you can see we have all domain names and IPs from a single DNS query. - Maybe we could perform discovery using this mechanism? I'm aware of DNS discovery implemented in KUBE_PING [7][8] but the code looks trivial [9] so maybe it should be implement inside JGroups? @Bela - WDYT? - PetSets do not integrate well with OpenShift 'new-app' command. In other words, our users will need to use provided yaml (or json) files to create Infinispan cluster. It's not a show-stopper but it's a bit less convenient than 'oc new-app'. - Since PetSets are alpha resources they need to be considered as secondary way to deploy Infinispan on Kubernetes and OpenShift. - Finally, the persistent volumes - since a Pod always gets the same volume, it would be safe to use any file-based cache store. If you'd like to play with PetSets on your local environment, here are necessary yaml files [10]. Thanks Sebastian [1] http://kubernetes.io/docs/user-guide/petset/ [2] For checking which APIs are accessible, use 'kubectl api-versions' [3] http://infinispan.org/docs/stable/user_guide/user_ guide.html#_Rolling_chapter [4] http://kubernetes.io/docs/user-guide/services/#headless-services [5] http://kubernetes.io/docs/user-guide/petset/#peer-discovery [6] https://gist.github.com/slaskawi/0866e63a39276f8ab66376229716a676 [7] https://github.com/jboss-openshift/openshift-ping/tree/master/dns [8] https://github.com/jgroups-extras/jgroups-kubernetes/tree/master/dns [9] http://stackoverflow.com/a/12405896/562699 [10] You might need to adjust ImageStream. https://gist.github.com/ slaskawi/7cffb5588dabb770f654557579c5f2d0 _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev