4:21 a.m.
Hey,
when WildFly connects to Inventory for the 1st time, we sync
the EAP information into inventory, which also includes the information
about which metrics are available.
Now when WildFly is deployed into Kubernetes or OpenShift, we
will see that WildFly is started, syncs and then dies at some point
in time, where k8s will not re-use the existing one, but start
a new one, which will have a different FeedId.
This will leave a WildFly in Inventory, that is later detected as
down in Metrics/Alerting, but the one in Inventory will stay
forever. Consequences are
- Inventory will get "full" with no-longer needed information
- Clients will retrieve data about non-"usable" servers
We need to figure out how to deal with this situation like e.g.:
- have inventory listen on k8s events and remove the server
when k8s removes it (not when it is going down; stopped pods
can stay around for some time)
- Have a different way of creating the feed id / server id so that
the state is "re-used". Something like feedId/server name could
be the name of the deployment config + version + k8s tenant.
Thoughts?
4:35 a.m.
Re-using the state would mean there's always 1 or 0 running instance of
that server at a given time, but what happens if it's scaled up to 2
instances or more? Is it a possible scenario? If yes, it seems complicated
to use dc+version+tenant as the feed id.
I filed an issue in JIRA about introducing some kind of TTL :
https://issues.jboss.org/browse/HWKINVENT-205
Maybe those expired feeds fall in the scope of this issue.
On Tue, Jan 24, 2017 at 10:21 AM, Heiko W.Rupp <hrupp(a)redhat.com> wrote:
Hey,
when WildFly connects to Inventory for the 1st time, we sync
the EAP information into inventory, which also includes the information
about which metrics are available.
Now when WildFly is deployed into Kubernetes or OpenShift, we
will see that WildFly is started, syncs and then dies at some point
in time, where k8s will not re-use the existing one, but start
a new one, which will have a different FeedId.
This will leave a WildFly in Inventory, that is later detected as
down in Metrics/Alerting, but the one in Inventory will stay
forever. Consequences are
- Inventory will get "full" with no-longer needed information
- Clients will retrieve data about non-"usable" servers
We need to figure out how to deal with this situation like e.g.:
- have inventory listen on k8s events and remove the server
when k8s removes it (not when it is going down; stopped pods
can stay around for some time)
- Have a different way of creating the feed id / server id so that
the state is "re-used". Something like feedId/server name could
be the name of the deployment config + version + k8s tenant.
Thoughts?
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
6:01 a.m.
On 24 Jan 2017, at 10:35, Joel Takvorian wrote:
Re-using the state would mean there's always 1 or 0 running
instance of
that server at a given time, but what happens if it's scaled up to 2
instances or more? Is it a possible scenario? If yes, it seems
Yes of course it is possible. And the number of running
servers for a given DC can be scaled up and down "at will".
complicated
to use dc+version+tenant as the feed id.
Yes I am aware that this is not enough - was more meant
to be an example.
I filed an issue in JIRA about introducing some kind of TTL :
https://issues.jboss.org/browse/HWKINVENT-205
Maybe those expired feeds fall in the scope of this issue.
Yes, a TTL could be a good solution. If a server re-connects
we reset the "last seen" value. We need to make sure to set the
TTL long enough that a short outage does not expire
servers that are not in containers.
Perhaps we need to also store some information in inventory
if the server is in containers or not (well, we need that
info anyway, see e.g.
https://issues.jboss.org/browse/HAWKULAR-1186 )
For the inventory use case we should perhaps expose
this more prominently than just 'hidden away' in some
property in some json.
5:13 a.m.
On 01/24/2017 10:21 AM, Heiko W.Rupp wrote:
Now when WildFly is deployed into Kubernetes or OpenShift, we
will see that WildFly is started, syncs and then dies at some point
in time, where k8s will not re-use the existing one, but start
a new one, which will have a different FeedId.
It might be interesting to re-read the following threads, as some ideas
were thrown around this subject already:
[Hawkular-dev] OpenShift Pet vs Cattle metaphor (10/13/2016)
[Hawkular-dev] Identification of WildFly in container in a
Kube/Openshift env (07/03/2016)
- Juca.
6:12 a.m.
On 24 Jan 2017, at 11:13, Juraci Paixão Kröhling wrote:
[Hawkular-dev] OpenShift Pet vs Cattle metaphor (10/13/2016)
[Hawkular-dev] Identification of WildFly in container in a
Kube/Openshift env (07/03/2016)
Thanks for the reminder - I had a feeling I wrote about that in the
past :)
And I missed the cited Pet vs Cattle thread even :-(
I think though, that requiring WildFly running in a PetSet for
this to work is a nono
8:19 a.m.
Another possible angle is to somehow involve alerting and a possible
custom action. We have the ability to do things when detecting down, or
if we haven't seen an up avail in some period of time, or to respond to
inventory events.
On 1/24/2017 6:12 AM, Heiko W.Rupp wrote:
On 24 Jan 2017, at 11:13, Juraci Paixão Kröhling wrote:
> [Hawkular-dev] OpenShift Pet vs Cattle metaphor (10/13/2016)
> [Hawkular-dev] Identification of WildFly in container in a
> Kube/Openshift env (07/03/2016)
Thanks for the reminder - I had a feeling I wrote about that in the
past :)
And I missed the cited Pet vs Cattle thread even :-(
I think though, that requiring WildFly running in a PetSet for
this to work is a nono
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
2646
days inactive
2647
days old
5 comments
4 participants
participants (4)
-
Heiko W.Rupp -
Jay Shaughnessy -
Joel Takvorian -
Juraci Paixão Kröhling