10:54 a.m.
Hi,
I know I brought that up in the past ... (and I am sure Juca will easily
find where :-)
In the good old datacenter, one deployed an application server with
application, added it to monitoring and it hummed there happily. The
server sometimes got rebooted for OS updates or other things, but the
app-server always stayed the same and the monitoring system knew all its
pets by their name and the admins happily did this:
http://www.starwars-union.de/bilder/news/20110401_sunset.jpg
Nowadays in orchestration system the situation more looks like this
http://www.animationsfilme.ch/wp-content/uploads/2013/07/DespicableMe_01.jpg
where containers come and go and an application in container once it has
died is not re-started but a new container with its own ID is started.
Of course we can identify applications with labels so that I don't need
to know the container id. So I can gather and display metrics for those
and all is fine.
But: all those containers will create new
- metric ids
- inventory entries
- ???
The question is now: how long do we want/need to keep them?
Hawkular-metrics has a TTL for the datapoints, but I think metric
definitions are not evicted.
Similar a container being killed can't easily tell inventory that its
entry can go away.
For inventory-new we could use the expiration feature of
Hawkular-metrics for datapoints, where e.g. the agent would regularly
sync data and thus refresh the last-seen time to keep an entry "alive".
Also for the pure metrics - how much of historic data do we want/need.
And if we would e.g. aggregate those for long(er) term storage I think
we could perhaps actually aggregate over all individual time series over
many parallel pods and aggregate them into one for the entire
application.
Heiko
2:21 a.m.
For inventory-new we could use the expiration feature of
Hawkular-metrics for datapoints, where e.g. the agent would regularly
sync data and thus refresh the last-seen time to keep an entry "alive".
If datapoints expire (that is, inventory "blobs"), their underlying metrics
will still be present in the database, even with no data associated. We may
want to delete them as well.
2:52 a.m.
On 9 Mar 2017, at 9:21, Joel Takvorian wrote:
If datapoints expire (that is, inventory "blobs"), their
underlying
metrics
will still be present in the database, even with no data associated.
We may want to delete them as well.
While I would expect them to also expire by TTL.
It is though a discussion we need to have, as this may not be
what is wanted:
Suppose you have an application with label app=shop.
Now that application has a component that comes from a 3rd party vendor
and which has a memory hole. The vendor is not willing to provide a
quick fix, but only at their usual 3 month cadence. To keep your shop
online, you will either 'manually' restart it every night. Or have a
liveness check for Kubernetes, which checks for free memory and when
this is exhausted will report the container as non-live. Kubernetes will
kill the pod and start a new one.
In any case here, you get a new pod for your app=shop and the old server
in inventory is no longer needed, but e.g. for capacity planning you
want to keep all metrics for app=shop.
What may hold true though is that e.g. after a week the granularity of
the data may decrease and we may only keep aggregates. In this case the
auto-expiry of the data could/would need to trigger an aggregation into
another time-series that holds the aggregates. And perhaps at that point
(as I wrote before) drop the container id(*) part from the metric id and
actually "aggregate into the label" E.g.
container1.cpu.usage{app=shop1} -> label.app=shop1.cpu.usage
container2.cpu.usage{app=shop1} -> label.app=shop1.cpu.usage
(the 'label.' prefix is used here to distinguish from "normal metrics"
for illustration purposes
If container 1 and 2 above happen to run in parallel then the "usual"
aggregation rules for a group would apply
label.app=shop1.cpu.usage.max = max (container1.cpu.usage{app=shop1},
container2.cpu.usage{app=shop1})
and so on. Basically aggregation of all parallel running containers and
over a time interval.
4:35 a.m.
On Wednesday, March 8, 2017 5:54:46 PM CET Heiko W.Rupp wrote:
Hi,
I know I brought that up in the past ... (and I am sure Juca will easily
find where :-)
In the good old datacenter, one deployed an application server with
application, added it to monitoring and it hummed there happily. The
server sometimes got rebooted for OS updates or other things, but the
app-server always stayed the same and the monitoring system knew all its
pets by their name and the admins happily did this:
http://www.starwars-union.de/bilder/news/20110401_sunset.jpg
Nowadays in orchestration system the situation more looks like this
http://www.animationsfilme.ch/wp-content/uploads/2013/07/DespicableMe_01.jpg
where containers come and go and an application in container once it has
died is not re-started but a new container with its own ID is started.
Of course we can identify applications with labels so that I don't need
to know the container id. So I can gather and display metrics for those
and all is fine.
But: all those containers will create new
- metric ids
- inventory entries
- ???
The question is now: how long do we want/need to keep them?
Isn't the question rather "how do we associate them with the
application(s)"?
Because if we want to track e.g. CPU load generated by an application in a
container - isn't that something users would want to look at history of? IMHO,
using the ephemeral "container id" as (part of) metric ids is a wrong thing to
do, because really the user isn't interested in the container itself, but the
applications that are running in it and their consumption of container's
resources.
Hawkular-metrics has a TTL for the datapoints, but I think metric
definitions are not evicted.
Similar a container being killed can't easily tell inventory that its
entry can go away.
For inventory-new we could use the expiration feature of
Hawkular-metrics for datapoints, where e.g. the agent would regularly
sync data and thus refresh the last-seen time to keep an entry "alive".
Also for the pure metrics - how much of historic data do we want/need.
And if we would e.g. aggregate those for long(er) term storage I think
we could perhaps actually aggregate over all individual time series over
many parallel pods and aggregate them into one for the entire
application.
Heiko
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
--
Lukas Krejci
6:13 a.m.
On 9 Mar 2017, at 11:35, Lukas Krejci wrote:
> The question is now: how long do we want/need to keep them?
Isn't the question rather "how do we associate them with the
application(s)"?
Yes. And I think labels are the way of choice here.
Because if we want to track e.g. CPU load generated by an application
in a container - isn't that something users would want to look at
history of? IMHO, using the ephemeral "container id" as (part of)
metric ids is a wrong thing to
do, because really the user isn't interested in the container itself,
but the applications that are running in it and their consumption of
container's resources.
I think one needs to differentiate two time ranges here
- container active or just died
- historic view in the application
In both time ranges it is of course important to know how the overall
application behaves.
While the container is active one does not only want to know how the
application behaves, but also how each individual container does. If
there is only one running in parallel (scale=1) then it is less of an
issue as application=container.
But once you scale the application up, this is no longer true. And
individual containers may behave differently ( e.g. this memory hole I
described does not always show ) so one may want to have the data to act
accordingly. Either live or in a post mortem.
When aggregation kicks in, the difference between containers goes moot
and just looking at it by label is ok.
I think for most (all?) aggregations it will not matter if the data is
reported to one time-series for all parallel running containers or on
one per container.
6:35 a.m.
Hi,
This is actually a solved issue if you want to track a single metric's
history. Remember, metricId has no context information, it could (and it
should have been so that people don't get confused) a randomly generated
uuid.
The way you track the container's historic CPU usage is by using the tags:
For example, to get all the datapoints for a single container, you use:
container_name=hawkular-metrics
The podid does not make any difference here, it's just some random
information. Now, on the other hand, if you want to compare how the
hawkular-metrics memory usage has changed between different pod versions
or pod ids, you can do that also by actually using the tags of pod_id
and container_version.
Just use pod_id as one sorting method and container_version as one (I
don't remember the exact names of these tags). Tags are the key to
sorting and finding the information you want - not the metricId.
MetricId is something you should completely forget, it's not relevant.
The Kubernetes metrics have tags available that you can use for sorting,
grouping and finding.
- Micke
On 03/09/2017 12:35 PM, Lukas Krejci wrote:
Isn't the question rather "how do we associate them with the
application(s)"?
Because if we want to track e.g. CPU load generated by an application in a
container - isn't that something users would want to look at history of? IMHO,
using the ephemeral "container id" as (part of) metric ids is a wrong thing to
do, because really the user isn't interested in the container itself, but the
applications that are running in it and their consumption of container's
resources.
4:29 p.m.
If I understand how tags are getting used, I think tags will provide the
ability to keep track of history regardless of whether we are dealing with
a new container or an application which is no longer alive.
However, what is not clear to me is what happens when the old container
goes away. I am assuming that we will no longer be attempting to probe it
or gather data from it?
Ultimately what I, as a user, want to be able to do is change network
configuration or platform or container without changing what defines my
application. For example, if I have a helloworld.war deployed to an EAP
server and am monitoring it, I really don't care about the other stuff. If
it changes, so be it. But my helloworld application should still be
reflected the same.
If the database my helloworld application uses changes, so be it. I still
want to treat the datasource as the same datasource it always was even if
the underlying implementation changes and a new inventory object to
represent it gets created. My expectation is that _helloworld-datasource_
is _datasource-pg94_ and _datasource-pg95_ and _datasource_mysql_ combined.
Even though in reality I started with PostgreSQL 9.4; upgraded to
PostgreSQL 9.5; and decided to migrate to MySQL at some point. In the end,
I want to view the datasource metrics based on the entire life-cycle as
that datasource relates to my helloworld application.
The same could be said for my helloworld application's underlying
application server. Perhaps I have my WAR deployed to EAP 7.1 to start but
later I upgrade to EAP 8. It is still my helloworld application.
In the past I would suggest providing a "linking" capability that would
allow me to identify the EAP 8 instance as a continuation or upgrade of the
specific EAP 7.1 instance. Or the MySQL datasource as a continuation or
upgrade of the PostgreSQL 9.5 datasource and the 9.5 datasource an upgrade
or continuation of the PostgreSQL 9.4 datasource. However, if I understand
how tagging is getting implemented, tags would also accomplish this.
If I have separate objects in inventory, they would just get tagged as
helloworld application. When I view history, I am seeing the milestones
across the various inventory objects that make up what I am referring to as
helloworld application. For example, if I had to spin up a new container
every day for the last 7 days, then perhaps it would be interesting to see
that indicated in the history, but the metric values are considered to be a
single series as they relate to the all 7 containers.
But to take the burden of the user to tag each new container that
represents helloworld, I would hope that we can figure out how to do this
automatically and then allow the user to update/fix tags if they are wrong?
In any event, just my thoughts as this does relate to a very big pain point
with the legacy RHQ and JBoss ON implementation.
--
Larry O'Leary
On Fri, Mar 10, 2017 at 6:35 AM, Michael Burman <miburman(a)redhat.com> wrote:
Hi,
This is actually a solved issue if you want to track a single metric's
history. Remember, metricId has no context information, it could (and it
should have been so that people don't get confused) a randomly generated
uuid.
The way you track the container's historic CPU usage is by using the tags:
For example, to get all the datapoints for a single container, you use:
container_name=hawkular-metrics
The podid does not make any difference here, it's just some random
information. Now, on the other hand, if you want to compare how the
hawkular-metrics memory usage has changed between different pod versions
or pod ids, you can do that also by actually using the tags of pod_id
and container_version.
Just use pod_id as one sorting method and container_version as one (I
don't remember the exact names of these tags). Tags are the key to
sorting and finding the information you want - not the metricId.
MetricId is something you should completely forget, it's not relevant.
The Kubernetes metrics have tags available that you can use for sorting,
grouping and finding.
- Micke
On 03/09/2017 12:35 PM, Lukas Krejci wrote:
> Isn't the question rather "how do we associate them with the
> application(s)"?
> Because if we want to track e.g. CPU load generated by an application in
a
> container - isn't that something users would want to look at history of?
IMHO,
> using the ephemeral "container id" as (part of) metric ids is a wrong
thing to
> do, because really the user isn't interested in the container itself,
but the
> applications that are running in it and their consumption of container's
> resources.
>
>
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
8:11 a.m.
On 03/14/2017 11:29 PM, Larry O'Leary wrote:
If I understand how tags are getting used, I think tags will provide
the ability to keep track of history regardless of whether
But to take the burden of the user to tag each new container that
represents helloworld, I would hope that we can figure out how to do
this automatically and then allow the user to update/fix tags if they
are wrong?
I think most of your concerns are not our domain anymore, but instead
how Openshift for example manages it and whether we can store that
"inventory information". Here's one example of what we've extracted by
gathering information:
{
"tags": {
"resource_id_description": "Identifier(s) specific to a metric",
"labels":
"deployment:router-1,deploymentconfig:router,router:router",
"nodename": "localhost",
"resource_id": "/",
"type": "pod_container",
"hostname": "localhost",
"container_base_image":
"openshift/origin-haproxy-router:latest",
"namespace_id": "ef59e1bb-ea0d-11e6-9dc8-a0d3c1f893c0",
"descriptor_name": "filesystem/usage",
"pod_name": "router-1-bwvdt",
"container_name": "router",
"units": "bytes",
"host_id": "localhost",
"group_id": "router/filesystem/usage",
"pod_namespace": "default",
"pod_id": "fe42efce-ea0d-11e6-9dc8-a0d3c1f893c0",
"namespace_name": "default"
},
"tenantId": "default",
"dataRetention": 7,
"minTimestamp": 1488967200000,
"type": "gauge",
"id":
"router/fe42efce-ea0d-11e6-9dc8-a0d3c1f893c0/filesystem/usage//",
"maxTimestamp": 1489581220000
}
If your "helloworld" is always running in the project
"ef59e1bb-ea0d-11e6-9dc8-a0d3c1f893c0" then there's one search criteria
that will not change even if you redeploy it or change the EAP to a
newer version etc.
Next up we can look at descriptor_name (or group_id), that is the metric
you might want to follow. This one does not change either, so you can
still keep a track of the newest running instance in this way. labels
are in Openshift which gives you the information you might seek, such as
labeling it "application_name = Our ultimate frontend application". Now
you can track with that name also, regardless of whether the pod changed
underneath.
So these issues you've noted are mostly due to how you use Openshift. If
you mark it there so that it can be followed, then our tags allow you to
follow it. But now it gives you the ability to follow it, even if you
have multiple pods running concurrently or if you update versions, the
pod crashes, changes node etc.
But, you can also track all those options also. "Show me the
filesystem/usage between node1 and node2" or "show me how the previous
version compared to new version". Or list all the instances running
physically on node "localhost".
We don't do inventoring for the container environments, but we try to
gather as much information as possible to allow the end-user to create
their own groupings without restricting the possibilities. And in
dynamic environment.
I'm not sure if I answered to your problem definition - as I don't think
it's in our hands anymore. This comment might not apply to ManageIQ
environment ;)
Now, once this pod goes down this metricId will not receive new metrics
(new pod_id will come up and new metricId created). We don't however
delete this history of old pod_ids and their metric definitions even if
the datapoints associated with them have expired. Your queries - if done
with tags - do not change and they do not have to care how many times
the underlying pod was restarted. Which I think was one of your worries.
Consider the metricId as internal information of Hawkular-Metrics.
- Micke
2810
days inactive
2817
days old
7 comments
5 participants
participants (5)
-
Heiko W.Rupp -
Joel Takvorian -
Larry O'Leary -
Lukas Krejci -
Michael Burman