<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
<font face="Calibri">+1. It seems to me that underlying metric ids
are something we just want to hide as an implementation detail.
Querying for a "family name" and narrowing by other tags gives you
a useful set of TS.</font><br>
<br>
<div class="moz-cite-prefix">On 2/17/2017 2:44 AM, Joel Takvorian
wrote:<br>
</div>
<blockquote
cite="mid:CAJo5TFkmO42UQuNpQb01uiUy4gtj64w7jHH9KT3GvXUBkNCmrA@mail.gmail.com"
type="cite">
<div dir="ltr">For the curly braces in Grafana, I'm going to
investigate.
<div><br>
</div>
<div>For your second point, I'm trying to put me in the shoes of
an ops: if I want to create a dashboard that shows a labelled
metric (in term of prometheus label), I'd like to see all its
avatars in the same chart to be able to compare them, see in
what they converge or in what they diverge. And maybe compare
them in all pods of a given container name. That would be
queries with tags:</div>
<div><br>
</div>
<div>Query tags: </div>
<div>- container_name: something</div>
<div>- family_name (or "metric_base_name", or whatever name we
give to that tag): what_i_ate</div>
<div><br>
</div>
<div>I can't be 100% sure that it's going to be used, as people
do what they want in Grafana. But it seems interesting to me.
The question is: what's the cost of adding a tag? I believe
metric tags are relatively cheap in term of storage. So,
having both "metric_name" (what_i_ate{food=Banana}) and
"family_name" (what_i_ate) would solve all our issues, no?</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Feb 16, 2017 at 6:23 PM, John
Mazzitelli <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:mazz@redhat.com" target="_blank">mazz@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">I need to
resurrect this thread now that some others have had
experience with what we have - specifically, what Thomas
reported in this issue: <a moz-do-not-send="true"
href="https://github.com/hawkular/hawkular-openshift-agent/issues/126"
rel="noreferrer" target="_blank">https://github.com/hawkular/<wbr>hawkular-openshift-agent/<wbr>issues/126</a><br>
<br>
It has to do with Prometheus metrics and how HOSA names and
tags them in H-Metrics.<br>
<br>
Just some quick background first:<br>
<br>
Prometheus metrics have two parts - a "family name" (like
"http_response_count") and labels (like "method"). This
means you can have N metrics in Prometheus with the same
metric family name but each with different label values
(like "http_response_count{method=<wbr>GET}" and
"http_response_count{method=<wbr>POST}". Each unique
combination of family name plus label values represent a
different set of time series data (so
http_response_count{method=<wbr>GET} is one set of time
series data and http_response_count{method=<wbr>POST} is
another set of time series data).<br>
<br>
H-Metrics doesn't really have this concept of metric family.
H-Metrics has metric definitions each with unique names (or
"metric IDs") and a set of tags (h-metrics uses the name
"tags" rather than "labels"). In H-Metrics, you cannot have
N metrics with the same name (ID). You must have unique IDs
to represent different sets of time series data.<br>
<br>
OK, with that quick intro, two things:<br>
<br>
=====<br>
<br>
1) Metrics coming from Prometheus by default will be stored
in H-Metrics with metric IDs like:<br>
<br>
metric_family_name{label_<wbr>name1=value1,label_name2=<wbr>value2}<br>
<br>
Basically, HOSA stores the H-Metric ID so it looks identical
to the metric data coming from Prometheus endpoints (name
with labels comma-separated and enclosed within curly
braces).<br>
<br>
But Grafana might have issues with the curly braces.
However, the original opinion when this was first
implemented in HOSA was that just using underscores in
H-Metrics IDs, for example:<br>
<br>
metric_family_name_label_<wbr>name1_value1_label_name2_<wbr>value2<br>
<br>
will make querying from H-Metrics more difficult (it all
looks like one big name and it is hard to distinguish the
labels in the name).<br>
<br>
QUESTION #1a: Does Grafana really have an issue with
displaying metrics whose names have curly braces - {} - and
commas in them?<br>
QUESTION #1b: If so, what should the default metric ID look
like when we have Prometheus labels like this, if not by
using curly braces and commas?<br>
<br>
=====<br>
<br>
2) These Prometheus metrics don't look right in the current
OpenShift UI. If we have two Prometheus metrics stored in
H-Metrics with the IDs:<br>
<br>
what_i_ate{food=Banana}<br>
what_i_ate{food=Apple}<br>
<br>
what you see in the OpenShift UI console is two metric
graphs each with the same metric name "what_i_ate" - you
don't know which ones they are.<br>
<br>
Why? Application metrics like these are now shown in the
OpenShift UI and it works fine even for Prometheus metrics
UNLESS the Prometheus metrics had labels (like the example
above with Prometheus labels food=Apple or food=Banana).
This is because when we tag these metrics in H-Metrics, one
tag we add to the metric definition is "metric_name" and for
Prometheus the value of this tag is the METRIC FAMILY name.
This is what Joel was asking for (see the last messages in
this thread). But the OS UI console uses this metric_name
tag for the label of the graph (the full, real ID of the
metric is ugly to make sure its unique within the cluster -
e.g. "pod/3e4553ew-34553d-345433-<wbr>123a/custom/what_i_ate{food=<wbr>Banana}"
- so we don't really want to show that to a user).<br>
<br>
QUESTION #2a: Should I switch back and make metric_name be
the last part of the actual metric ID (not Prometheus family
name) like "what_i_ate{food=Banana}" so the OS UI console
works? Or do we fix the OS UI console to parse the full
metric ID and only show the last part (after the "/custom/"
part) thus leaving "metric_name" tag in H-Metrics be the
Prometheus metric family name and make querying easier (a-la
Joel's suggestion).<br>
<br>
QUESTION #2b: Is having metric family name a useful thing to
have as a H-Metric tag in the first place? If so, I will
have to get HOSA to create a new tag "base_metric_name" if
"metric_name" is to be fixed to get the OS UI to work. But
does having the Prometheus metric family name even a useful
thing? Joel seemed to think so; I would like to make sure it
is a useful thing before I go and implement this change.<br>
<div class="HOEnZb">
<div class="h5"><br>
----- Forwarded Message -----<br>
From: "John Mazzitelli" <<a moz-do-not-send="true"
href="mailto:mazz@redhat.com">mazz@redhat.com</a>><br>
To: "Discussions around Hawkular development" <<a
moz-do-not-send="true"
href="mailto:hawkular-dev@lists.jboss.org">hawkular-dev@lists.jboss.org</a>><br>
Sent: Wednesday, February 1, 2017 11:47:18 AM<br>
Subject: Re: [Hawkular-dev] HOSA and conversion from
prometheus to hawkular metrics<br>
<br>
<a moz-do-not-send="true"
href="https://github.com/hawkular/hawkular-openshift-agent/blob/master/deploy/openshift/hawkular-openshift-agent-configmap.yaml#L20"
rel="noreferrer" target="_blank">https://github.com/hawkular/<wbr>hawkular-openshift-agent/blob/<wbr>master/deploy/openshift/<wbr>hawkular-openshift-agent-<wbr>configmap.yaml#L20</a><br>
<br>
:D<br>
<br>
That's already there - the ${METRIC:name} resolves to
the name of the metric (not the new ID) and our default
config puts that tag on every metric.<br>
<br>
<br>
----- Original Message -----<br>
><br>
> +1, if that is not being done I think it would
good. Actually, it's probably<br>
> a good "best practice" as it make it easier to
slice and dice the data.<br>
><br>
> On 2/1/2017 10:35 AM, Joel Takvorian wrote:<br>
><br>
><br>
><br>
> +1<br>
><br>
> Conversion based on labels seems more sane.<br>
><br>
> I wonder if a new tag that recalls the prometheus
metric name would be<br>
> useful; ex. "baseName=jvm_memory_pool_<wbr>bytes_committed",
to retrieve all<br>
> metrics of that family. Just an idea.<br>
><br>
> On Wed, Feb 1, 2017 at 4:25 PM, John Mazzitelli
< <a moz-do-not-send="true"
href="mailto:mazz@redhat.com">mazz@redhat.com</a> >
wrote:<br>
><br>
><br>
> > Are you also tagging the Prometheus metrics
with the labels?<br>
><br>
> Yes, that is what was originally being done, and
that is still in there.<br>
><br>
> ----- Original Message -----<br>
> ><br>
> > Mazz, this makes sense to me. Our decision to
use unique ids (well +type)<br>
> > is<br>
> > going to lead to this sort of thing. The ids
are going to basically be<br>
> > large<br>
> > concatenations of the tags that identify the
data. Then, additionally we're<br>
> > going to have to tag the metrics with the same
name/value pairs that are<br>
> > present in the id. Are you also tagging the
Prometheus metrics with the<br>
> > labels?<br>
> ><br>
> > On 2/1/2017 9:38 AM, John Mazzitelli wrote:<br>
> ><br>
> ><br>
> ><br>
> > The past several days I've been working on an
enhancement to HOSA that came<br>
> > in from the community (in fact, I would
consider it a bug). I'm about ready<br>
> > to merge the PR [1] for this and do a HOSA
1.1.0.Final release. I wanted to<br>
> > post this to announce it and see if there is
any feedback, too.<br>
> ><br>
> > Today, HOSA collects metrics from any
Prometheus endpoint which you declare<br>
> > -<br>
> > example:<br>
> ><br>
> > metrics<br>
> > - name: go_memstats_sys_bytes<br>
> > - name: process_max_fds<br>
> > - name: process_open_fds<br>
> ><br>
> > But if a Prometheus metric has labels,
Prometheus itself considers each<br>
> > metric with a unique combination of labels as
an individual time series<br>
> > metric. This is different than how Hawkular
Metric works - each Hawkular<br>
> > Metric metric ID (even if its metric
definition or its datapoints have<br>
> > tags)<br>
> > is a single time series metric. We need to
account for this difference. For<br>
> > example, if our agent is configured with:<br>
> ><br>
> > metrics:<br>
> > - name: jvm_memory_pool_bytes_<wbr>committed<br>
> ><br>
> > And the Prometheus endpoint emits that metric
with a label called "pool"<br>
> > like<br>
> > this:<br>
> ><br>
> > jvm_memory_pool_bytes_<wbr>committed{pool="Code
Cache",} 2.7787264E7<br>
> > jvm_memory_pool_bytes_<wbr>committed{pool="PS
Eden Space",} 2.3068672E7<br>
> ><br>
> > then to Prometheus this is actually 2 time
series metrics (the number of<br>
> > bytes committed per pool type), not 1. Even
though the metric name is the<br>
> > same (what Prometheus calls a "metric family
name"), there are two unique<br>
> > combinations of labels - one with "Code Cache"
and one with "PS Eden Space"<br>
> > - so they are 2 distinct time series metric
data.<br>
> ><br>
> > Today, the agent only creates a single
Hawkular-Metric in this case, with<br>
> > each datapoint tagged with those Prometheus
labels on the appropriate data<br>
> > point. But we don't want to aggregate them
like that since we lose the<br>
> > granularity that the Prometheus endpoint gives
us (that is, the number of<br>
> > bytes committed in each pool type). I will say
I think we might be able to<br>
> > get that granularity back through datapoint
tag queries in Hawkular-Metrics<br>
> > but I don't know how well (if at all) that is
supported and how efficient<br>
> > such queries would be even if supported, and
how efficient storage of these<br>
> > metrics would be if we tag every data point
with these labels (not sure if<br>
> > that is the general purpose of tags in
H-Metrics). But, regardless, the<br>
> > fact<br>
> > that these really are different time series
metrics should (IMO) be<br>
> > represented as different time series metrics
(via metric definitions/metric<br>
> > IDs) in Hawkular-Metrics.<br>
> ><br>
> > To support labeled Prometheus endpoint data
like this, the agent needs to<br>
> > split this one named metric into N
Hawkular-Metrics metrics (where N is the<br>
> > number of unique label combinations for that
named metric). So even though<br>
> > the agent is configured with the one metric<br>
> > "jvm_memory_pool_bytes_<wbr>committed" we need
to actually create two<br>
> > Hawkular-Metric metric definitions (with two
different and unique metric<br>
> > IDs<br>
> > obviously).<br>
> ><br>
> > The PR [1] that is ready to go does this. By
default it will create<br>
> > multiple<br>
> > metric definitions/metric IDs in the form<br>
> > "metric-family-name{<wbr>labelName1=labelValue1,<wbr>labelName2=labelValue2,...}"<br>
> > unless you want a different form in which case
you can define an "id" and<br>
> > put in "${labelName}" in the ID you declare
(such as<br>
> > "${oneLabelName}_my_own_<wbr>metric_name_${<wbr>theOtherLabelName}"
or whatever). But<br>
> > I suspect the default format will be what most
people want and thus nothing<br>
> > needs to be done. In the above example, two
metric definitions with the<br>
> > following IDs are created:<br>
> ><br>
> > 1. jvm_memory_pool_bytes_<wbr>committed{pool=Code
Cache}<br>
> > 2. jvm_memory_pool_bytes_<wbr>committed{pool=PS
Eden Space}<br>
> ><br>
> > --John Mazz<br>
> ><br>
> > [1] <a moz-do-not-send="true"
href="https://github.com/hawkular/hawkular-openshift-agent/pull/117"
rel="noreferrer" target="_blank">https://github.com/hawkular/<wbr>hawkular-openshift-agent/pull/<wbr>117</a><br>
______________________________<wbr>_________________<br>
hawkular-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:hawkular-dev@lists.jboss.org">hawkular-dev@lists.jboss.org</a><br>
<a moz-do-not-send="true"
href="https://lists.jboss.org/mailman/listinfo/hawkular-dev"
rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/hawkular-dev</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
hawkular-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:hawkular-dev@lists.jboss.org">hawkular-dev@lists.jboss.org</a>
<a class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/hawkular-dev">https://lists.jboss.org/mailman/listinfo/hawkular-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>