<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
<font face="Calibri">+1, if that is not being done I think it would
good. Actually, it's probably a good "best practice" as it make
it easier to slice and dice the data.</font><br>
<br>
<div class="moz-cite-prefix">On 2/1/2017 10:35 AM, Joel Takvorian
wrote:<br>
</div>
<blockquote
cite="mid:CAJo5TFku2jXdUR2wPOvKjeUotsdAw64V02hYGMNoyoZGm7Ad9g@mail.gmail.com"
type="cite">
<div dir="ltr">+1
<div><br>
<div>Conversion based on labels seems more sane.</div>
<div><br>
</div>
<div>I wonder if a new tag that recalls the prometheus metric
name would be useful; ex.
"baseName=jvm_memory_pool_bytes_committed", to retrieve all
metrics of that family. Just an idea. </div>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, Feb 1, 2017 at 4:25 PM, John
Mazzitelli <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:mazz@redhat.com" target="_blank">mazz@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex"><span
class="">> Are you also tagging the Prometheus metrics
with the labels?<br>
<br>
</span>Yes, that is what was originally being done, and that
is still in there.<br>
<div class="HOEnZb">
<div class="h5"><br>
----- Original Message -----<br>
><br>
> Mazz, this makes sense to me. Our decision to use
unique ids (well +type) is<br>
> going to lead to this sort of thing. The ids are
going to basically be large<br>
> concatenations of the tags that identify the data.
Then, additionally we're<br>
> going to have to tag the metrics with the same
name/value pairs that are<br>
> present in the id. Are you also tagging the
Prometheus metrics with the<br>
> labels?<br>
><br>
> On 2/1/2017 9:38 AM, John Mazzitelli wrote:<br>
><br>
><br>
><br>
> The past several days I've been working on an
enhancement to HOSA that came<br>
> in from the community (in fact, I would consider it
a bug). I'm about ready<br>
> to merge the PR [1] for this and do a HOSA
1.1.0.Final release. I wanted to<br>
> post this to announce it and see if there is any
feedback, too.<br>
><br>
> Today, HOSA collects metrics from any Prometheus
endpoint which you declare -<br>
> example:<br>
><br>
> metrics<br>
> - name: go_memstats_sys_bytes<br>
> - name: process_max_fds<br>
> - name: process_open_fds<br>
><br>
> But if a Prometheus metric has labels, Prometheus
itself considers each<br>
> metric with a unique combination of labels as an
individual time series<br>
> metric. This is different than how Hawkular Metric
works - each Hawkular<br>
> Metric metric ID (even if its metric definition or
its datapoints have tags)<br>
> is a single time series metric. We need to account
for this difference. For<br>
> example, if our agent is configured with:<br>
><br>
> metrics:<br>
> - name: jvm_memory_pool_bytes_<wbr>committed<br>
><br>
> And the Prometheus endpoint emits that metric with
a label called "pool" like<br>
> this:<br>
><br>
> jvm_memory_pool_bytes_<wbr>committed{pool="Code
Cache",} 2.7787264E7<br>
> jvm_memory_pool_bytes_<wbr>committed{pool="PS
Eden Space",} 2.3068672E7<br>
><br>
> then to Prometheus this is actually 2 time series
metrics (the number of<br>
> bytes committed per pool type), not 1. Even though
the metric name is the<br>
> same (what Prometheus calls a "metric family
name"), there are two unique<br>
> combinations of labels - one with "Code Cache" and
one with "PS Eden Space"<br>
> - so they are 2 distinct time series metric data.<br>
><br>
> Today, the agent only creates a single
Hawkular-Metric in this case, with<br>
> each datapoint tagged with those Prometheus labels
on the appropriate data<br>
> point. But we don't want to aggregate them like
that since we lose the<br>
> granularity that the Prometheus endpoint gives us
(that is, the number of<br>
> bytes committed in each pool type). I will say I
think we might be able to<br>
> get that granularity back through datapoint tag
queries in Hawkular-Metrics<br>
> but I don't know how well (if at all) that is
supported and how efficient<br>
> such queries would be even if supported, and how
efficient storage of these<br>
> metrics would be if we tag every data point with
these labels (not sure if<br>
> that is the general purpose of tags in H-Metrics).
But, regardless, the fact<br>
> that these really are different time series metrics
should (IMO) be<br>
> represented as different time series metrics (via
metric definitions/metric<br>
> IDs) in Hawkular-Metrics.<br>
><br>
> To support labeled Prometheus endpoint data like
this, the agent needs to<br>
> split this one named metric into N Hawkular-Metrics
metrics (where N is the<br>
> number of unique label combinations for that named
metric). So even though<br>
> the agent is configured with the one metric<br>
> "jvm_memory_pool_bytes_<wbr>committed" we need to
actually create two<br>
> Hawkular-Metric metric definitions (with two
different and unique metric IDs<br>
> obviously).<br>
><br>
> The PR [1] that is ready to go does this. By
default it will create multiple<br>
> metric definitions/metric IDs in the form<br>
> "metric-family-name{<wbr>labelName1=labelValue1,<wbr>labelName2=labelValue2,...}"<br>
> unless you want a different form in which case you
can define an "id" and<br>
> put in "${labelName}" in the ID you declare (such
as<br>
> "${oneLabelName}_my_own_<wbr>metric_name_${<wbr>theOtherLabelName}"
or whatever). But<br>
> I suspect the default format will be what most
people want and thus nothing<br>
> needs to be done. In the above example, two metric
definitions with the<br>
> following IDs are created:<br>
><br>
> 1. jvm_memory_pool_bytes_<wbr>committed{pool=Code
Cache}<br>
> 2. jvm_memory_pool_bytes_<wbr>committed{pool=PS
Eden Space}<br>
><br>
> --John Mazz<br>
><br>
> [1] <a moz-do-not-send="true"
href="https://github.com/hawkular/hawkular-openshift-agent/pull/117"
rel="noreferrer" target="_blank">https://github.com/hawkular/<wbr>hawkular-openshift-agent/pull/<wbr>117</a><br>
> ______________________________<wbr>_________________<br>
> hawkular-dev mailing list <a
moz-do-not-send="true"
href="mailto:hawkular-dev@lists.jboss.org">hawkular-dev@lists.jboss.org</a><br>
> <a moz-do-not-send="true"
href="https://lists.jboss.org/mailman/listinfo/hawkular-dev"
rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/hawkular-dev</a><br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> hawkular-dev mailing list<br>
> <a moz-do-not-send="true"
href="mailto:hawkular-dev@lists.jboss.org">hawkular-dev@lists.jboss.org</a><br>
> <a moz-do-not-send="true"
href="https://lists.jboss.org/mailman/listinfo/hawkular-dev"
rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/hawkular-dev</a><br>
><br>
______________________________<wbr>_________________<br>
hawkular-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:hawkular-dev@lists.jboss.org">hawkular-dev@lists.jboss.org</a><br>
<a moz-do-not-send="true"
href="https://lists.jboss.org/mailman/listinfo/hawkular-dev"
rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/hawkular-dev</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
hawkular-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:hawkular-dev@lists.jboss.org">hawkular-dev@lists.jboss.org</a>
<a class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/hawkular-dev">https://lists.jboss.org/mailman/listinfo/hawkular-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>