<div dir="ltr">+1<div><br><div>Conversion based on labels seems more sane.</div><div><br></div><div>I wonder if a new tag that recalls the prometheus metric name would be useful; ex. "baseName=jvm_memory_pool_bytes_committed", to retrieve all metrics of that family. Just an idea. </div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Feb 1, 2017 at 4:25 PM, John Mazzitelli <span dir="ltr"><<a href="mailto:mazz@redhat.com" target="_blank">mazz@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">> Are you also tagging the Prometheus metrics with the labels?<br>
<br>
</span>Yes, that is what was originally being done, and that is still in there.<br>
<div class="HOEnZb"><div class="h5"><br>
----- Original Message -----<br>
><br>
> Mazz, this makes sense to me. Our decision to use unique ids (well +type) is<br>
> going to lead to this sort of thing. The ids are going to basically be large<br>
> concatenations of the tags that identify the data. Then, additionally we're<br>
> going to have to tag the metrics with the same name/value pairs that are<br>
> present in the id. Are you also tagging the Prometheus metrics with the<br>
> labels?<br>
><br>
> On 2/1/2017 9:38 AM, John Mazzitelli wrote:<br>
><br>
><br>
><br>
> The past several days I've been working on an enhancement to HOSA that came<br>
> in from the community (in fact, I would consider it a bug). I'm about ready<br>
> to merge the PR [1] for this and do a HOSA 1.1.0.Final release. I wanted to<br>
> post this to announce it and see if there is any feedback, too.<br>
><br>
> Today, HOSA collects metrics from any Prometheus endpoint which you declare -<br>
> example:<br>
><br>
> metrics<br>
> - name: go_memstats_sys_bytes<br>
> - name: process_max_fds<br>
> - name: process_open_fds<br>
><br>
> But if a Prometheus metric has labels, Prometheus itself considers each<br>
> metric with a unique combination of labels as an individual time series<br>
> metric. This is different than how Hawkular Metric works - each Hawkular<br>
> Metric metric ID (even if its metric definition or its datapoints have tags)<br>
> is a single time series metric. We need to account for this difference. For<br>
> example, if our agent is configured with:<br>
><br>
> metrics:<br>
> - name: jvm_memory_pool_bytes_<wbr>committed<br>
><br>
> And the Prometheus endpoint emits that metric with a label called "pool" like<br>
> this:<br>
><br>
> jvm_memory_pool_bytes_<wbr>committed{pool="Code Cache",} 2.7787264E7<br>
> jvm_memory_pool_bytes_<wbr>committed{pool="PS Eden Space",} 2.3068672E7<br>
><br>
> then to Prometheus this is actually 2 time series metrics (the number of<br>
> bytes committed per pool type), not 1. Even though the metric name is the<br>
> same (what Prometheus calls a "metric family name"), there are two unique<br>
> combinations of labels - one with "Code Cache" and one with "PS Eden Space"<br>
> - so they are 2 distinct time series metric data.<br>
><br>
> Today, the agent only creates a single Hawkular-Metric in this case, with<br>
> each datapoint tagged with those Prometheus labels on the appropriate data<br>
> point. But we don't want to aggregate them like that since we lose the<br>
> granularity that the Prometheus endpoint gives us (that is, the number of<br>
> bytes committed in each pool type). I will say I think we might be able to<br>
> get that granularity back through datapoint tag queries in Hawkular-Metrics<br>
> but I don't know how well (if at all) that is supported and how efficient<br>
> such queries would be even if supported, and how efficient storage of these<br>
> metrics would be if we tag every data point with these labels (not sure if<br>
> that is the general purpose of tags in H-Metrics). But, regardless, the fact<br>
> that these really are different time series metrics should (IMO) be<br>
> represented as different time series metrics (via metric definitions/metric<br>
> IDs) in Hawkular-Metrics.<br>
><br>
> To support labeled Prometheus endpoint data like this, the agent needs to<br>
> split this one named metric into N Hawkular-Metrics metrics (where N is the<br>
> number of unique label combinations for that named metric). So even though<br>
> the agent is configured with the one metric<br>
> "jvm_memory_pool_bytes_<wbr>committed" we need to actually create two<br>
> Hawkular-Metric metric definitions (with two different and unique metric IDs<br>
> obviously).<br>
><br>
> The PR [1] that is ready to go does this. By default it will create multiple<br>
> metric definitions/metric IDs in the form<br>
> "metric-family-name{<wbr>labelName1=labelValue1,<wbr>labelName2=labelValue2,...}"<br>
> unless you want a different form in which case you can define an "id" and<br>
> put in "${labelName}" in the ID you declare (such as<br>
> "${oneLabelName}_my_own_<wbr>metric_name_${<wbr>theOtherLabelName}" or whatever). But<br>
> I suspect the default format will be what most people want and thus nothing<br>
> needs to be done. In the above example, two metric definitions with the<br>
> following IDs are created:<br>
><br>
> 1. jvm_memory_pool_bytes_<wbr>committed{pool=Code Cache}<br>
> 2. jvm_memory_pool_bytes_<wbr>committed{pool=PS Eden Space}<br>
><br>
> --John Mazz<br>
><br>
> [1] <a href="https://github.com/hawkular/hawkular-openshift-agent/pull/117" rel="noreferrer" target="_blank">https://github.com/hawkular/<wbr>hawkular-openshift-agent/pull/<wbr>117</a><br>
> ______________________________<wbr>_________________<br>
> hawkular-dev mailing list <a href="mailto:hawkular-dev@lists.jboss.org">hawkular-dev@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/hawkular-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/hawkular-dev</a><br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> hawkular-dev mailing list<br>
> <a href="mailto:hawkular-dev@lists.jboss.org">hawkular-dev@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/hawkular-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/hawkular-dev</a><br>
><br>
______________________________<wbr>_________________<br>
hawkular-dev mailing list<br>
<a href="mailto:hawkular-dev@lists.jboss.org">hawkular-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/hawkular-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/hawkular-dev</a><br>
</div></div></blockquote></div><br></div>