Re: [Hawkular-dev] HOSA and conversion from prometheus to hawkular metrics

> Are you also tagging the Prometheus metrics with the labels? Yes, that is what was originally being done, and that is still in there. ----- Original Message ----- > > Mazz, this makes sense to me. Our decision to use unique ids (well +type) is > going to lead to this sort of thing. The ids are going to basically be large > concatenations of the tags that identify the data. Then, additionally we're > going to have to tag the metrics with the same name/value pairs that are > present in the id. Are you also tagging the Prometheus metrics with the > labels? > > On 2/1/2017 9:38 AM, John Mazzitelli wrote: > > > > The past several days I've been working on an enhancement to HOSA that came > in from the community (in fact, I would consider it a bug). I'm about ready > to merge the PR [1] for this and do a HOSA 1.1.0.Final release. I wanted to > post this to announce it and see if there is any feedback, too. > > Today, HOSA collects metrics from any Prometheus endpoint which you declare - > example: > > metrics > - name: go_memstats_sys_bytes > - name: process_max_fds > - name: process_open_fds > > But if a Prometheus metric has labels, Prometheus itself considers each > metric with a unique combination of labels as an individual time series > metric. This is different than how Hawkular Metric works - each Hawkular > Metric metric ID (even if its metric definition or its datapoints have tags) > is a single time series metric. We need to account for this difference. For > example, if our agent is configured with: > > metrics: > - name: jvm_memory_pool_bytes_committed > > And the Prometheus endpoint emits that metric with a label called "pool" like > this: > > jvm_memory_pool_bytes_committed{pool="Code Cache",} 2.7787264E7 > jvm_memory_pool_bytes_committed{pool="PS Eden Space",} 2.3068672E7 > > then to Prometheus this is actually 2 time series metrics (the number of > bytes committed per pool type), not 1. Even though the metric name is the > same (what Prometheus calls a "metric family name"), there are two unique > combinations of labels - one with "Code Cache" and one with "PS Eden Space" > - so they are 2 distinct time series metric data. > > Today, the agent only creates a single Hawkular-Metric in this case, with > each datapoint tagged with those Prometheus labels on the appropriate data > point. But we don't want to aggregate them like that since we lose the > granularity that the Prometheus endpoint gives us (that is, the number of > bytes committed in each pool type). I will say I think we might be able to > get that granularity back through datapoint tag queries in Hawkular-Metrics > but I don't know how well (if at all) that is supported and how efficient > such queries would be even if supported, and how efficient storage of these > metrics would be if we tag every data point with these labels (not sure if > that is the general purpose of tags in H-Metrics). But, regardless, the fact > that these really are different time series metrics should (IMO) be > represented as different time series metrics (via metric definitions/metric > IDs) in Hawkular-Metrics. > > To support labeled Prometheus endpoint data like this, the agent needs to > split this one named metric into N Hawkular-Metrics metrics (where N is the > number of unique label combinations for that named metric). So even though > the agent is configured with the one metric > "jvm_memory_pool_bytes_committed" we need to actually create two > Hawkular-Metric metric definitions (with two different and unique metric IDs > obviously). > > The PR [1] that is ready to go does this. By default it will create multiple > metric definitions/metric IDs in the form > "metric-family-name{labelName1=labelValue1,labelName2=labelValue2,...}" > unless you want a different form in which case you can define an "id" and > put in "${labelName}" in the ID you declare (such as > "${oneLabelName}_my_own_metric_name_${theOtherLabelName}" or whatever). But > I suspect the default format will be what most people want and thus nothing > needs to be done. In the above example, two metric definitions with the > following IDs are created: > > 1. jvm_memory_pool_bytes_committed{pool=Code Cache} > 2. jvm_memory_pool_bytes_committed{pool=PS Eden Space} > > --John Mazz > > [1] https://github.com/hawkular/hawkular-openshift-agent/pull/117 > _______________________________________________ > hawkular-dev mailing list hawkular-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev