[Hawkular-dev] HOSA and conversion from prometheus to hawkular metrics

[Joel Takvorian]

+1

Conversion based on labels seems more sane.

I wonder if a new tag that recalls the prometheus metric name would be
useful; ex. "baseName=jvm_memory_pool_bytes_committed", to retrieve all
metrics of that family. Just an idea.

On Wed, Feb 1, 2017 at 4:25 PM, John Mazzitelli <mazz at redhat.com> wrote:

> > Are you also tagging the Prometheus metrics with the labels?
>
> Yes, that is what was originally being done, and that is still in there.
>
> ----- Original Message -----
> >
> > Mazz, this makes sense to me. Our decision to use unique ids (well
> +type) is
> > going to lead to this sort of thing. The ids are going to basically be
> large
> > concatenations of the tags that identify the data. Then, additionally
> we're
> > going to have to tag the metrics with the same name/value pairs that are
> > present in the id. Are you also tagging the Prometheus metrics with the
> > labels?
> >
> > On 2/1/2017 9:38 AM, John Mazzitelli wrote:
> >
> >
> >
> > The past several days I've been working on an enhancement to HOSA that
> came
> > in from the community (in fact, I would consider it a bug). I'm about
> ready
> > to merge the PR [1] for this and do a HOSA 1.1.0.Final release. I wanted
> to
> > post this to announce it and see if there is any feedback, too.
> >
> > Today, HOSA collects metrics from any Prometheus endpoint which you
> declare -
> > example:
> >
> >    metrics
> >    - name: go_memstats_sys_bytes
> >    - name: process_max_fds
> >    - name: process_open_fds
> >
> > But if a Prometheus metric has labels, Prometheus itself considers each
> > metric with a unique combination of labels as an individual time series
> > metric. This is different than how Hawkular Metric works - each Hawkular
> > Metric metric ID (even if its metric definition or its datapoints have
> tags)
> > is a single time series metric. We need to account for this difference.
> For
> > example, if our agent is configured with:
> >
> >    metrics:
> >    - name: jvm_memory_pool_bytes_committed
> >
> > And the Prometheus endpoint emits that metric with a label called "pool"
> like
> > this:
> >
> >    jvm_memory_pool_bytes_committed{pool="Code Cache",} 2.7787264E7
> >    jvm_memory_pool_bytes_committed{pool="PS Eden Space",} 2.3068672E7
> >
> > then to Prometheus this is actually 2 time series metrics (the number of
> > bytes committed per pool type), not 1. Even though the metric name is the
> > same (what Prometheus calls a "metric family name"), there are two unique
> > combinations of labels - one with "Code Cache" and one with "PS Eden
> Space"
> > - so they are 2 distinct time series metric data.
> >
> > Today, the agent only creates a single Hawkular-Metric in this case, with
> > each datapoint tagged with those Prometheus labels on the appropriate
> data
> > point. But we don't want to aggregate them like that since we lose the
> > granularity that the Prometheus endpoint gives us (that is, the number of
> > bytes committed in each pool type). I will say I think we might be able
> to
> > get that granularity back through datapoint tag queries in
> Hawkular-Metrics
> > but I don't know how well (if at all) that is supported and how efficient
> > such queries would be even if supported, and how efficient storage of
> these
> > metrics would be if we tag every data point with these labels (not sure
> if
> > that is the general purpose of tags in H-Metrics). But, regardless, the
> fact
> > that these really are different time series metrics should (IMO) be
> > represented as different time series metrics (via metric
> definitions/metric
> > IDs) in Hawkular-Metrics.
> >
> > To support labeled Prometheus endpoint data like this, the agent needs to
> > split this one named metric into N Hawkular-Metrics metrics (where N is
> the
> > number of unique label combinations for that named metric). So even
> though
> > the agent is configured with the one metric
> > "jvm_memory_pool_bytes_committed" we need to actually create two
> > Hawkular-Metric metric definitions (with two different and unique metric
> IDs
> > obviously).
> >
> > The PR [1] that is ready to go does this. By default it will create
> multiple
> > metric definitions/metric IDs in the form
> > "metric-family-name{labelName1=labelValue1,labelName2=labelValue2,...}"
> > unless you want a different form in which case you can define an "id" and
> > put in "${labelName}" in the ID you declare (such as
> > "${oneLabelName}_my_own_metric_name_${theOtherLabelName}" or whatever).
> But
> > I suspect the default format will be what most people want and thus
> nothing
> > needs to be done. In the above example, two metric definitions with the
> > following IDs are created:
> >
> > 1. jvm_memory_pool_bytes_committed{pool=Code Cache}
> > 2. jvm_memory_pool_bytes_committed{pool=PS Eden Space}
> >
> > --John Mazz
> >
> > [1] https://github.com/hawkular/hawkular-openshift-agent/pull/117
> > _______________________________________________
> > hawkular-dev mailing list hawkular-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/hawkular-dev
> >
> >
> > _______________________________________________
> > hawkular-dev mailing list
> > hawkular-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/hawkular-dev
> >
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170201/0ae2141a/attachment-0001.html