[Hawkular-dev] HOSA and conversion from prometheus to hawkular metrics

Jay Shaughnessy jshaughn at redhat.com
Wed Feb 1 11:40:31 EST 2017 

+1, if that is not being done I think it would good.  Actually, it's 
probably a good "best practice" as it make it easier to slice and dice 
the data.

On 2/1/2017 10:35 AM, Joel Takvorian wrote:
> +1
>
> Conversion based on labels seems more sane.
>
> I wonder if a new tag that recalls the prometheus metric name would be 
> useful; ex. "baseName=jvm_memory_pool_bytes_committed", to retrieve 
> all metrics of that family. Just an idea.
>
> On Wed, Feb 1, 2017 at 4:25 PM, John Mazzitelli <mazz at redhat.com 
> <mailto:mazz at redhat.com>> wrote:
>
>     > Are you also tagging the Prometheus metrics with the labels?
>
>     Yes, that is what was originally being done, and that is still in
>     there.
>
>     ----- Original Message -----
>     >
>     > Mazz, this makes sense to me. Our decision to use unique ids
>     (well +type) is
>     > going to lead to this sort of thing. The ids are going to
>     basically be large
>     > concatenations of the tags that identify the data. Then,
>     additionally we're
>     > going to have to tag the metrics with the same name/value pairs
>     that are
>     > present in the id. Are you also tagging the Prometheus metrics
>     with the
>     > labels?
>     >
>     > On 2/1/2017 9:38 AM, John Mazzitelli wrote:
>     >
>     >
>     >
>     > The past several days I've been working on an enhancement to
>     HOSA that came
>     > in from the community (in fact, I would consider it a bug). I'm
>     about ready
>     > to merge the PR [1] for this and do a HOSA 1.1.0.Final release.
>     I wanted to
>     > post this to announce it and see if there is any feedback, too.
>     >
>     > Today, HOSA collects metrics from any Prometheus endpoint which
>     you declare -
>     > example:
>     >
>     >    metrics
>     >    - name: go_memstats_sys_bytes
>     >    - name: process_max_fds
>     >    - name: process_open_fds
>     >
>     > But if a Prometheus metric has labels, Prometheus itself
>     considers each
>     > metric with a unique combination of labels as an individual time
>     series
>     > metric. This is different than how Hawkular Metric works - each
>     Hawkular
>     > Metric metric ID (even if its metric definition or its
>     datapoints have tags)
>     > is a single time series metric. We need to account for this
>     difference. For
>     > example, if our agent is configured with:
>     >
>     >    metrics:
>     >    - name: jvm_memory_pool_bytes_committed
>     >
>     > And the Prometheus endpoint emits that metric with a label
>     called "pool" like
>     > this:
>     >
>     >    jvm_memory_pool_bytes_committed{pool="Code Cache",} 2.7787264E7
>     >    jvm_memory_pool_bytes_committed{pool="PS Eden Space",}
>     2.3068672E7
>     >
>     > then to Prometheus this is actually 2 time series metrics (the
>     number of
>     > bytes committed per pool type), not 1. Even though the metric
>     name is the
>     > same (what Prometheus calls a "metric family name"), there are
>     two unique
>     > combinations of labels - one with "Code Cache" and one with "PS
>     Eden Space"
>     > - so they are 2 distinct time series metric data.
>     >
>     > Today, the agent only creates a single Hawkular-Metric in this
>     case, with
>     > each datapoint tagged with those Prometheus labels on the
>     appropriate data
>     > point. But we don't want to aggregate them like that since we
>     lose the
>     > granularity that the Prometheus endpoint gives us (that is, the
>     number of
>     > bytes committed in each pool type). I will say I think we might
>     be able to
>     > get that granularity back through datapoint tag queries in
>     Hawkular-Metrics
>     > but I don't know how well (if at all) that is supported and how
>     efficient
>     > such queries would be even if supported, and how efficient
>     storage of these
>     > metrics would be if we tag every data point with these labels
>     (not sure if
>     > that is the general purpose of tags in H-Metrics). But,
>     regardless, the fact
>     > that these really are different time series metrics should (IMO) be
>     > represented as different time series metrics (via metric
>     definitions/metric
>     > IDs) in Hawkular-Metrics.
>     >
>     > To support labeled Prometheus endpoint data like this, the agent
>     needs to
>     > split this one named metric into N Hawkular-Metrics metrics
>     (where N is the
>     > number of unique label combinations for that named metric). So
>     even though
>     > the agent is configured with the one metric
>     > "jvm_memory_pool_bytes_committed" we need to actually create two
>     > Hawkular-Metric metric definitions (with two different and
>     unique metric IDs
>     > obviously).
>     >
>     > The PR [1] that is ready to go does this. By default it will
>     create multiple
>     > metric definitions/metric IDs in the form
>     >
>     "metric-family-name{labelName1=labelValue1,labelName2=labelValue2,...}"
>     > unless you want a different form in which case you can define an
>     "id" and
>     > put in "${labelName}" in the ID you declare (such as
>     > "${oneLabelName}_my_own_metric_name_${theOtherLabelName}" or
>     whatever). But
>     > I suspect the default format will be what most people want and
>     thus nothing
>     > needs to be done. In the above example, two metric definitions
>     with the
>     > following IDs are created:
>     >
>     > 1. jvm_memory_pool_bytes_committed{pool=Code Cache}
>     > 2. jvm_memory_pool_bytes_committed{pool=PS Eden Space}
>     >
>     > --John Mazz
>     >
>     > [1]
>     https://github.com/hawkular/hawkular-openshift-agent/pull/117
>     <https://github.com/hawkular/hawkular-openshift-agent/pull/117>
>     > _______________________________________________
>     > hawkular-dev mailing list hawkular-dev at lists.jboss.org
>     <mailto:hawkular-dev at lists.jboss.org>
>     > https://lists.jboss.org/mailman/listinfo/hawkular-dev
>     <https://lists.jboss.org/mailman/listinfo/hawkular-dev>
>     >
>     >
>     > _______________________________________________
>     > hawkular-dev mailing list
>     > hawkular-dev at lists.jboss.org <mailto:hawkular-dev at lists.jboss.org>
>     > https://lists.jboss.org/mailman/listinfo/hawkular-dev
>     <https://lists.jboss.org/mailman/listinfo/hawkular-dev>
>     >
>     _______________________________________________
>     hawkular-dev mailing list
>     hawkular-dev at lists.jboss.org <mailto:hawkular-dev at lists.jboss.org>
>     https://lists.jboss.org/mailman/listinfo/hawkular-dev
>     <https://lists.jboss.org/mailman/listinfo/hawkular-dev>
>
>
>
>
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170201/8482d884/attachment.html

More information about the hawkular-dev mailing list