[Hawkular-dev] Ability to group by datapoint tag in Grafana

John Mazzitelli mazz at redhat.com
Tue Jan 3 17:21:43 EST 2017 

Ah, OK. That's different than what I was talking about.

The Hawkular OpenShift Agent allows you to define any number of tags to apply to a metric definition. This is the "Tags" entry in your configmap or the global configs defined in the global agent config... like this:

collector:
  tags:
    pod_name: {$POD:name}
    ...

or:

endpoints:
- type: jolokia
  tags:
    foo: bar
  ...

Those tags are defined on the metric definition alone. The code you pointed out:

> https://github.com/hawkular/hawkular-openshift-agent/blob/master/collector/impl/prometheus_metrics_collector.go#L192

if you notice, the tags added to the data points there are ONLY the actual Prometheus labels that were found on the Prometheus metric itself - the agent just copies those Prometheus labels to the h-metrics data points as tags (i.e. those tags are added to each data point). The tags I talk about above (the Hawkular tags) are not added to those data points.

But perhaps that is all you care about, which would be good :-) and you can probably use what John Sanda talked about.

----- Original Message -----
> Hi John,
> 
> Am not sure what you mean by "*OpenShift Agent does NOT tag individual data
> points*", since this tags datapoint's:
> 
> -
> https://github.com/hawkular/hawkular-openshift-agent/blob/master/collector/impl/prometheus_metrics_collector.go#L192
> 
> On Tue, Jan 3, 2017 at 6:38 PM, John Mazzitelli <mazz at redhat.com> wrote:
> 
> > >From reading that JIRA, it seems like that Hawkular-Metrics feature only
> > works if you have tags that are on individual data points (as opposed to
> > the metric definition itself). Not sure if I am mistaken or not, but just
> > wanted to point this out because, for the record, the Hawkular OpenShift
> > Agent does NOT tag individual data points. It only inserts tags on the
> > metric definition. Seems inefficient to tag individual data points with the
> > exact same tags over and over again - hence we put the tags on metric
> > definitions.
> >
> >
> > ----- Original Message -----
> > > @John S,
> > > From looking at the JIRA, i've tried calling /stats and /data (
> > deprecated )
> > > but don't get back data i'd want.
> > >
> > > If i search via tags; i get all empty buckets (both metric and datapoint
> > > tags). If i search via metricNames; i get back buckets but since its
> > > aggregated i am back to the original problem.
> > >
> > > Am i querying via tags incorrectly? cURL commands i've been using:
> > >
> > >
> > >     * create data -
> > >     https://gist.github.com/garethahealy/0f46aad5d2da41b82aad5af317fad7
> > 88
> > >     * get buckets -
> > >     https://gist.github.com/garethahealy/3a5913b5a6956d709366ab6985fefe
> > c6
> > >
> > > @Joel,
> > > I'd not done the other endpoints as i wanted to prove it first, and then
> > if
> > > accepted, complete the rest.
> > >
> > > Your point about insertion; I'll let others comment who know the metrics
> > > codebase better than myself.
> > >
> > > Cheers.
> > >
> > > On Tue, Jan 3, 2017 at 4:15 PM, John Sanda < jsanda at redhat.com > wrote:
> > >
> > >
> > >
> > > Gareth, I replied in the ticket. Please take a look at
> > > https://issues.jboss.org/browse/HWKMETRICS-373 to see if that provides
> > the
> > > functionality you need.
> > >
> > >
> > >
> > >
> > > On Jan 1, 2017, at 6:15 PM, Gareth Healy < garethahealy at gmail.com >
> > wrote:
> > >
> > > Have raised:
> > >
> > >
> > >     * https://issues.jboss.org/browse/HWKMETRICS-563
> > >
> > > Will work on the grafana datasource support in next few days.
> > >
> > > Cheers.
> > >
> > > On Fri, Dec 30, 2016 at 6:29 PM, John Mazzitelli < mazz at redhat.com >
> > wrote:
> > >
> > >
> > > Thanks.
> > >
> > > You should submit a feature request in the H-Metrics JIRA and link that
> > JIRA
> > > to a PR for the H-Metrics team to peer review:
> > >
> > > https://issues.jboss.org/projects/HWKMETRICS
> > >
> > >
> > > ----- Original Message -----
> > > > I did some hacking (day 1 of Rx, so probably not the best "solution").
> > But
> > > > works...
> > > >
> > > > -
> > > > https://github.com/garethahealy/hawkular-metrics/commit/
> > 75c616f9be71a0b85ce5dee310c4dff828bb8f38
> > > >
> > > >
> > > > Sample output:
> > > >
> > > > - https://gist.github.com/garethahealy/00a90bbee2556b6f0a338ece87096c
> > 89
> > > >
> > > >
> > > > And some test cURL commands:
> > > >
> > > > - https://gist.github.com/garethahealy/0f46aad5d2da41b82aad5af317fad7
> > 88
> > > >
> > > >
> > > > Cheers.
> > > >
> > > > On Thu, Dec 29, 2016 at 4:33 PM, John Mazzitelli < mazz at redhat.com >
> > wrote:
> > > >
> > > > > This would be a feature request on Hawkular-Metrics (if they don't do
> > > > > something like this already - I do not know. I will defer to the
> > > > > H-Metrics
> > > > > folks to talk about how they do querying off of tags).
> > > > >
> > > > > ----- Original Message -----
> > > > > > The OpenShift Agent when monitoring a prometheus endpoint creates a
> > > > > single
> > > > > > metric with tagged datapoints, i.e.:
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > https://github.com/coreos/etcd/blob/master/
> > Documentation/v2/metrics.md#
> > > > > http-requests
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > I1228 21:02:01.820530 1 metrics_storage.go:155] TRACE: Stored [3]
> > > > > [counter]
> > > > > > datapoints for metric named
> > > > > > [pod/fa32a887-cd08-11e6-ab2e-525400c583ad/custom/etcd_http_
> > received_total]:
> > > > > [
> > > > > > {2016-12-28 21:02:01.638767339 +0000 UTC 622 map[method:DELETE]}
> > > > > > {2016-12-28 21:02:01.638767339 +0000 UTC 414756 map[method:GET]}
> > > > > > {2016-12-28 21:02:01.638767339 +0000 UTC 33647 map[method:PUT]}
> > > > > > ]
> > > > > >
> > > > > > But when trying to view this via the grafana datasource, only 1
> > metric
> > > > > and
> > > > > > the aggregated counts are shown. What i'd like to do is something
> > like
> > > > > the
> > > > > > below:
> > > > > >
> > > > > >
> > > > > >
> > > > > > {
> > > > > > "start": 1482999755690,
> > > > > > "end": 1483000020093,
> > > > > > "order": "ASC",
> > > > > > "tags": "pod_namespace:etcd-testing",
> > > > > > "groupDatapointsByTagKey": "method"
> > > > > > }
> > > > > >
> > > > > > Search via tags or name (as-is) and group the datapoints by a tag
> > key,
> > > > > which
> > > > > > would give you 3 lines, instead of 1.
> > > > > >
> > > > > > Does that sound possible?
> > > > > >
> > > > > > Cheers.
> > > > > >
> > > > > > _______________________________________________
> > > > > > hawkular-dev mailing list
> > > > > > hawkular-dev at lists.jboss.org
> > > > > > https://lists.jboss.org/mailman/listinfo/hawkular-dev
> > > > > >
> > > > > _______________________________________________
> > > > > hawkular-dev mailing list
> > > > > hawkular-dev at lists.jboss.org
> > > > > https://lists.jboss.org/mailman/listinfo/hawkular-dev
> > > > >
> > > >
> > >
> > > _______________________________________________
> > > hawkular-dev mailing list
> > > hawkular-dev at lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/hawkular-dev
> > >
> > >
> > > _______________________________________________
> > > hawkular-dev mailing list
> > > hawkular-dev at lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/hawkular-dev
> > >
> > >
> > >
> > > _______________________________________________
> > > hawkular-dev mailing list
> > > hawkular-dev at lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/hawkular-dev
> > >
> > _______________________________________________
> > hawkular-dev mailing list
> > hawkular-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/hawkular-dev
> >
>

More information about the hawkular-dev mailing list