I did some hacking (day 1 of Rx, so probably not the best "solution"). But works... - https://github.com/garethahealy/hawkular-metrics/commit/75c616f9be71a0b85... Sample output: - https://gist.github.com/garethahealy/00a90bbee2556b6f0a338ece87096c89 And some test cURL commands: - https://gist.github.com/garethahealy/0f46aad5d2da41b82aad5af317fad788 Cheers. On Thu, Dec 29, 2016 at 4:33 PM, John Mazzitelli <mazz(a)redhat.com&gt; wrote:

Have raised: - https://issues.jboss.org/browse/HWKMETRICS-563 Will work on the grafana datasource support in next few days. Cheers. On Fri, Dec 30, 2016 at 6:29 PM, John Mazzitelli <mazz(a)redhat.com&gt; wrote:

Gareth, I replied in the ticket. Please take a look at https://issues.jboss.org/browse/HWKMETRICS-373 <https://issues.jboss.org/browse/HWKMETRICS-373> to see if that provides the functionality you need.

----- Original Message -----

Ah, OK. That's different than what I was talking about. The Hawkular OpenShift Agent allows you to define any number of tags to apply to a metric definition. This is the "Tags" entry in your configmap or the global configs defined in the global agent config... like this: collector: tags: pod_name: {$POD:name} ... or: endpoints: - type: jolokia tags: foo: bar ... Those tags are defined on the metric definition alone. The code you pointed out: if you notice, the tags added to the data points there are ONLY the actual Prometheus labels that were found on the Prometheus metric itself - the agent just copies those Prometheus labels to the h-metrics data points as tags (i.e. those tags are added to each data point). The tags I talk about above (the Hawkular tags) are not added to those data points. But perhaps that is all you care about, which would be good :-) and you can probably use what John Sanda talked about. ----- Original Message ----- > Hi John, > > Am not sure what you mean by "*OpenShift Agent does NOT tag individual data > points*", since this tags datapoint's: > > - > > On Tue, Jan 3, 2017 at 6:38 PM, John Mazzitelli <mazz(a)redhat.com&gt; wrote: > > > >From reading that JIRA, it seems like that Hawkular-Metrics feature only > > works if you have tags that are on individual data points (as opposed to > > the metric definition itself). Not sure if I am mistaken or not, but just > > wanted to point this out because, for the record, the Hawkular OpenShift > > Agent does NOT tag individual data points. It only inserts tags on the > > metric definition. Seems inefficient to tag individual data points with the > > exact same tags over and over again - hence we put the tags on metric > > definitions. > > > > > > ----- Original Message ----- > > > @John S, > > > From looking at the JIRA, i've tried calling /stats and /data ( > > deprecated ) > > > but don't get back data i'd want. > > > > > > If i search via tags; i get all empty buckets (both metric and datapoint > > > tags). If i search via metricNames; i get back buckets but since its > > > aggregated i am back to the original problem. > > > > > > Am i querying via tags incorrectly? cURL commands i've been using: > > > > > > > > > * create data - > > > https://gist.github.com/garethahealy/0f46aad5d2da41b82aad5af317fad7 > > 88 > > > * get buckets - > > > https://gist.github.com/garethahealy/3a5913b5a6956d709366ab6985fefe > > c6 > > > > > > @Joel, > > > I'd not done the other endpoints as i wanted to prove it first, and then > > if > > > accepted, complete the rest. > > > > > > Your point about insertion; I'll let others comment who know the metrics > > > codebase better than myself. > > > > > > Cheers. > > > > > > On Tue, Jan 3, 2017 at 4:15 PM, John Sanda < jsanda(a)redhat.com > wrote: > > > > > > > > > > > > Gareth, I replied in the ticket. Please take a look at > > > https://issues.jboss.org/browse/HWKMETRICS-373 to see if that provides > > the > > > functionality you need. > > > > > > > > > > > > > > > On Jan 1, 2017, at 6:15 PM, Gareth Healy < garethahealy(a)gmail.com > > > wrote: > > > > > > Have raised: > > > > > > > > > * https://issues.jboss.org/browse/HWKMETRICS-563 > > > > > > Will work on the grafana datasource support in next few days. > > > > > > Cheers. > > > > > > On Fri, Dec 30, 2016 at 6:29 PM, John Mazzitelli < mazz(a)redhat.com > > > wrote: > > > > > > > > > Thanks. > > > > > > You should submit a feature request in the H-Metrics JIRA and link that > > JIRA > > > to a PR for the H-Metrics team to peer review: > > > > > > https://issues.jboss.org/projects/HWKMETRICS > > > > > > > > > ----- Original Message ----- > > > > I did some hacking (day 1 of Rx, so probably not the best "solution"). > > But > > > > works... > > > > > > > > - > > > > https://github.com/garethahealy/hawkular-metrics/commit/ > > 75c616f9be71a0b85ce5dee310c4dff828bb8f38 > > > > > > > > > > > > Sample output: > > > > > > > > - https://gist.github.com/garethahealy/00a90bbee2556b6f0a338ece87096c > > 89 > > > > > > > > > > > > And some test cURL commands: > > > > > > > > - https://gist.github.com/garethahealy/0f46aad5d2da41b82aad5af317fad7 > > 88 > > > > > > > > > > > > Cheers. > > > > > > > > On Thu, Dec 29, 2016 at 4:33 PM, John Mazzitelli < mazz(a)redhat.com > > > wrote: > > > > > > > > > This would be a feature request on Hawkular-Metrics (if they don't do > > > > > something like this already - I do not know. I will defer to the > > > > > H-Metrics > > > > > folks to talk about how they do querying off of tags). > > > > > > > > > > ----- Original Message ----- > > > > > > The OpenShift Agent when monitoring a prometheus endpoint creates a > > > > > single > > > > > > metric with tagged datapoints, i.e.: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/coreos/etcd/blob/master/ > > Documentation/v2/metrics.md# > > > > > http-requests > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I1228 21:02:01.820530 1 metrics_storage.go:155] TRACE: Stored [3] > > > > > [counter] > > > > > > datapoints for metric named > > > > > > [pod/fa32a887-cd08-11e6-ab2e-525400c583ad/custom/etcd_http_ > > received_total]: > > > > > [ > > > > > > {2016-12-28 21:02:01.638767339 +0000 UTC 622 map[method:DELETE]} > > > > > > {2016-12-28 21:02:01.638767339 +0000 UTC 414756 map[method:GET]} > > > > > > {2016-12-28 21:02:01.638767339 +0000 UTC 33647 map[method:PUT]} > > > > > > ] > > > > > > > > > > > > But when trying to view this via the grafana datasource, only 1 > > metric > > > > > and > > > > > > the aggregated counts are shown. What i'd like to do is something > > like > > > > > the > > > > > > below: > > > > > > > > > > > > > > > > > > > > > > > > { > > > > > > "start": 1482999755690, > > > > > > "end": 1483000020093, > > > > > > "order": "ASC", > > > > > > "tags": "pod_namespace:etcd-testing", > > > > > > "groupDatapointsByTagKey": "method" > > > > > > } > > > > > > > > > > > > Search via tags or name (as-is) and group the datapoints by a tag > > key, > > > > > which > > > > > > would give you 3 lines, instead of 1. > > > > > > > > > > > > Does that sound possible? > > > > > > > > > > > > Cheers. > > > > > > > > > > > > _______________________________________________ > > > > > > hawkular-dev mailing list > > > > > > hawkular-dev(a)lists.jboss.org > > > > > > https://lists.jboss.org/mailman/listinfo/hawkular-dev > > > > > > > > > > > _______________________________________________ > > > > > hawkular-dev mailing list > > > > > hawkular-dev(a)lists.jboss.org > > > > > https://lists.jboss.org/mailman/listinfo/hawkular-dev > > > > > > > > > > > > > > > _______________________________________________ > > > hawkular-dev mailing list > > > hawkular-dev(a)lists.jboss.org > > > https://lists.jboss.org/mailman/listinfo/hawkular-dev > > > > > > > > > _______________________________________________ > > > hawkular-dev mailing list > > > hawkular-dev(a)lists.jboss.org > > > https://lists.jboss.org/mailman/listinfo/hawkular-dev > > > > > > > > > > > > _______________________________________________ > > > hawkular-dev mailing list > > > hawkular-dev(a)lists.jboss.org > > > https://lists.jboss.org/mailman/listinfo/hawkular-dev > > > > > _______________________________________________ > > hawkular-dev mailing list > > hawkular-dev(a)lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/hawkular-dev > > >

:-) And that, I think, is what I meant by tags on "metric definitions" - I think the H-Metrics team calls them "timeseries tags".

Any thoughts on the below? i.e.: should the agent create individual metrics or should the endpoints be able to group. Without this, it makes prometheus gauge metrics pretty useless. Cheers. On Tue, Jan 10, 2017 at 1:26 PM, Gareth Healy <garethahealy(a)gmail.com&gt; wrote:

Note that when HOSA stores the data point in H-Metrics, it does add the prometheus labels to that datapoint as H-Metric tags. So H-Metrics does have each datapoint properly tagged. I haven't followed H-Metrics close enough to know if they plan on addressing how to query that kind of tagged data to make it useful. On Thu, 2017-01-19 at 14:44 -0500, John Mazzitelli wrote:

On Thu, 2017-01-19 at 15:21 -0500, John Sanda wrote: I do not know... that would be a question for Gareth to answer.

You are using the wrong endpoint. It should be: $HAWKULAR_URL/gauges/{metric_id}/stats/tags/type/type:routes?start=1484320827578&end=1484329973940 where {metric_id} is pod%2Fded071be-d9a6-11e6-8140-525400c583ad%2Fcustom%2Fprometheus_MyCamel_MeanProcessingTime And for this endpoint the bucketDuration parameter is not supported. The buckets are determined by the tag filters. The endpoint URL has changed from the initial implementation (and needs to be documented) but the examples in the ticket of how the buckets are determined are still valid.

Hi, Your syntax is still incorrect. It's either metricId or tags, not both (at least not in this case). You should call /gauges/stats?tags=name:*&start=.. https://github.com/hawkular/hawkular-metrics/blob/master/api/metrics-api-... If you wish to use id/stats, then you can't have tags at all. /gauges/$METRIC_ID/stats?start=.. https://github.com/hawkular/hawkular-metrics/blob/master/api/metrics-api-... - Micke On 01/20/2017 11:21 AM, Gareth Healy wrote: