On Apr 17, 2017, at 6:10 PM, Benjamin Heiskell <bheiskell@datto.com> wrote:

Hey all,

I recently deployed Hawkular because I found its scaling model appealing. It has been working well for our basic monitoring usage when combined with Grafana.

One of my consumers is trying to graph KPIs that involve cross-metric mathematics like sums, division, and moving averages (i.e., post-ingress transformations). I looked into how to accomplish this with Hawkular, but I didn't see any obvious way to do it. The stats endpoint seems to only support aggregations on a single metric/tag at a time. I'm finding that we can't visualize these metrics the same way you can with other TSDBs like Graphite/Prometheus.

The stats endpoints actually do support querying across multiple metrics; however, docs are really lacking which makes it that much easier to miss. I have some docs updates underway. I will try to get to include sections on stats queries. Let me provide a few examples for now.

GET /hawkular/metrics/gauges/stats?tags=$tagsQuery

where tagsQuery might be like:

'hostname=host1'

'hostname=host1 AND env=stage’

'hostname=host1 OR hostname=host2’

'hostname IN [host1, host2, host3]'

'hostname NOT IN [host4, host5]’

The /gauges/stats endpoints supports all the other parameter that are supported by /gauges/{id}/stats. There is a corresponding endpoint for counters:

GET /hawkular/metrics/counters/stats?tags=$tagsQuery

You can also query for rates:

GET /hawkular/metrics/counters/rates/stats=$tagsQuery

Again, these endpoints all support the same query parameters.

There are POST version as well where the query parameters are instead specified as JSON in the request body.

POST /hawkular/metrics/gauges/stats/query

POST /hawkular/metrics/counters/stats/query

The POST endpoints were added in support of the Grafana data source plugin. See https://issues.jboss.org/browse/HWKMETRICS-410 and https://issues.jboss.org/browse/HWKMETRICS-465 for background on the POST endpoints.

I did a little research, and found Heiko Rupp's blog post on HawkFX where he seems to already be exploring this idea [1]. Perhaps the query language item on the 2017 road map refers to incorporating this?

I would like to assist in developing this feature. Given that there is already a tags DSL, it seems like it would make sense to extend that to support metric transformations.

Grafana is also considering adding this to Grafana itself, but it doesn't look like the ticket has any traction [2]. Additionally, shipping potentially megabytes of data to Grafana (or the user's browser) to aggregate doesn't seem ideal.

I documented a few TSDB DSLs styles here [3].

What do you all think?

Absolutely agree that Hawkular Metrics would benefit greatly from query language. It would compliment the Grafana plugin nicely. Contributions of any kind, mailing list, IRC (#hawkular) discussion, pull requests, etc, are always welcomed and appreciated. I suggest creating a ticket at https://issues.jboss.org/secure/CreateIssue!default.jspa (be sure to select Hawkular Metrics for the project.

Thanks,
Ben

[1] http://pilhuhn.blogspot.com/2016/09/computed-metrics-for-hawkfx.html
[2] https://github.com/grafana/grafana/issues/3677
[3] https://gist.github.com/bheiskell/25af2543ee8588f70895b0311ef291d3

_______________________________________________
hawkular-dev mailing list
hawkular-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev