5:10 p.m.
Hey all,
I recently deployed Hawkular because I found its scaling model appealing.
It has been working well for our basic monitoring usage when combined with
Grafana.
One of my consumers is trying to graph KPIs that involve cross-metric
mathematics like sums, division, and moving averages (i.e., post-ingress
transformations). I looked into how to accomplish this with Hawkular, but I
didn't see any obvious way to do it. The stats endpoint seems to only
support aggregations on a single metric/tag at a time. I'm finding that we
can't visualize these metrics the same way you can with other TSDBs like
Graphite/Prometheus.
I did a little research, and found Heiko Rupp's blog post on HawkFX where
he seems to already be exploring this idea [1]. Perhaps the query language
item on the 2017 road map refers to incorporating this?
I would like to assist in developing this feature. Given that there is
already a tags DSL, it seems like it would make sense to extend that to
support metric transformations.
Grafana is also considering adding this to Grafana itself, but it doesn't
look like the ticket has any traction [2]. Additionally, shipping
potentially megabytes of data to Grafana (or the user's browser) to
aggregate doesn't seem ideal.
I documented a few TSDB DSLs styles here [3].
What do you all think?
Thanks,
Ben
[1] http://pilhuhn.blogspot.com/2016/09/computed-metrics-for-hawkfx.html
[2] https://github.com/grafana/grafana/issues/3677
[3] https://gist.github.com/bheiskell/25af2543ee8588f70895b0311ef291d3
6:10 p.m.
Really great idea!
On Apr 18, 2017 1:05 AM, "Benjamin Heiskell" <bheiskell(a)datto.com> wrote:
Hey all,
I recently deployed Hawkular because I found its scaling model appealing.
It has been working well for our basic monitoring usage when combined with
Grafana.
One of my consumers is trying to graph KPIs that involve cross-metric
mathematics like sums, division, and moving averages (i.e., post-ingress
transformations). I looked into how to accomplish this with Hawkular, but I
didn't see any obvious way to do it. The stats endpoint seems to only
support aggregations on a single metric/tag at a time. I'm finding that we
can't visualize these metrics the same way you can with other TSDBs like
Graphite/Prometheus.
I did a little research, and found Heiko Rupp's blog post on HawkFX where
he seems to already be exploring this idea [1]. Perhaps the query language
item on the 2017 road map refers to incorporating this?
I would like to assist in developing this feature. Given that there is
already a tags DSL, it seems like it would make sense to extend that to
support metric transformations.
Grafana is also considering adding this to Grafana itself, but it doesn't
look like the ticket has any traction [2]. Additionally, shipping
potentially megabytes of data to Grafana (or the user's browser) to
aggregate doesn't seem ideal.
I documented a few TSDB DSLs styles here [3].
What do you all think?
Thanks,
Ben
[1] http://pilhuhn.blogspot.com/2016/09/computed-metrics-for-hawkfx.html
[2] https://github.com/grafana/grafana/issues/3677
[3] https://gist.github.com/bheiskell/25af2543ee8588f70895b0311ef291d3
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
7:01 p.m.
On 4/17/2017 6:10 PM, Benjamin Heiskell wrote:
Hey all,
I recently deployed Hawkular because I found its scaling model
appealing. It has been working well for our basic monitoring usage
when combined with Grafana.
One of my consumers is trying to graph KPIs that involve cross-metric
mathematics like sums, division, and moving averages (i.e.,
post-ingress transformations). I looked into how to accomplish this
with Hawkular, but I didn't see any obvious way to do it. The stats
endpoint seems to only support aggregations on a single metric/tag at
a time. I'm finding that we can't visualize these metrics the same way
you can with other TSDBs like Graphite/Prometheus.
It's maybe not what you're looking for but the model for aggregating
across metrics is to use a tag query to define the set of applicable
metrics. The tag query expressions are fairly robust [1], of course the
tags need to be there to make it useful. But given the tags it is
possible to do sums, percentiles, etc... As an aside, there is small
expression language for using those aggregates for alerting. If you're
interested check out
http://www.hawkular.org/hawkular-metrics/docs/user-guide/#_query_based_al....
[1] The tag query language has just recently been released and I'm not
sure if the documentation has been updated (Snegrea?). But you can look
here https://github.com/hawkular/hawkular-metrics/pull/725.
I did a little research, and found Heiko Rupp's blog post on HawkFX
where he seems to already be exploring this idea [1]. Perhaps the
query language item on the 2017 road map refers to incorporating this?
I would like to assist in developing this feature. Given that there is
already a tags DSL, it seems like it would make sense to extend that
to support metric transformations.
Grafana is also considering adding this to Grafana itself, but it
doesn't look like the ticket has any traction [2]. Additionally,
shipping potentially megabytes of data to Grafana (or the user's
browser) to aggregate doesn't seem ideal.
I documented a few TSDB DSLs styles here [3].
What do you all think?
Thanks,
Ben
[1] http://pilhuhn.blogspot.com/2016/09/computed-metrics-for-hawkfx.html
[2] https://github.com/grafana/grafana/issues/3677
[3] https://gist.github.com/bheiskell/25af2543ee8588f70895b0311ef291d3
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
9:33 p.m.
Hi Ben,
Comments inline...
On Apr 17, 2017, at 6:10 PM, Benjamin Heiskell
<bheiskell(a)datto.com> wrote:
Hey all,
I recently deployed Hawkular because I found its scaling model appealing. It has been
working well for our basic monitoring usage when combined with Grafana.
One of my consumers is trying to graph KPIs that involve cross-metric mathematics like
sums, division, and moving averages (i.e., post-ingress transformations). I looked into
how to accomplish this with Hawkular, but I didn't see any obvious way to do it. The
stats endpoint seems to only support aggregations on a single metric/tag at a time.
I'm finding that we can't visualize these metrics the same way you can with other
TSDBs like Graphite/Prometheus.
The stats endpoints actually do support querying across multiple metrics; however, docs
are really lacking which makes it that much easier to miss. I have some docs updates
underway. I will try to get to include sections on stats queries. Let me provide a few
examples for now.
GET /hawkular/metrics/gauges/stats?tags=$tagsQuery
where tagsQuery might be like:
'hostname=host1'
'hostname=host1 AND env=stage’
'hostname=h <http://myserver.com/>ost1 OR hostname=host2’
'hostname IN [host1, host2, host3]'
'hostname NOT IN [host4, host5]’
The /gauges/stats endpoints supports all the other parameter that are supported by
/gauges/{id}/stats. There is a corresponding endpoint for counters:
GET /hawkular/metrics/counters/stats?tags=$tagsQuery
You can also query for rates:
GET /hawkular/metrics/counters/rates/stats=$tagsQuery
Again, these endpoints all support the same query parameters.
There are POST version as well where the query parameters are instead specified as JSON in
the request body.
POST /hawkular/metrics/gauges/stats/query
POST /hawkular/metrics/counters/stats/query
The POST endpoints were added in support of the Grafana data source plugin. See
https://issues.jboss.org/browse/HWKMETRICS-410
<https://issues.jboss.org/browse/HWKMETRICS-410> and
https://issues.jboss.org/browse/HWKMETRICS-465
<https://issues.jboss.org/browse/HWKMETRICS-465> for background on the POST
endpoints.
There are some other endpoints that provide batch query support. I will save the details
on those since are not inquiring about them.
I did a little research, and found Heiko Rupp's blog post on HawkFX where he seems to
already be exploring this idea [1]. Perhaps the query language item on the 2017 road map
refers to incorporating this?
I would like to assist in developing this feature. Given that there is already a tags
DSL, it seems like it would make sense to extend that to support metric transformations.
Grafana is also considering adding this to Grafana itself, but it doesn't look like
the ticket has any traction [2]. Additionally, shipping potentially megabytes of data to
Grafana (or the user's browser) to aggregate doesn't seem ideal.
I documented a few TSDB DSLs styles here [3].
What do you all think?
Absolutely agree that Hawkular Metrics would benefit greatly from query language. It would
compliment the Grafana plugin nicely. Contributions of any kind, mailing list, IRC
(#hawkular) discussion, pull requests, etc, are always welcomed and appreciated. I suggest
creating a ticket at https://issues.jboss.org/secure/CreateIssue!default.jspa
<https://issues.jboss.org/secure/CreateIssue!default.jspa> (be sure to select
Hawkular Metrics for the project.
Thanks,
Ben
[1] http://pilhuhn.blogspot.com/2016/09/computed-metrics-for-hawkfx.html
<http://pilhuhn.blogspot.com/2016/09/computed-metrics-for-hawkfx.html>
[2] https://github.com/grafana/grafana/issues/3677
<https://github.com/grafana/grafana/issues/3677>
[3] https://gist.github.com/bheiskell/25af2543ee8588f70895b0311ef291d3
<https://gist.github.com/bheiskell/25af2543ee8588f70895b0311ef291d3>
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
3:14 a.m.
On 18 Apr 2017, at 0:10, Benjamin Heiskell wrote:
cross-metric
mathematics like sums, division, and moving averages (i.e.,
As I wrote in my post, the the blog post in [1] is a great motivator
for such a language.
What do you all think?
I like that :-)
And as others have said already, it is not enough to just use
the aggregations over tags, but it should allow for more complete
mathematical operations.
[1] https://www.circonus.com/2016/06/percentages-arent-people/
3:25 a.m.
On Tue, Apr 18, 2017 at 10:14 AM, Heiko W.Rupp <hrupp(a)redhat.com> wrote:
On 18 Apr 2017, at 0:10, Benjamin Heiskell wrote:
> cross-metric
> mathematics like sums, division, and moving averages (i.e.,
As I wrote in my post, the the blog post in [1] is a great motivator
for such a language.
> What do you all think?
I like that :-)
And as others have said already, it is not enough to just use
the aggregations over tags, but it should allow for more complete
mathematical operations.
[1] https://www.circonus.com/2016/06/percentages-arent-people/
This post is great.
I think that it is good to mention that Hawkular offers support for Nelson
Rules [2][3] which provide additional techniques of process control to
detect scenarios as described in the post.
Modeling is a very interesting and open topic, so offering a rich set of
tooling will help users to model better their business scenarios.
[2] https://en.wikipedia.org/wiki/Nelson_rules
[3] http://www.hawkular.org/docs/rest/rest-alerts.html#NelsonCondition
7:55 a.m.
Hi,
If we take the same approach as with tags query language, then some sort
of SQL is probably what we'll want - combined with some window functions
for time series handling (introduced in SQL:2003). We could perhaps take
Apache Calcite as a SQL query engine and implement the necessary
functions there - as an example or as a prototype (I don't know the
resource requirements of Calcite, but at least we would "easily" get a
proper SQL functionality). That same approach is used by Spark SQL to
implement SQL query language there (and few other products).
The other approach is to also look at some time series query languages
that the research has invented who knows how many (none of them have
gathered attention - but that doesn't necessarily mean they're bad, just
that they were never part of any product).
- Micke
On 04/18/2017 01:10 AM, Benjamin Heiskell wrote:
Hey all,
I recently deployed Hawkular because I found its scaling model
appealing. It has been working well for our basic monitoring usage
when combined with Grafana.
One of my consumers is trying to graph KPIs that involve cross-metric
mathematics like sums, division, and moving averages (i.e.,
post-ingress transformations). I looked into how to accomplish this
with Hawkular, but I didn't see any obvious way to do it. The stats
endpoint seems to only support aggregations on a single metric/tag at
a time. I'm finding that we can't visualize these metrics the same way
you can with other TSDBs like Graphite/Prometheus.
I did a little research, and found Heiko Rupp's blog post on HawkFX
where he seems to already be exploring this idea [1]. Perhaps the
query language item on the 2017 road map refers to incorporating this?
I would like to assist in developing this feature. Given that there is
already a tags DSL, it seems like it would make sense to extend that
to support metric transformations.
Grafana is also considering adding this to Grafana itself, but it
doesn't look like the ticket has any traction [2]. Additionally,
shipping potentially megabytes of data to Grafana (or the user's
browser) to aggregate doesn't seem ideal.
I documented a few TSDB DSLs styles here [3].
What do you all think?
Thanks,
Ben
[1] http://pilhuhn.blogspot.com/2016/09/computed-metrics-for-hawkfx.html
[2] https://github.com/grafana/grafana/issues/3677
[3] https://gist.github.com/bheiskell/25af2543ee8588f70895b0311ef291d3
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
2827
days inactive
2829
days old
6 comments
7 participants
participants (7)
-
Anton -
Benjamin Heiskell -
Heiko W.Rupp -
Jay Shaughnessy -
John Sanda -
Lucas Ponce -
Michael Burman