10:36 a.m.
Hi,
I just want to share some ideas about the eventuality of having a language
to perform some arithmetic / aggregations on metrics, before it goes out of
my head...
Here's an example of what I would personally love to see in Hawkular:
----------------------------------------------
*Example:*
*sum(stats(rate((id(my_metric), tags(a=foo AND b=bar),
regexp(something_.+)), 5m), 10m))*
|
|=> "id", "tags" and "regexp" all return a set of raw
metrics (0-1 for id,
0-n for tags and regexp)
|==> "(a,b,c)" takes n parameters, all sets of metrics, and flatten them in
a single set
|===> rate(set_of_raw_metrics, rate_period) computes the rate for each of
them and return a set of metrics (map n=>n)
|====> stats(set_of_raw_metrics, bucket_size) bucketize the raw metrics,
returning the same number of bucketized metrics (map n=>n)
|=====> sum(set_of_stats_metrics) sums every buckets, returning a single
bucketized metric (fold n=>1)
*Other:*
Functions like "sum" that take stats_metrics could have overloaded shortcut
"sum(set_of_raw_metrics, bucket_size)" to perform the bucketing.
In other words above example could be rewritten:
*sum(rate((id(my_metric), tags(a=foo AND b=bar), regexp(something_.+)),
5m), 10m)*
Note: we can do aggregations like "sum" on raw data if necessary, it just
means we have to interpolate.
*Scalar operations:*
*sum((id(a_metric_in_milliseconds), 1000*id(a_metric_in_seconds)), 10m)*
----------------------------------------------
Of course many other functions could come growing the library.
Now I suppose the big question, if we want to do such thing, is "are we
going to invent our own language?" I don't know if there are standards for
this, and if they are good.
The Prometheus query language cannot be transposed because a label is not a
tag and it makes no sense for us to write something like
"my_metric{tag=foo}", it's either "my_metric" or
"tag=foo".
The same language could be used both for read-time on-the-fly aggregations
and write-time / cron-based rollups.
WDYT?
12:51 p.m.
Take a look at the query language of Prometheus:
https://prometheus.io/docs/querying/basics/
----- Original Message -----
Hi,
I just want to share some ideas about the eventuality of having a language to
perform some arithmetic / aggregations on metrics, before it goes out of my
head...
Here's an example of what I would personally love to see in Hawkular:
----------------------------------------------
Example:
sum(stats(rate((id(my_metric), tags(a=foo AND b=bar), regexp(something_.+)),
5m), 10m))
|
|=> "id", "tags" and "regexp" all return a set of raw
metrics (0-1 for id,
|0-n for tags and regexp)
|==> "(a,b,c)" takes n parameters, all sets of metrics, and flatten them in
a
|single set
|===> rate(set_of_raw_metrics, rate_period) computes the rate for each of
|them and return a set of metrics (map n=>n)
|====> stats(set_of_raw_metrics, bucket_size) bucketize the raw metrics,
|returning the same number of bucketized metrics (map n=>n)
|=====> sum(set_of_stats_metrics) sums every buckets, returning a single
|bucketized metric (fold n=>1)
Other:
Functions like "sum" that take stats_metrics could have overloaded shortcut
"sum(set_of_raw_metrics, bucket_size)" to perform the bucketing.
In other words above example could be rewritten:
sum(rate((id(my_metric), tags(a=foo AND b=bar), regexp(something_.+)), 5m),
10m)
Note: we can do aggregations like "sum" on raw data if necessary, it just
means we have to interpolate.
Scalar operations:
sum((id(a_metric_in_milliseconds), 1000* id(a_metric_in_seconds)), 10m)
----------------------------------------------
Of course many other functions could come growing the library.
Now I suppose the big question, if we want to do such thing, is "are we going
to invent our own language?" I don't know if there are standards for this,
and if they are good.
The Prometheus query language cannot be transposed because a label is not a
tag and it makes no sense for us to write something like
"my_metric{tag=foo}", it's either "my_metric" or
"tag=foo".
The same language could be used both for read-time on-the-fly aggregations
and write-time / cron-based rollups.
WDYT?
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
1:51 a.m.
Hi,
I'd personally would want to see another type of functional language,
such that would look like the one when you do Streams / RxJava. I guess
it's closer to the one in MongoDB/RethinkDB/InfluxDB 2.0/Riemann, than
the one in Prometheus/Graphite (PromQL is mostly evolution of Graphite's
query language).
And it could allow us to provide features such as streaming metrics,
since we could transform those queries directly to reactive queries (we
should take advantage of the fact that we process the metrics by
reacting to them in the backend).
- Micke
On 07/19/2017 06:36 PM, Joel Takvorian wrote:
Hi,
I just want to share some ideas about the eventuality of having a
language to perform some arithmetic / aggregations on metrics, before
it goes out of my head...
Here's an example of what I would personally love to see in Hawkular:
----------------------------------------------
*Example:*
/sum(stats(rate((id(my_metric), tags(a=foo AND b=bar),
regexp(something_.+)), 5m), 10m))/
|
|=> "id", "tags" and "regexp" all return a set of raw
metrics (0-1 for
id, 0-n for tags and regexp)
|==> "(a,b,c)" takes n parameters, all sets of metrics, and flatten
them in a single set
|===> rate(set_of_raw_metrics, rate_period) computes the rate for each
of them and return a set of metrics (map n=>n)
|====> stats(set_of_raw_metrics, bucket_size) bucketize the raw
metrics, returning the same number of bucketized metrics (map n=>n)
|=====> sum(set_of_stats_metrics) sums every buckets, returning a
single bucketized metric (fold n=>1)
*Other:*
Functions like "sum" that take stats_metrics could have overloaded
shortcut "sum(set_of_raw_metrics, bucket_size)" to perform the bucketing.
In other words above example could be rewritten:
/sum(rate((id(my_metric), tags(a=foo AND b=bar),
regexp(something_.+)), 5m), 10m)/
Note: we can do aggregations like "sum" on raw data if necessary, it
just means we have to interpolate.
*Scalar operations:*
/sum((id(a_metric_in_milliseconds), *1000**id(a_metric_in_seconds)), 10m)/
----------------------------------------------
Of course many other functions could come growing the library.
Now I suppose the big question, if we want to do such thing, is "are
we going to invent our own language?" I don't know if there are
standards for this, and if they are good.
The Prometheus query language cannot be transposed because a label is
not a tag and it makes no sense for us to write something like
"my_metric{tag=foo}", it's either "my_metric" or
"tag=foo".
The same language could be used both for read-time on-the-fly
aggregations and write-time / cron-based rollups.
WDYT?
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
6:58 a.m.
The only one I've used a lot among those you mention is mongodb query
language. And honestly I don't like it very much, I find it quite
counter-intuitive to use. It's all JSON, everything must be named and
bracketed, I was quickly lost in a opened-closed-nested curly-braces hell.
For instance, querying for age > 40 is "{age:{$gt:40}}". It's probably
easy
to implement as a query language, but it can become a nightmare for users.
In db world I see it as a big step backward from SQL, which is much more
"natural". (I just found that blog post that explains it better than me:
http://www.vertabelo.com/blog/notes-from-the-lab/sql-vs-mongo-query )
On Thu, Jul 20, 2017 at 8:51 AM, Michael Burman <miburman(a)redhat.com> wrote:
Hi,
I'd personally would want to see another type of functional language,
such that would look like the one when you do Streams / RxJava. I guess
it's closer to the one in MongoDB/RethinkDB/InfluxDB 2.0/Riemann, than
the one in Prometheus/Graphite (PromQL is mostly evolution of Graphite's
query language).
And it could allow us to provide features such as streaming metrics,
since we could transform those queries directly to reactive queries (we
should take advantage of the fact that we process the metrics by
reacting to them in the backend).
- Micke
On 07/19/2017 06:36 PM, Joel Takvorian wrote:
> Hi,
>
> I just want to share some ideas about the eventuality of having a
> language to perform some arithmetic / aggregations on metrics, before
> it goes out of my head...
>
> Here's an example of what I would personally love to see in Hawkular:
>
> ----------------------------------------------
> *Example:*
> /sum(stats(rate((id(my_metric), tags(a=foo AND b=bar),
> regexp(something_.+)), 5m), 10m))/
> |
> |=> "id", "tags" and "regexp" all return a set of
raw metrics (0-1 for
> id, 0-n for tags and regexp)
> |==> "(a,b,c)" takes n parameters, all sets of metrics, and flatten
> them in a single set
> |===> rate(set_of_raw_metrics, rate_period) computes the rate for each
> of them and return a set of metrics (map n=>n)
> |====> stats(set_of_raw_metrics, bucket_size) bucketize the raw
> metrics, returning the same number of bucketized metrics (map n=>n)
> |=====> sum(set_of_stats_metrics) sums every buckets, returning a
> single bucketized metric (fold n=>1)
>
>
> *Other:*
> Functions like "sum" that take stats_metrics could have overloaded
> shortcut "sum(set_of_raw_metrics, bucket_size)" to perform the bucketing.
> In other words above example could be rewritten:
> /sum(rate((id(my_metric), tags(a=foo AND b=bar),
> regexp(something_.+)), 5m), 10m)/
>
> Note: we can do aggregations like "sum" on raw data if necessary, it
> just means we have to interpolate.
>
> *Scalar operations:*
> /sum((id(a_metric_in_milliseconds), *1000**id(a_metric_in_seconds)),
10m)/
> ----------------------------------------------
>
> Of course many other functions could come growing the library.
>
> Now I suppose the big question, if we want to do such thing, is "are
> we going to invent our own language?" I don't know if there are
> standards for this, and if they are good.
>
> The Prometheus query language cannot be transposed because a label is
> not a tag and it makes no sense for us to write something like
> "my_metric{tag=foo}", it's either "my_metric" or
"tag=foo".
>
> The same language could be used both for read-time on-the-fly
> aggregations and write-time / cron-based rollups.
>
> WDYT?
>
>
>
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
7:35 a.m.
Perhaps you could get some ideas from the expression language we put
together for the current metrics alerter. It exists today but we could
certainly change the alerter to use a new query language, if/when it
exists. I'd guess a lot of people don't realize you can even
query/alert on hMetrics using these expressions. For example, you could
do something like this to test if the 90th percentile is greater than
twice the median for a metric set defined by a tagQuery, either
combining the metrics or testing them each individually.
( q(qNow,%90) > ( 2 * q(qNow,median) ) )
http://www.hawkular.org/hawkular-metrics/docs/user-guide/#_query_based_al...
On 7/20/2017 7:58 AM, Joel Takvorian wrote:
The only one I've used a lot among those you mention is mongodb
query
language. And honestly I don't like it very much, I find it quite
counter-intuitive to use. It's all JSON, everything must be named and
bracketed, I was quickly lost in a opened-closed-nested curly-braces
hell. For instance, querying for age > 40 is "{age:{$gt:40}}". It's
probably easy to implement as a query language, but it can become a
nightmare for users. In db world I see it as a big step backward from
SQL, which is much more "natural". (I just found that blog post that
explains it better than me:
http://www.vertabelo.com/blog/notes-from-the-lab/sql-vs-mongo-query )
On Thu, Jul 20, 2017 at 8:51 AM, Michael Burman <miburman(a)redhat.com
<mailto:miburman@redhat.com>> wrote:
Hi,
I'd personally would want to see another type of functional language,
such that would look like the one when you do Streams / RxJava. I
guess
it's closer to the one in MongoDB/RethinkDB/InfluxDB 2.0/Riemann, than
the one in Prometheus/Graphite (PromQL is mostly evolution of
Graphite's
query language).
And it could allow us to provide features such as streaming metrics,
since we could transform those queries directly to reactive
queries (we
should take advantage of the fact that we process the metrics by
reacting to them in the backend).
- Micke
On 07/19/2017 06:36 PM, Joel Takvorian wrote:
> Hi,
>
> I just want to share some ideas about the eventuality of having a
> language to perform some arithmetic / aggregations on metrics,
before
> it goes out of my head...
>
> Here's an example of what I would personally love to see in
Hawkular:
>
> ----------------------------------------------
> *Example:*
> /sum(stats(rate((id(my_metric), tags(a=foo AND b=bar),
> regexp(something_.+)), 5m), 10m))/
> |
> |=> "id", "tags" and "regexp" all return a set
of raw metrics
(0-1 for
> id, 0-n for tags and regexp)
> |==> "(a,b,c)" takes n parameters, all sets of metrics, and
flatten
> them in a single set
> |===> rate(set_of_raw_metrics, rate_period) computes the rate
for each
> of them and return a set of metrics (map n=>n)
> |====> stats(set_of_raw_metrics, bucket_size) bucketize the raw
> metrics, returning the same number of bucketized metrics (map n=>n)
> |=====> sum(set_of_stats_metrics) sums every buckets, returning a
> single bucketized metric (fold n=>1)
>
>
> *Other:*
> Functions like "sum" that take stats_metrics could have overloaded
> shortcut "sum(set_of_raw_metrics, bucket_size)" to perform the
bucketing.
> In other words above example could be rewritten:
> /sum(rate((id(my_metric), tags(a=foo AND b=bar),
> regexp(something_.+)), 5m), 10m)/
>
> Note: we can do aggregations like "sum" on raw data if necessary, it
> just means we have to interpolate.
>
> *Scalar operations:*
> /sum((id(a_metric_in_milliseconds),
*1000**id(a_metric_in_seconds)), 10m)/
> ----------------------------------------------
>
> Of course many other functions could come growing the library.
>
> Now I suppose the big question, if we want to do such thing, is "are
> we going to invent our own language?" I don't know if there are
> standards for this, and if they are good.
>
> The Prometheus query language cannot be transposed because a
label is
> not a tag and it makes no sense for us to write something like
> "my_metric{tag=foo}", it's either "my_metric" or
"tag=foo".
>
> The same language could be used both for read-time on-the-fly
> aggregations and write-time / cron-based rollups.
>
> WDYT?
>
>
>
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev(a)lists.jboss.org <mailto:hawkular-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
<https://lists.jboss.org/mailman/listinfo/hawkular-dev>
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org <mailto:hawkular-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/hawkular-dev
<https://lists.jboss.org/mailman/listinfo/hawkular-dev>
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
2683
days inactive
2684
days old
4 comments
4 participants
participants (4)
-
Jay Shaughnessy -
Joel Takvorian -
John Mazzitelli -
Michael Burman