I just want to share some ideas about the eventuality of having a language to perform some arithmetic / aggregations on metrics, before it goes out of my head...
sum(stats(rate((id(my_metric), tags(a=foo AND b=bar), regexp(something_.+)), 5m), 10m))
|
|=> "id", "tags" and "regexp" all return a set of raw metrics (0-1 for id, 0-n for tags and regexp)
|==> "(a,b,c)" takes n parameters, all sets of metrics, and flatten them in a single set
|===> rate(set_of_raw_metrics, rate_period) computes the rate for each of them and return a set of metrics (map n=>n)
|====> stats(set_of_raw_metrics, bucket_size) bucketize the raw metrics, returning the same number of bucketized metrics (map n=>n)
|=====> sum(set_of_stats_metrics) sums every buckets, returning a single bucketized metric (fold n=>1)
Other:
Functions like "sum" that take stats_metrics could have overloaded shortcut "sum(set_of_raw_metrics, bucket_size)" to perform the bucketing.
In other words above example could be rewritten:
sum(rate((id(my_metric), tags(a=foo AND b=bar), regexp(something_.+)), 5m), 10m)
Note: we can do aggregations like "sum" on raw data if necessary, it just means we have to interpolate.
Scalar operations:
sum((id(a_metric_in_milliseconds), 1000*id(a_metric_in_seconds)), 10m)
----------------------------------------------
Of course many other functions could come growing the library.
Now I suppose the big question, if we want to do such thing, is "are we going to invent our own language?" I don't know if there are standards for this, and if they are good.
The Prometheus query language cannot be transposed because a label is not a tag and it makes no sense for us to write something like "my_metric{tag=foo}", it's either "my_metric" or "tag=foo".
The same language could be used both for read-time on-the-fly aggregations and write-time / cron-based rollups.
WDYT?