New Hawkular blog post from noreply(a)hawkular.org (Joel Takvorian): http://ift.tt/2uPct0T There have been improvements lately in the Hawkular Grafana datasource that are worth mentioning. The way to query metrics by tags has changed since plugin v1.0.8. It now takes advantage of the Hawkular Metrics' tags query language, that was introduced server-side in Metrics 0.24.0 and enhanced in 0.25.0. To sum it up, Metrics integrates a parser that allows queries such as: tag1 = 'awesome!' AND tag2 NOT IN ['foo', 'bar']. The datasource in now able to fetch bucketized metrics stats, instead of raw metrics. It consists in aggregating datapoints in slices of time (buckets) and providing, for each slice, some statistics like min, max, average and more. The exact content of a bucket is described in Metrics REST API. Hawkular has always been able to provide metric stats server-side, but being able to use them in the Grafana plugin is new, introduced in v1.0.9. The new query interface Tags query language The first change is that you don’t have to choose between query by tag and query by metric id anymore, you can do both at the same time. Querying by tag will refine the available list of metric names (much like a filter) and can result in multiple metrics from a single query. By selecting a metric name, you restrict the query to only display that one. This filtering is really nice when there’s tons of metrics available, like in the case of hundreds of OpenShift pods being monitored with the same tenant. The simple key/value pairs interface is now replaced with a more elaborated query builder, following the pattern: 'tag-key' 'operator' 'tag-value(s)' ['AND/OR' etc.] The following images show a walk-through: Selecting the tag key Selecting the tag operator Selecting the tag value The text fields include dynamic suggestions, you can use Grafana template variables within tag values, or enter free text. Once you’ve set up a tag query expression, the relevant metrics immediately show up on the chart and the list of available metrics in the dropdown list in updated. Filtered metrics This query builder lets you build almost any tag query that the Hawkular server understands. There are however some corner cases. For now this builder doesn’t allow you to prioritize expressions with parentheses. For instance, you cannot build c1 = 'foo' OR (b1 != 'B' AND a1 = 'abcd'). As a workaround you can turn off the query builder and directly type in your query expression. Toggle editor mode It will be sent as is to the Hawkular Metrics server. This will also be useful to fill the gap if the language evolves server-side and this plugin isn’t updated immediately. Stats query The other important feature is the ability to run stats queries against Hawkular Metrics, instead of raw queries. There are several reasons to do this: it reduces the network load, and client-side processing load, especially when raw data would contain tons of datapoints it enables some aggregation methods it also allows higher-level analysis with stats such as percentiles To enable it, just clear the raw checkbox. Toggle stats mode When you clear the raw checkbox, you can configure Multiple series aggregation to None, Sum or Average and can configure Stat type as avg, min, max, median, sum and different percentiles. You can display several different Stat types within the same query. Stats without aggregation: each two metrics show avg, min and max Same query with series aggregation: the two metrics are averaged into one, which shows avg, min and max If the query returns multiple series, use Multiple series aggregation to define if and how to aggregate them. None will show them individually on the chart. But consider for instance the case of an OpenShift pod with many replicas, and you’re tracking their memory usage. It may be more relevant here to aggregate all of them, both sum and average are meaningful here. The Stat type option refers to an aggregation at a different level: not between multiple metrics, but within a single metric, all raw datapoints are aggregated within time buckets. Conclusion These two improvements aim a common goal, that is facilitating querying over large amounts of data. This is becoming crucial especially in the context of microservices and applications running on container platforms, as the number of metrics explodes. Proper metrics tagging is the corner stone to make sense of this data. from Hawkular Blog