[Hawkular-dev] Ability to group by datapoint tag in Grafana

[John Mazzitelli]

So the problem (if I understand correctly) is that if a Prometheus
metric has labels, we need to create individual metrics because
otherwise there is going to be one hawkular metric definition but it
can't represent all the data because of the different labels on the
Prometheus metric (which really represent multiple metrics).

Won't this be a problem with counter metrics too? It all seems to stem
from the fact of Prometheus labels.

You should document this in a github issue with all the details so we
can track it there.

At a quick glance without thinking about it much, I think your proposal
makes sense:

metrics:
- name: kube_pod_container_requested_cpu_cores
  id: kube_pod_container_requested_cpu_cores_${container}_${node}
  unique_metric_per_label: true

I'd like to avoid having to specifying "unique_metric_per_label" at all
and just have it inferred. Perhaps something like this: "if ID has at
least one "${x}" it means we need to create unique metric definitions
per label"

On Thu, 2017-01-19 at 17:50 +0000, Gareth Healy wrote:
> Any thoughts on the below? i.e.: should the agent create individual
> metrics or should the endpoints be able to group.
> 
> Without this, it makes prometheus gauge metrics pretty useless.
> 
> Cheers.
> 
> On Tue, Jan 10, 2017 at 1:26 PM, Gareth Healy <garethahealy at gmail.com
> > wrote:
> > To carry on the discussion around whether the agent should create
> > single metrics or hawkular should group the datapoints raised by
> > Joel.
> > 
> > I've done a bit more digging into different prometheus endpoints
> > and heres an example from some work being done by Kuberenetes:
> > 
> > https://github.com/kubernetes/kube-state-metrics
> > # HELP kube_node_info Information about a cluster node.
> > # TYPE kube_node_info gauge
> > kube_node_info{container_runtime_version="docker://1.10.3",kernel_v
> > ersion="3.10.0-
> > 327.28.3.el7.x86_64",kubelet_version="v1.3.0+52492b4",kubeproxy_ver
> > sion="v1.3.0+52492b4",node="origin",os_image="CentOS Linux 7
> > (Core)"} 1
> > 
> > # HELP kube_pod_status_scheduled Describes the status of the
> > scheduling process for the pod.
> > # TYPE kube_pod_status_scheduled gauge
> > kube_pod_status_scheduled{condition="true",namespace="cockpit",pod=
> > "openshift-cockpit-1-0jub2"} 1
> > 
> > # HELP kube_pod_container_requested_cpu_cores The number of
> > requested cpu cores by a container.
> > # TYPE kube_pod_container_requested_cpu_cores gauge
> > kube_pod_container_requested_cpu_cores{container="registry",namespa
> > ce="default",node="origin",pod="docker-registry-1-i37wn"} 0.1
> > 
> > If the agent was to create single metrics, the idea in my mind
> > would be as follows:
> > Add a new boolean flag to create unique metrics based on labels
> > (default=false, to keep functionality as-is)
> > Allow use of "id" field to contain value replacements based on
> > label value (probably need to be sanitised / escaped??)
> > i.e.:
> > 
> > > metrics: 
> > > - name: kube_pod_container_requested_cpu_cores 
> > >   id:
> > > kube_pod_container_requested_cpu_cores_${container}_${node} 
> > >   unique_metric_per_label: true
> > Which would create a gauge called:
> > 
> > > kube_pod_container_requested_cpu_cores_registry_origin
> > Does that sound feasible? and thoughts?
> > 
> > On Wed, Jan 4, 2017 at 1:06 PM, John Mazzitelli <mazz at redhat.com>
> > wrote:
> > > > Here, query by tag means time series tags, not data points
> > > tags. That's
> > > > probably why you don't get anything. Maybe we should more
> > > clearly
> > > > disambiguate these two concepts as there's often some
> > > confusion.
> > > 
> > > :-) And that, I think, is what I meant by tags on "metric
> > > definitions" - I think the H-Metrics team calls them "timeseries
> > > tags".
> > > _______________________________________________
> > > hawkular-dev mailing list
> > > hawkular-dev at lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/hawkular-dev
> > > 
> > 
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev