[Hawkular-dev] Ability to group by datapoint tag in Grafana

[John Mazzitelli]

Note that when HOSA stores the data point in H-Metrics, it does add the
prometheus labels to that datapoint as H-Metric tags. So H-Metrics does
have each datapoint properly tagged.

I haven't followed H-Metrics close enough to know if they plan on
addressing how to query that kind of tagged data to make it useful.

On Thu, 2017-01-19 at 14:44 -0500, John Mazzitelli wrote:
> So the problem (if I understand correctly) is that if a Prometheus
> metric has labels, we need to create individual metrics because
> otherwise there is going to be one hawkular metric definition but it
> can't represent all the data because of the different labels on the
> Prometheus metric (which really represent multiple metrics).
> 
> Won't this be a problem with counter metrics too? It all seems to
> stem
> from the fact of Prometheus labels.
> 
> You should document this in a github issue with all the details so we
> can track it there.
> 
> At a quick glance without thinking about it much, I think your
> proposal
> makes sense:
> 
> metrics:
> - name: kube_pod_container_requested_cpu_cores
>   id: kube_pod_container_requested_cpu_cores_${container}_${node}
>   unique_metric_per_label: true
> 
> I'd like to avoid having to specifying "unique_metric_per_label" at
> all
> and just have it inferred. Perhaps something like this: "if ID has at
> least one "${x}" it means we need to create unique metric definitions
> per label"
> 
> On Thu, 2017-01-19 at 17:50 +0000, Gareth Healy wrote:
> > 
> > Any thoughts on the below? i.e.: should the agent create individual
> > metrics or should the endpoints be able to group.
> > 
> > Without this, it makes prometheus gauge metrics pretty useless.
> > 
> > Cheers.
> > 
> > On Tue, Jan 10, 2017 at 1:26 PM, Gareth Healy <garethahealy at gmail.c
> > om
> > > 
> > > wrote:
> > > To carry on the discussion around whether the agent should create
> > > single metrics or hawkular should group the datapoints raised by
> > > Joel.
> > > 
> > > I've done a bit more digging into different prometheus endpoints
> > > and heres an example from some work being done by Kuberenetes:
> > > 
> > > https://github.com/kubernetes/kube-state-metrics
> > > # HELP kube_node_info Information about a cluster node.
> > > # TYPE kube_node_info gauge
> > > kube_node_info{container_runtime_version="docker://1.10.3",kernel
> > > _v
> > > ersion="3.10.0-
> > > 327.28.3.el7.x86_64",kubelet_version="v1.3.0+52492b4",kubeproxy_v
> > > er
> > > sion="v1.3.0+52492b4",node="origin",os_image="CentOS Linux 7
> > > (Core)"} 1
> > > 
> > > # HELP kube_pod_status_scheduled Describes the status of the
> > > scheduling process for the pod.
> > > # TYPE kube_pod_status_scheduled gauge
> > > kube_pod_status_scheduled{condition="true",namespace="cockpit",po
> > > d=
> > > "openshift-cockpit-1-0jub2"} 1
> > > 
> > > # HELP kube_pod_container_requested_cpu_cores The number of
> > > requested cpu cores by a container.
> > > # TYPE kube_pod_container_requested_cpu_cores gauge
> > > kube_pod_container_requested_cpu_cores{container="registry",names
> > > pa
> > > ce="default",node="origin",pod="docker-registry-1-i37wn"} 0.1
> > > 
> > > If the agent was to create single metrics, the idea in my mind
> > > would be as follows:
> > > Add a new boolean flag to create unique metrics based on labels
> > > (default=false, to keep functionality as-is)
> > > Allow use of "id" field to contain value replacements based on
> > > label value (probably need to be sanitised / escaped??)
> > > i.e.:
> > > 
> > > > 
> > > > metrics: 
> > > > - name: kube_pod_container_requested_cpu_cores 
> > > >   id:
> > > > kube_pod_container_requested_cpu_cores_${container}_${node} 
> > > >   unique_metric_per_label: true
> > > Which would create a gauge called:
> > > 
> > > > 
> > > > kube_pod_container_requested_cpu_cores_registry_origin
> > > Does that sound feasible? and thoughts?
> > > 
> > > On Wed, Jan 4, 2017 at 1:06 PM, John Mazzitelli <mazz at redhat.com>
> > > wrote:
> > > > 
> > > > > 
> > > > > Here, query by tag means time series tags, not data points
> > > > tags. That's
> > > > > 
> > > > > probably why you don't get anything. Maybe we should more
> > > > clearly
> > > > > 
> > > > > disambiguate these two concepts as there's often some
> > > > confusion.
> > > > 
> > > > :-) And that, I think, is what I meant by tags on "metric
> > > > definitions" - I think the H-Metrics team calls them
> > > > "timeseries
> > > > tags".
> > > > _______________________________________________
> > > > hawkular-dev mailing list
> > > > hawkular-dev at lists.jboss.org
> > > > https://lists.jboss.org/mailman/listinfo/hawkular-dev
> > > > 
> > _______________________________________________
> > hawkular-dev mailing list
> > hawkular-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/hawkular-dev
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev