Re: [Hawkular-dev] Ability to group by datapoint tag in Grafana

So the problem (if I understand correctly) is that if a Prometheus metric has labels, we need to create individual metrics because otherwise there is going to be one hawkular metric definition but it can't represent all the data because of the different labels on the Prometheus metric (which really represent multiple metrics). Won't this be a problem with counter metrics too? It all seems to stem from the fact of Prometheus labels. You should document this in a github issue with all the details so we can track it there. At a quick glance without thinking about it much, I think your proposal makes sense: metrics: - name: kube_pod_container_requested_cpu_cores   id: kube_pod_container_requested_cpu_cores_${container}_${node}   unique_metric_per_label: true I'd like to avoid having to specifying "unique_metric_per_label" at all and just have it inferred. Perhaps something like this: "if ID has at least one "${x}" it means we need to create unique metric definitions per label" On Thu, 2017-01-19 at 17:50 +0000, Gareth Healy wrote: > > Any thoughts on the below? i.e.: should the agent create individual > metrics or should the endpoints be able to group. > > Without this, it makes prometheus gauge metrics pretty useless. > > Cheers. > > On Tue, Jan 10, 2017 at 1:26 PM, Gareth Healy <garethahealy(a)gmail.c > om > > > > wrote: > > To carry on the discussion around whether the agent should create > > single metrics or hawkular should group the datapoints raised by > > Joel. > > > > I've done a bit more digging into different prometheus endpoints > > and heres an example from some work being done by Kuberenetes: > > > > https://github.com/kubernetes/kube-state-metrics > > # HELP kube_node_info Information about a cluster node. > > # TYPE kube_node_info gauge > > kube_node_info{container_runtime_version="docker://1.10.3",kernel > > _v > > ersion="3.10.0- > > 327.28.3.el7.x86_64",kubelet_version="v1.3.0+52492b4",kubeproxy_v > > er > > sion="v1.3.0+52492b4",node="origin",os_image="CentOS Linux 7 > > (Core)"} 1 > > > > # HELP kube_pod_status_scheduled Describes the status of the > > scheduling process for the pod. > > # TYPE kube_pod_status_scheduled gauge > > kube_pod_status_scheduled{condition="true",namespace="cockpit",po > > d= > > "openshift-cockpit-1-0jub2"} 1 > > > > # HELP kube_pod_container_requested_cpu_cores The number of > > requested cpu cores by a container. > > # TYPE kube_pod_container_requested_cpu_cores gauge > > kube_pod_container_requested_cpu_cores{container="registry",names > > pa > > ce="default",node="origin",pod="docker-registry-1-i37wn"} 0.1 > > > > If the agent was to create single metrics, the idea in my mind > > would be as follows: > > Add a new boolean flag to create unique metrics based on labels > > (default=false, to keep functionality as-is) > > Allow use of "id" field to contain value replacements based on > > label value (probably need to be sanitised / escaped??) > > i.e.: > > > > > > > > metrics:  > > > - name: kube_pod_container_requested_cpu_cores  > > >   id: > > > kube_pod_container_requested_cpu_cores_${container}_${node}  > > >   unique_metric_per_label: true > > Which would create a gauge called: > > > > > > > > kube_pod_container_requested_cpu_cores_registry_origin > > Does that sound feasible? and thoughts? > > > > On Wed, Jan 4, 2017 at 1:06 PM, John Mazzitelli <mazz(a)redhat.com&gt; > > wrote: > > > > > > > > > > > Here, query by tag means time series tags, not data points > > > tags. That's > > > > > > > > probably why you don't get anything. Maybe we should more > > > clearly > > > > > > > > disambiguate these two concepts as there's often some > > > confusion. > > > > > > :-) And that, I think, is what I meant by tags on "metric > > > definitions" - I think the H-Metrics team calls them > > > "timeseries > > > tags". > > > _______________________________________________ > > > hawkular-dev mailing list > > > hawkular-dev(a)lists.jboss.org > > > https://lists.jboss.org/mailman/listinfo/hawkular-dev > > > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev