Re: [Hawkular-dev] Ability to group by datapoint tag in Grafana

On Jan 19, 2017, at 2:58 PM, John Mazzitelli <jmazzite(a)redhat.com&gt; wrote: Note that when HOSA stores the data point in H-Metrics, it does add the prometheus labels to that datapoint as H-Metric tags. So H-Metrics does have each datapoint properly tagged. I haven't followed H-Metrics close enough to know if they plan on addressing how to query that kind of tagged data to make it useful. On Thu, 2017-01-19 at 14:44 -0500, John Mazzitelli wrote: > So the problem (if I understand correctly) is that if a Prometheus > metric has labels, we need to create individual metrics because > otherwise there is going to be one hawkular metric definition but it > can't represent all the data because of the different labels on the > Prometheus metric (which really represent multiple metrics). > > Won't this be a problem with counter metrics too? It all seems to > stem > from the fact of Prometheus labels. > > You should document this in a github issue with all the details so we > can track it there. > > At a quick glance without thinking about it much, I think your > proposal > makes sense: > > metrics: > - name: kube_pod_container_requested_cpu_cores > id: kube_pod_container_requested_cpu_cores_${container}_${node} > unique_metric_per_label: true > > I'd like to avoid having to specifying "unique_metric_per_label" at > all > and just have it inferred. Perhaps something like this: "if ID has at > least one "${x}" it means we need to create unique metric definitions > per label" > > On Thu, 2017-01-19 at 17:50 +0000, Gareth Healy wrote: >> >> Any thoughts on the below? i.e.: should the agent create individual >> metrics or should the endpoints be able to group. >> >> Without this, it makes prometheus gauge metrics pretty useless. >> >> Cheers. >> >> On Tue, Jan 10, 2017 at 1:26 PM, Gareth Healy <garethahealy(a)gmail.c >> om >>> >>> wrote: >>> To carry on the discussion around whether the agent should create >>> single metrics or hawkular should group the datapoints raised by >>> Joel. >>> >>> I've done a bit more digging into different prometheus endpoints >>> and heres an example from some work being done by Kuberenetes: >>> >>> https://github.com/kubernetes/kube-state-metrics >>> # HELP kube_node_info Information about a cluster node. >>> # TYPE kube_node_info gauge >>> kube_node_info{container_runtime_version="docker://1.10.3",kernel >>> _v >>> ersion="3.10.0- >>> 327.28.3.el7.x86_64",kubelet_version="v1.3.0+52492b4",kubeproxy_v >>> er >>> sion="v1.3.0+52492b4",node="origin",os_image="CentOS Linux 7 >>> (Core)"} 1 >>> >>> # HELP kube_pod_status_scheduled Describes the status of the >>> scheduling process for the pod. >>> # TYPE kube_pod_status_scheduled gauge >>> kube_pod_status_scheduled{condition="true",namespace="cockpit",po >>> d= >>> "openshift-cockpit-1-0jub2"} 1 >>> >>> # HELP kube_pod_container_requested_cpu_cores The number of >>> requested cpu cores by a container. >>> # TYPE kube_pod_container_requested_cpu_cores gauge >>> kube_pod_container_requested_cpu_cores{container="registry",names >>> pa >>> ce="default",node="origin",pod="docker-registry-1-i37wn"} 0.1 >>> >>> If the agent was to create single metrics, the idea in my mind >>> would be as follows: >>> Add a new boolean flag to create unique metrics based on labels >>> (default=false, to keep functionality as-is) >>> Allow use of "id" field to contain value replacements based on >>> label value (probably need to be sanitised / escaped??) >>> i.e.: >>> >>>> >>>> metrics: >>>> - name: kube_pod_container_requested_cpu_cores >>>> id: >>>> kube_pod_container_requested_cpu_cores_${container}_${node} >>>> unique_metric_per_label: true >>> Which would create a gauge called: >>> >>>> >>>> kube_pod_container_requested_cpu_cores_registry_origin >>> Does that sound feasible? and thoughts? >>> >>> On Wed, Jan 4, 2017 at 1:06 PM, John Mazzitelli <mazz(a)redhat.com&gt; >>> wrote: >>>> >>>>> >>>>> Here, query by tag means time series tags, not data points >>>> tags. That's >>>>> >>>>> probably why you don't get anything. Maybe we should more >>>> clearly >>>>> >>>>> disambiguate these two concepts as there's often some >>>> confusion. >>>> >>>> :-) And that, I think, is what I meant by tags on "metric >>>> definitions" - I think the H-Metrics team calls them >>>> "timeseries >>>> tags". >>>> _______________________________________________ >>>> hawkular-dev mailing list >>>> hawkular-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/hawkular-dev >>>> >> _______________________________________________ >> hawkular-dev mailing list >> hawkular-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/hawkular-dev > _______________________________________________ > hawkular-dev mailing list > hawkular-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev