<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">I think I already asked this before, but does the tag based bucketing introduced in <a href="https://issues.jboss.org/browse/HWKMETRICS-373" class="">https://issues.jboss.org/browse/HWKMETRICS-373</a> help here?<div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jan 19, 2017, at 2:58 PM, John Mazzitelli <<a href="mailto:jmazzite@redhat.com" class="">jmazzite@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Note that when HOSA stores the data point in H-Metrics, it does add the<br class="">prometheus labels to that datapoint as H-Metric tags. So H-Metrics does<br class="">have each datapoint properly tagged.<br class=""><br class="">I haven't followed H-Metrics close enough to know if they plan on<br class="">addressing how to query that kind of tagged data to make it useful.<br class=""><br class="">On Thu, 2017-01-19 at 14:44 -0500, John Mazzitelli wrote:<br class=""><blockquote type="cite" class="">So the problem (if I understand correctly) is that if a Prometheus<br class="">metric has labels, we need to create individual metrics because<br class="">otherwise there is going to be one hawkular metric definition but it<br class="">can't represent all the data because of the different labels on the<br class="">Prometheus metric (which really represent multiple metrics).<br class=""><br class="">Won't this be a problem with counter metrics too? It all seems to<br class="">stem<br class="">from the fact of Prometheus labels.<br class=""><br class="">You should document this in a github issue with all the details so we<br class="">can track it there.<br class=""><br class="">At a quick glance without thinking about it much, I think your<br class="">proposal<br class="">makes sense:<br class=""><br class="">metrics:<br class="">- name: kube_pod_container_requested_cpu_cores<br class=""> id: kube_pod_container_requested_cpu_cores_${container}_${node}<br class=""> unique_metric_per_label: true<br class=""><br class="">I'd like to avoid having to specifying "unique_metric_per_label" at<br class="">all<br class="">and just have it inferred. Perhaps something like this: "if ID has at<br class="">least one "${x}" it means we need to create unique metric definitions<br class="">per label"<br class=""><br class="">On Thu, 2017-01-19 at 17:50 +0000, Gareth Healy wrote:<br class=""><blockquote type="cite" class=""><br class="">Any thoughts on the below? i.e.: should the agent create individual<br class="">metrics or should the endpoints be able to group.<br class=""><br class="">Without this, it makes prometheus gauge metrics pretty useless.<br class=""><br class="">Cheers.<br class=""><br class="">On Tue, Jan 10, 2017 at 1:26 PM, Gareth Healy <<a href="mailto:garethahealy@gmail.c" class="">garethahealy@gmail.c</a><br class="">om<br class=""><blockquote type="cite" class=""><br class="">wrote:<br class="">To carry on the discussion around whether the agent should create<br class="">single metrics or hawkular should group the datapoints raised by<br class="">Joel.<br class=""><br class="">I've done a bit more digging into different prometheus endpoints<br class="">and heres an example from some work being done by Kuberenetes:<br class=""><br class=""><a href="https://github.com/kubernetes/kube-state-metrics" class="">https://github.com/kubernetes/kube-state-metrics</a><br class=""># HELP kube_node_info Information about a cluster node.<br class=""># TYPE kube_node_info gauge<br class="">kube_node_info{container_runtime_version="docker://1.10.3",kernel<br class="">_v<br class="">ersion="3.10.0-<br class="">327.28.3.el7.x86_64",kubelet_version="v1.3.0+52492b4",kubeproxy_v<br class="">er<br class="">sion="v1.3.0+52492b4",node="origin",os_image="CentOS Linux 7<br class="">(Core)"} 1<br class=""><br class=""># HELP kube_pod_status_scheduled Describes the status of the<br class="">scheduling process for the pod.<br class=""># TYPE kube_pod_status_scheduled gauge<br class="">kube_pod_status_scheduled{condition="true",namespace="cockpit",po<br class="">d=<br class="">"openshift-cockpit-1-0jub2"} 1<br class=""><br class=""># HELP kube_pod_container_requested_cpu_cores The number of<br class="">requested cpu cores by a container.<br class=""># TYPE kube_pod_container_requested_cpu_cores gauge<br class="">kube_pod_container_requested_cpu_cores{container="registry",names<br class="">pa<br class="">ce="default",node="origin",pod="docker-registry-1-i37wn"} 0.1<br class=""><br class="">If the agent was to create single metrics, the idea in my mind<br class="">would be as follows:<br class="">Add a new boolean flag to create unique metrics based on labels<br class="">(default=false, to keep functionality as-is)<br class="">Allow use of "id" field to contain value replacements based on<br class="">label value (probably need to be sanitised / escaped??)<br class="">i.e.:<br class=""><br class=""><blockquote type="cite" class=""><br class="">metrics: <br class="">- name: kube_pod_container_requested_cpu_cores <br class=""> id:<br class="">kube_pod_container_requested_cpu_cores_${container}_${node} <br class=""> unique_metric_per_label: true<br class=""></blockquote>Which would create a gauge called:<br class=""><br class=""><blockquote type="cite" class=""><br class="">kube_pod_container_requested_cpu_cores_registry_origin<br class=""></blockquote>Does that sound feasible? and thoughts?<br class=""><br class="">On Wed, Jan 4, 2017 at 1:06 PM, John Mazzitelli <mazz@redhat.com><br class="">wrote:<br class=""><blockquote type="cite" class=""><br class=""><blockquote type="cite" class=""><br class="">Here, query by tag means time series tags, not data points<br class=""></blockquote>tags. That's<br class=""><blockquote type="cite" class=""><br class="">probably why you don't get anything. Maybe we should more<br class=""></blockquote>clearly<br class=""><blockquote type="cite" class=""><br class="">disambiguate these two concepts as there's often some<br class=""></blockquote>confusion.<br class=""><br class="">:-) And that, I think, is what I meant by tags on "metric<br class="">definitions" - I think the H-Metrics team calls them<br class="">"timeseries<br class="">tags".<br class="">_______________________________________________<br class="">hawkular-dev mailing list<br class="">hawkular-dev@lists.jboss.org<br class="">https://lists.jboss.org/mailman/listinfo/hawkular-dev<br class=""><br class=""></blockquote></blockquote>_______________________________________________<br class="">hawkular-dev mailing list<br class=""><a href="mailto:hawkular-dev@lists.jboss.org" class="">hawkular-dev@lists.jboss.org</a><br class="">https://lists.jboss.org/mailman/listinfo/hawkular-dev<br class=""></blockquote>_______________________________________________<br class="">hawkular-dev mailing list<br class=""><a href="mailto:hawkular-dev@lists.jboss.org" class="">hawkular-dev@lists.jboss.org</a><br class="">https://lists.jboss.org/mailman/listinfo/hawkular-dev<br class=""></blockquote>_______________________________________________<br class="">hawkular-dev mailing list<br class=""><a href="mailto:hawkular-dev@lists.jboss.org" class="">hawkular-dev@lists.jboss.org</a><br class="">https://lists.jboss.org/mailman/listinfo/hawkular-dev<br class=""></div></div></blockquote></div><br class=""></div></body></html>