[Hawkular-dev] Ability to group by datapoint tag in Grafana

[John Sanda]

I think I already asked this before, but does the tag based bucketing introduced in https://issues.jboss.org/browse/HWKMETRICS-373 <https://issues.jboss.org/browse/HWKMETRICS-373> help here?

> On Jan 19, 2017, at 2:58 PM, John Mazzitelli <jmazzite at redhat.com> wrote:
> 
> Note that when HOSA stores the data point in H-Metrics, it does add the
> prometheus labels to that datapoint as H-Metric tags. So H-Metrics does
> have each datapoint properly tagged.
> 
> I haven't followed H-Metrics close enough to know if they plan on
> addressing how to query that kind of tagged data to make it useful.
> 
> On Thu, 2017-01-19 at 14:44 -0500, John Mazzitelli wrote:
>> So the problem (if I understand correctly) is that if a Prometheus
>> metric has labels, we need to create individual metrics because
>> otherwise there is going to be one hawkular metric definition but it
>> can't represent all the data because of the different labels on the
>> Prometheus metric (which really represent multiple metrics).
>> 
>> Won't this be a problem with counter metrics too? It all seems to
>> stem
>> from the fact of Prometheus labels.
>> 
>> You should document this in a github issue with all the details so we
>> can track it there.
>> 
>> At a quick glance without thinking about it much, I think your
>> proposal
>> makes sense:
>> 
>> metrics:
>> - name: kube_pod_container_requested_cpu_cores
>>   id: kube_pod_container_requested_cpu_cores_${container}_${node}
>>   unique_metric_per_label: true
>> 
>> I'd like to avoid having to specifying "unique_metric_per_label" at
>> all
>> and just have it inferred. Perhaps something like this: "if ID has at
>> least one "${x}" it means we need to create unique metric definitions
>> per label"
>> 
>> On Thu, 2017-01-19 at 17:50 +0000, Gareth Healy wrote:
>>> 
>>> Any thoughts on the below? i.e.: should the agent create individual
>>> metrics or should the endpoints be able to group.
>>> 
>>> Without this, it makes prometheus gauge metrics pretty useless.
>>> 
>>> Cheers.
>>> 
>>> On Tue, Jan 10, 2017 at 1:26 PM, Gareth Healy <garethahealy at gmail.c
>>> om
>>>> 
>>>> wrote:
>>>> To carry on the discussion around whether the agent should create
>>>> single metrics or hawkular should group the datapoints raised by
>>>> Joel.
>>>> 
>>>> I've done a bit more digging into different prometheus endpoints
>>>> and heres an example from some work being done by Kuberenetes:
>>>> 
>>>> https://github.com/kubernetes/kube-state-metrics
>>>> # HELP kube_node_info Information about a cluster node.
>>>> # TYPE kube_node_info gauge
>>>> kube_node_info{container_runtime_version="docker://1.10.3",kernel
>>>> _v
>>>> ersion="3.10.0-
>>>> 327.28.3.el7.x86_64",kubelet_version="v1.3.0+52492b4",kubeproxy_v
>>>> er
>>>> sion="v1.3.0+52492b4",node="origin",os_image="CentOS Linux 7
>>>> (Core)"} 1
>>>> 
>>>> # HELP kube_pod_status_scheduled Describes the status of the
>>>> scheduling process for the pod.
>>>> # TYPE kube_pod_status_scheduled gauge
>>>> kube_pod_status_scheduled{condition="true",namespace="cockpit",po
>>>> d=
>>>> "openshift-cockpit-1-0jub2"} 1
>>>> 
>>>> # HELP kube_pod_container_requested_cpu_cores The number of
>>>> requested cpu cores by a container.
>>>> # TYPE kube_pod_container_requested_cpu_cores gauge
>>>> kube_pod_container_requested_cpu_cores{container="registry",names
>>>> pa
>>>> ce="default",node="origin",pod="docker-registry-1-i37wn"} 0.1
>>>> 
>>>> If the agent was to create single metrics, the idea in my mind
>>>> would be as follows:
>>>> Add a new boolean flag to create unique metrics based on labels
>>>> (default=false, to keep functionality as-is)
>>>> Allow use of "id" field to contain value replacements based on
>>>> label value (probably need to be sanitised / escaped??)
>>>> i.e.:
>>>> 
>>>>> 
>>>>> metrics: 
>>>>> - name: kube_pod_container_requested_cpu_cores 
>>>>>   id:
>>>>> kube_pod_container_requested_cpu_cores_${container}_${node} 
>>>>>   unique_metric_per_label: true
>>>> Which would create a gauge called:
>>>> 
>>>>> 
>>>>> kube_pod_container_requested_cpu_cores_registry_origin
>>>> Does that sound feasible? and thoughts?
>>>> 
>>>> On Wed, Jan 4, 2017 at 1:06 PM, John Mazzitelli <mazz at redhat.com>
>>>> wrote:
>>>>> 
>>>>>> 
>>>>>> Here, query by tag means time series tags, not data points
>>>>> tags. That's
>>>>>> 
>>>>>> probably why you don't get anything. Maybe we should more
>>>>> clearly
>>>>>> 
>>>>>> disambiguate these two concepts as there's often some
>>>>> confusion.
>>>>> 
>>>>> :-) And that, I think, is what I meant by tags on "metric
>>>>> definitions" - I think the H-Metrics team calls them
>>>>> "timeseries
>>>>> tags".
>>>>> _______________________________________________
>>>>> hawkular-dev mailing list
>>>>> hawkular-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>>>>> 
>>> _______________________________________________
>>> hawkular-dev mailing list
>>> hawkular-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>> _______________________________________________
>> hawkular-dev mailing list
>> hawkular-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170119/b4c14da5/attachment.html