Note that when HOSA stores the data point in H-Metrics, it does add the
prometheus labels to that datapoint as H-Metric tags. So H-Metrics does
have each datapoint properly tagged.
I haven't followed H-Metrics close enough to know if they plan on
addressing how to query that kind of tagged data to make it useful.
On Thu, 2017-01-19 at 14:44 -0500, John Mazzitelli wrote:
So the problem (if I understand correctly) is that if a Prometheus
metric has labels, we need to create individual metrics because
otherwise there is going to be one hawkular metric definition but it
can't represent all the data because of the different labels on the
Prometheus metric (which really represent multiple metrics).
Won't this be a problem with counter metrics too? It all seems to
stem
from the fact of Prometheus labels.
You should document this in a github issue with all the details so we
can track it there.
At a quick glance without thinking about it much, I think your
proposal
makes sense:
metrics:
- name: kube_pod_container_requested_cpu_cores
id: kube_pod_container_requested_cpu_cores_${container}_${node}
unique_metric_per_label: true
I'd like to avoid having to specifying "unique_metric_per_label" at
all
and just have it inferred. Perhaps something like this: "if ID has at
least one "${x}" it means we need to create unique metric definitions
per label"
On Thu, 2017-01-19 at 17:50 +0000, Gareth Healy wrote:
Any thoughts on the below? i.e.: should the agent create individual
metrics or should the endpoints be able to group.
Without this, it makes prometheus gauge metrics pretty useless.
Cheers.
On Tue, Jan 10, 2017 at 1:26 PM, Gareth Healy <garethahealy@gmail.c
om
wrote:
To carry on the discussion around whether the agent should create
single metrics or hawkular should group the datapoints raised by
Joel.
I've done a bit more digging into different prometheus endpoints
and heres an example from some work being done by Kuberenetes:
https://github.com/kubernetes/kube-state-metrics
# HELP kube_node_info Information about a cluster node.
# TYPE kube_node_info gauge
kube_node_info{container_runtime_version="docker://1.10.3",kernel
_v
ersion="3.10.0-
327.28.3.el7.x86_64",kubelet_version="v1.3.0+52492b4",kubeproxy_v
er
sion="v1.3.0+52492b4",node="origin",os_image="CentOS Linux 7
(Core)"} 1
# HELP kube_pod_status_scheduled Describes the status of the
scheduling process for the pod.
# TYPE kube_pod_status_scheduled gauge
kube_pod_status_scheduled{condition="true",namespace="cockpit",po
d=
"openshift-cockpit-1-0jub2"} 1
# HELP kube_pod_container_requested_cpu_cores The number of
requested cpu cores by a container.
# TYPE kube_pod_container_requested_cpu_cores gauge
kube_pod_container_requested_cpu_cores{container="registry",names
pa
ce="default",node="origin",pod="docker-registry-1-i37wn"} 0.1
If the agent was to create single metrics, the idea in my mind
would be as follows:
Add a new boolean flag to create unique metrics based on labels
(default=false, to keep functionality as-is)
Allow use of "id" field to contain value replacements based on
label value (probably need to be sanitised / escaped??)
i.e.:
metrics:
- name: kube_pod_container_requested_cpu_cores
id:
kube_pod_container_requested_cpu_cores_${container}_${node}
unique_metric_per_label: true
Which would create a gauge called:
kube_pod_container_requested_cpu_cores_registry_origin
Does that sound feasible? and thoughts?
On Wed, Jan 4, 2017 at 1:06 PM, John Mazzitelli <mazz@redhat.com>
wrote:
Here, query by tag means time series tags, not data points
tags. That's
probably why you don't get anything. Maybe we should more
clearly
disambiguate these two concepts as there's often some
confusion.
:-) And that, I think, is what I meant by tags on "metric
definitions" - I think the H-Metrics team calls them
"timeseries
tags".
_______________________________________________
hawkular-dev mailing list
hawkular-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
_______________________________________________
hawkular-dev mailing list
hawkular-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
_______________________________________________
hawkular-dev mailing list
hawkular-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
_______________________________________________
hawkular-dev mailing list
hawkular-dev@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/hawkular-dev