To carry on the discussion around whether the agent should create single metrics or hawkular should group the datapoints raised by Joel.

I've done a bit more digging into different prometheus endpoints and heres an example from some work being done by Kuberenetes:

https://github.com/kubernetes/kube-state-metrics
# HELP kube_node_info Information about a cluster node.
# TYPE kube_node_info gauge
kube_node_info{container_runtime_version="docker://1.10.3",kernel_version="3.10.0-327.28.3.el7.x86_64",kubelet_version="v1.3.0+52492b4",kubeproxy_version="v1.3.0+52492b4",node="origin",os_image="CentOS Linux 7 (Core)"} 1

# HELP kube_pod_status_scheduled Describes the status of the scheduling process for the pod.
# TYPE kube_pod_status_scheduled gauge
kube_pod_status_scheduled{condition="true",namespace="cockpit",pod="openshift-cockpit-1-0jub2"} 1

# HELP kube_pod_container_requested_cpu_cores The number of requested cpu cores by a container.
# TYPE kube_pod_container_requested_cpu_cores gauge
kube_pod_container_requested_cpu_cores{container="registry",namespace="default",node="origin",pod="docker-registry-1-i37wn"} 0.1

If the agent was to create single metrics, the idea in my mind would be as follows:

Add a new boolean flag to create unique metrics based on labels (default=false, to keep functionality as-is)
Allow use of "id" field to contain value replacements based on label value (probably need to be sanitised / escaped??)

i.e.:

metrics:
- name: kube_pod_container_requested_cpu_cores
id: kube_pod_container_requested_cpu_cores_${container}_${node}
unique_metric_per_label: true

Which would create a gauge called:

kube_pod_container_requested_cpu_cores_registry_origin

Does that sound feasible? and thoughts?

On Wed, Jan 4, 2017 at 1:06 PM, John Mazzitelli <mazz@redhat.com> wrote:

> Here, query by tag means time series tags, not data points tags. That's
> probably why you don't get anything. Maybe we should more clearly
> disambiguate these two concepts as there's often some confusion.

:-) And that, I think, is what I meant by tags on "metric definitions" - I think the H-Metrics team calls them "timeseries tags".

_______________________________________________
hawkular-dev mailing list
hawkular-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev