[Hawkular-dev] HOSA now limits amount of metrics per pod; new agent metrics added

[John Mazzitelli]

FYI: New enhancement to Hawkular OpenShift Agent (HOSA).

To avoid having a misconfigured or malicious pod from flooding HOSA and H-Metrics with large amounts of metric data, HOSA has now been enhanced to support the setting of "max_metrics_per_pod" (this is a setting in the agent global configuration). Its default is 50. Any pod that asks the agent to collect more than that (sum total across all of its endpoints) will be throttled down and only the maximum number of metrics will be stored for that pod. Note: when I say "metrics" here I do not mean datapoints - this limits the number of unique metric IDs allowed to be stored per pod)

If you enable the status endpoint, you'll see this in the yaml report when a max limit is reached for the endpoint in question:

openshift-infra|the-pod-name-73fgt|prometheus|http://172.19.0.5:8080/metrics: METRIC
    LIMIT EXCEEDED. Last collection at [Sat, 11 Feb 2017 13:46:44 +0000] gathered
    [54] metrics, [4] were discarded, in [1.697787ms]

A warning will also be logged in the log file:

   "Reached max limit of metrics for [openshift-infra|the-pod-name-73fgt|prometheus|http://172.19.0.5:8080/metrics] - discarding [4] collected metrics"

(As part of this code change, the status endpoint was enhanced to now show the number of metrics collected from each endpoint under each pod. This is not the total number of datapoints; it is showing unique metric IDs - this number will always be <= the max metrics per pod)

Finally, the agent now collects and emits 4 metrics of its own (in addition to all the other "go" related ones like memory used, etc). They are:

1 Counter:

hawkular_openshift_agent_metric_data_points_collected_total
  The total number of individual metric data points collected from all endpoints.

3 Gauges:

hawkular_openshift_agent_monitored_pods
  The number of pods currently being monitored.

hawkular_openshift_agent_monitored_endpoints
  The number of endpoints currently being monitored.

hawkular_openshift_agent_monitored_metrics
  The total number of metrics currently being monitored across all endpoints.

All of this is in master and will be in the next HOSA release, which I hope to do this weekend.