[Hawkular-dev] HOSA now limits amount of metrics per pod; new agent metrics added

[John Mazzitelli]

> will be in the next HOSA release, which I hope to do this weekend.

1.2.1.Final has been released:

https://hub.docker.com/r/hawkular/hawkular-openshift-agent/tags/

----- Original Message -----
> FYI: New enhancement to Hawkular OpenShift Agent (HOSA).
> 
> To avoid having a misconfigured or malicious pod from flooding HOSA and
> H-Metrics with large amounts of metric data, HOSA has now been enhanced to
> support the setting of "max_metrics_per_pod" (this is a setting in the agent
> global configuration). Its default is 50. Any pod that asks the agent to
> collect more than that (sum total across all of its endpoints) will be
> throttled down and only the maximum number of metrics will be stored for
> that pod. Note: when I say "metrics" here I do not mean datapoints - this
> limits the number of unique metric IDs allowed to be stored per pod)
> 
> If you enable the status endpoint, you'll see this in the yaml report when a
> max limit is reached for the endpoint in question:
> 
> openshift-infra|the-pod-name-73fgt|prometheus|http://172.19.0.5:8080/metrics:
> METRIC
>     LIMIT EXCEEDED. Last collection at [Sat, 11 Feb 2017 13:46:44 +0000]
>     gathered
>     [54] metrics, [4] were discarded, in [1.697787ms]
> 
> A warning will also be logged in the log file:
> 
>    "Reached max limit of metrics for
>    [openshift-infra|the-pod-name-73fgt|prometheus|http://172.19.0.5:8080/metrics]
>    - discarding [4] collected metrics"
> 
> (As part of this code change, the status endpoint was enhanced to now show
> the number of metrics collected from each endpoint under each pod. This is
> not the total number of datapoints; it is showing unique metric IDs - this
> number will always be <= the max metrics per pod)
> 
> Finally, the agent now collects and emits 4 metrics of its own (in addition
> to all the other "go" related ones like memory used, etc). They are:
> 
> 1 Counter:
> 
> hawkular_openshift_agent_metric_data_points_collected_total
>   The total number of individual metric data points collected from all
>   endpoints.
> 
> 3 Gauges:
> 
> hawkular_openshift_agent_monitored_pods
>   The number of pods currently being monitored.
> 
> hawkular_openshift_agent_monitored_endpoints
>   The number of endpoints currently being monitored.
> 
> hawkular_openshift_agent_monitored_metrics
>   The total number of metrics currently being monitored across all endpoints.
> 
> All of this is in master and will be in the next HOSA release, which I hope
> to do this weekend.