6:03 a.m.
I have a PoC working with the agent scraping a Prometheus endpoint and storing the metric
data in Hawkular-Metrics. It came together faster than I expected :)
This can be used as a standalone feature - in other words, you don't have to run the
agent collecting everything (DMR + JMX + Platform + Prometheus data) though you can if you
want a "uber-agent" to collect from multiple sources. But you can just have a
minimally configured agent collect just Prometheus data. The config in standalone.xml
would look like this (this is the entire agent subsystem config):
<subsystem xmlns="urn:org.hawkular.agent:agent:1.0"
enabled="true">
<diagnostics enabled="false"/>
<storage-adapter password="password"
serverOutboundSocketBindingRef="hawkular" type="HAWKULAR"
username="jdoe"/>
<managed-servers>
<remote-prometheus enabled="true"
name="My Prometheus Endpoint"
url="http://my-prometheus-endpoint:8080/metrics"
interval="10"
timeUnits="seconds" />
<!-- you can have more <remote-prometheus> elements here to
scrape multiple endpoints -->
</managed-servers>
</subsystem>
This is just a prototype. Still lots of questions to answer like:
1) What do we want the keys to look like in H-Metrics? How to translate Prometheus
keys/labels to H-Metrics? What this PoC does right now is build a huge H-Metrics key that
appends the feed ID, the Prometheus metric name and Prometheus labels so it looks like
this
"b6fb9814-e6b5-435a-b533-411d85867720_prometheus_rule_evaluation_failures_total_rule_type=alerting_"
(rule_type=alerting is a label).
2) Prometheus supports summary and histograms - what do we do with those? I ignore them
for now. (btw: even histograms isn't supported in Prometheus' Java model API that
is released, only supported in local snapshot build form only)
3) Do we want to create labels in H-Metrics to distinguish the Prometheus endpoint data?
How?
4) No security whatsoever in here - PoC assumes in-the-clear http access. We'd need to
add https, basic auth, ssl/security realms and anything else needed to access any
Prometheus endpoints that are secured. I'm sure Open Shift has some things we need to
work around to scrape Prometheus endpoints that are running in an OS environment - this is
something we need to figure out.
5) Notice there is no metadata in the above config. There is no integration with
H-Inventory - no resource types, no resources, no metric types. Prometheus doesn't
really have the notion of inventory - at best, each Prometheus endpoint could be a
"resource". But this PoC does nothing like that - it is a metrics-only solution
(which as I understand it is all we need for the Open Shift requirement anyway).
And I'm sure plenty more questions. But, point is, we have a working example of how
this can be done. Code is in this branch:
https://github.com/hawkular/hawkular-agent/tree/mazz/hwkagent-66-dynamic-...
--John Mazz