[Hawkular-dev] PoC hawkular wildfly agent and prometheus working

[John Mazzitelli]

I have a PoC working with the agent scraping a Prometheus endpoint and storing the metric data in Hawkular-Metrics. It came together faster than I expected :)

This can be used as a standalone feature - in other words, you don't have to run the agent collecting everything (DMR + JMX + Platform + Prometheus data) though you can if you want a "uber-agent" to collect from multiple sources. But you can just have a minimally configured agent collect just Prometheus data. The config in standalone.xml would look like this (this is the entire agent subsystem config):

        <subsystem xmlns="urn:org.hawkular.agent:agent:1.0" enabled="true">
            <diagnostics enabled="false"/>
            <storage-adapter password="password" serverOutboundSocketBindingRef="hawkular" type="HAWKULAR" username="jdoe"/>
            <managed-servers>
                 <remote-prometheus enabled="true"
                                    name="My Prometheus Endpoint"
                                    url="http://my-prometheus-endpoint:8080/metrics"
                                    interval="10"
                                    timeUnits="seconds" />
                  <!-- you can have more <remote-prometheus> elements here to scrape multiple endpoints -->
            </managed-servers>
        </subsystem>

This is just a prototype. Still lots of questions to answer like:

1) What do we want the keys to look like in H-Metrics? How to translate Prometheus keys/labels to H-Metrics? What this PoC does right now is build a huge H-Metrics key that appends the feed ID, the Prometheus metric name and Prometheus labels so it looks like this "b6fb9814-e6b5-435a-b533-411d85867720_prometheus_rule_evaluation_failures_total_rule_type=alerting_" (rule_type=alerting is a label).

2) Prometheus supports summary and histograms - what do we do with those? I ignore them for now. (btw: even histograms isn't supported in Prometheus' Java model API that is released, only supported in local snapshot build form only)

3) Do we want to create labels in H-Metrics to distinguish the Prometheus endpoint data? How?

4) No security whatsoever in here - PoC assumes in-the-clear http access. We'd need to add https, basic auth, ssl/security realms and anything else needed to access any Prometheus endpoints that are secured. I'm sure Open Shift has some things we need to work around to scrape Prometheus endpoints that are running in an OS environment - this is something we need to figure out.

5) Notice there is no metadata in the above config. There is no integration with H-Inventory - no resource types, no resources, no metric types. Prometheus doesn't really have the notion of inventory - at best, each Prometheus endpoint could be a "resource". But this PoC does nothing like that - it is a metrics-only solution (which as I understand it is all we need for the Open Shift requirement anyway).

And I'm sure plenty more questions. But, point is, we have a working example of how this can be done. Code is in this branch:

https://github.com/hawkular/hawkular-agent/tree/mazz/hwkagent-66-dynamic-prometheus

--John Mazz