[Hawkular-dev] major progress to report - prometheus-hawkular integration is working

[John Mazzitelli]

The team can report a pretty good milestone as of today. We have Prometheus linked with Hawkular in a nice way. Not fully integrated (we don't have, say, a single image with everything) but works pretty good for now.

Here's some details on how to demo this (you should have hawkular-services repo git cloned locally and the hawkular-1275 branch checked out):

1) Build hawkular-services dist with -Pdev - we will run it from the target/ directory in the next step:

mvn clean install -Pdev

2) Run Hawkular-Services:

hawkular-services/dist/target/hawkular-services-dist-*/bin/standalone.sh -b 0.0.0.0

3) Run Prometheus configured to scan our hawkular metrics endpoints - we have a test script that runs a pre-configured Prometheus in a docker container:

hawkular-services/dist/src/test/scripts/run-prometheus.sh 

OK, that's it. You are ready to see inventory and metrics.

At this point your server is running and an agent is running with it. We have an agent attached to h-services so the agent can monitor the server.

The agent itself has a prometheus metrics endpoint exposed via built-in jmx exporter. You can see the raw data from the agent if you point your browser to http://localhost:9779/metrics.

After you give it some time, the agent will report all inventory up to the server. The server will also create a Prometheus scrape configuration file to tell Prometheus where the agent's metrics endpoint is. The pre-configured Prometheus discovers any new scrape configuration files as new agents come online. So without doing anything manually, Prometheus will immediately start collecting metrics data from agents as they come online.

You can see the inventory in our little inventory UI - point your browser to http://localhost:8080/hawkular/inventory/ui and look at the tree of resources.

Now for the Prometheus integration - click on a resource in the inventory UI's left-hand tree, pick a resource that has metrics (say, "[WildFly Server] Local"). On the right side, you'll see the default base URL for prometheus listed as "http://localhost:9090" in a text field - you can leave that as-is or edit it if you need to. But this is referring to the Prometheus you started in step 3 above. Now drop down the Metrics menu and pick a metric. The browser opens a tab and takes you to the metric graph in Prometheus. You should see data.

What does all of this get you? Well, normally Prometheus data isn't organized by associated resources - its just a long list of labeled metrics that you have to make sense out of in order to organize the metrics yourself. Hawkular gives you a nice inventory view (your hierarchy of resources) that does this organization of Prometheus metrics for you. So while Prometheus gives you metric data and pretty graphs, Hawkular "organizes" those metrics by grouping them with their associated resources in hierarchical inventory view.

--John Mazz