6:54 p.m.
The team can report a pretty good milestone as of today. We have Prometheus linked with
Hawkular in a nice way. Not fully integrated (we don't have, say, a single image with
everything) but works pretty good for now.
Here's some details on how to demo this (you should have hawkular-services repo git
cloned locally and the hawkular-1275 branch checked out):
1) Build hawkular-services dist with -Pdev - we will run it from the target/ directory in
the next step:
mvn clean install -Pdev
2) Run Hawkular-Services:
hawkular-services/dist/target/hawkular-services-dist-*/bin/standalone.sh -b 0.0.0.0
3) Run Prometheus configured to scan our hawkular metrics endpoints - we have a test
script that runs a pre-configured Prometheus in a docker container:
hawkular-services/dist/src/test/scripts/run-prometheus.sh
OK, that's it. You are ready to see inventory and metrics.
At this point your server is running and an agent is running with it. We have an agent
attached to h-services so the agent can monitor the server.
The agent itself has a prometheus metrics endpoint exposed via built-in jmx exporter. You
can see the raw data from the agent if you point your browser to
http://localhost:9779/metrics.
After you give it some time, the agent will report all inventory up to the server. The
server will also create a Prometheus scrape configuration file to tell Prometheus where
the agent's metrics endpoint is. The pre-configured Prometheus discovers any new
scrape configuration files as new agents come online. So without doing anything manually,
Prometheus will immediately start collecting metrics data from agents as they come
online.
You can see the inventory in our little inventory UI - point your browser to
http://localhost:8080/hawkular/inventory/ui and look at the tree of resources.
Now for the Prometheus integration - click on a resource in the inventory UI's
left-hand tree, pick a resource that has metrics (say, "[WildFly Server]
Local"). On the right side, you'll see the default base URL for prometheus listed
as "http://localhost:9090" in a text field - you can leave that as-is or edit it
if you need to. But this is referring to the Prometheus you started in step 3 above. Now
drop down the Metrics menu and pick a metric. The browser opens a tab and takes you to the
metric graph in Prometheus. You should see data.
What does all of this get you? Well, normally Prometheus data isn't organized by
associated resources - its just a long list of labeled metrics that you have to make sense
out of in order to organize the metrics yourself. Hawkular gives you a nice inventory view
(your hierarchy of resources) that does this organization of Prometheus metrics for you.
So while Prometheus gives you metric data and pretty graphs, Hawkular
"organizes" those metrics by grouping them with their associated resources in
hierarchical inventory view.
--John Mazz