change to inventory metric model to store Prometheus info
by John Mazzitelli
I *think* I'm almost at the point where our inventory is able to store Prometheus metric data (metric family name and labels) in our inventory. Once done, you need only query inventory to get a resource's metric data and from there you will get the Prometheus metric family+labels so you know what you need to ask Prometheus for.
Currently, there is no place in our Metric domain object in inventory for this information. Right now I am putting it in that generic properties:
// don't copy properties - we are using the inventory properties for P labels right now
//m.getProperties().forEach((k, v) -> mb.property(k, v.toString()));
// map to the Prometheus metric timeseries
mb.property("hawkular.metric.family", m.getMetricFamily());
mb.properties(m.getMetricLabels());
The object "m" is the agent's own Metric object that has a generic properties map. Right now I don't think we store anything in there so that commented portion of the code is no big deal at the moment.
But notice the metric family name and the metric labels are all in the same generic properties for the inventory Metric object (that's what mb is).
I don't like this. It means who ever queries inventory has to strip out the "hawkular.metric.family" key from the metric generic properties - the rest of the map are the real labels.
I think we should make these first-class attributes in the inventory model:
mb.metricFamily(m.getMetricFamily())
mb.metricLabels(m.getMetricLabels())
Thoughts? (looking at you Joel :-)
[ps. this email was sent at 5:48pm on October 26, 2017]
6 years, 5 months
how to play with jmx exporter
by John Mazzitelli
If you want to test with jmx exporter, here's three lines of code you can add to the very bottom of your Hawkular Services distro's standalone.conf file:
----
wget http://central.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagen...
unzip -o -j $JBOSS_HOME/modules/system/layers/hawkular/org/hawkular/nest/main/deployments/hawkular-inventory-service.war WEB-INF/classes/wildfly-10-jmx-exporter.yml
JAVA_OPTS="$JAVA_OPTS -javaagent:$JBOSS_HOME/bin/jmx_prometheus_javaagent-0.10.jar=9779:$JBOSS_HOME/bin/wildfly-10-jmx-exporter.yml"
----
Then just run the server normally. Once up, you can see the metrics at http://localhost:9779/metrics
[this email was sent on October 25 at 9:30am EDT]
6 years, 5 months
h-services / travis / jenkins/torii
by John Mazzitelli
I updated H-Services to push out the docker image to Docker hub. I need someone to review/merge:
https://github.com/hawkular/hawkular-services/pull/200
You'll see the docker image up on docker hub here:
https://hub.docker.com/r/hawkular/hawkular-services/tags/
How it works is if someone merges a PR into master branch then docker-dist mvn module will be built with the maven goal of docker:build and the resulting image will be pushed to docker hub with docker tag of "latest". Right now the branch hawkular-1259 is special - travis will also push the docker image when we merge into that branch (with a docker tag of hawkular-1259). Once we get back onto master, I will take out that special stuff so we only release "latest" image from master.
What I need is: whoever (Juca?) has the magic keys to jenkins on that internal RH machine, turn off the pushing of the image to docker hub. This way we have all the code that does it out in the open on github not in some internal private machine.
I don't know if we still care about keeping that internal Jenkins integration job running that does the releases. If I knew exactly what it does, I could try to get travis to do that too (I have a feeling it will be more involved than the docker hub thing, so maybe we keep it). However, right now, that Jenkins job is always failing so I have no idea what good it is doing for us right now anyway (it's always failing on the hawkular-1259 branch).
6 years, 6 months
New Hawkular Blog Post: Hawkular Alerts in ManageIQ
by Thomas Heute
New Hawkular blog post from noreply(a)hawkular.org (Edgar Hernández): http://ift.tt/2ygxRgx
Hawkular project includes an alerting module that can be used to send notifications when certain conditions are met. The alerting module is bundled in Hawkular Middleware Manager (aka. Hawkular Services)
ManageIQ also has alerting capabilities and Hawkular Middleware Manager integrates to it. Once Hawkular is added as a provider, ManageIQ alerting features can be used to monitor servers managed by Hawkular.
ManageIQ alerting terminology
In ManageIQ, two terms are used:
Alert
Defines the which type of infrastructure item should be monitored, the conditions and the actions to take when the conditions are met.
Alert profile
It is a relation between a set of alerts and a set of infrastructure items. The set of alerts in the profile will all be applied to the set of infrastructure items. If any of the infrastructure items met the conditions in one of the alerts, the actions of the alert will run.
ManageIQ has support for several types of infrastructure items, but Hawkular Middleware Manager supports only a subset of them. Middleware servers are an example of infrastructure items that are supported by Hawkular.
Hawkular alerting terminology
Hawkular’s alerting module has several kinds of objects (and terms). The two objects that are relevant from a ManageIQ perspective are:
Group trigger
Defines the set of conditions that should be met to fire an alert or an event. A group trigger is a template and no alert nor events will be fired until a member is added to it.
Group member
It’s an association of a group trigger with the actual data to be evaluated. Internally to Hawkular, group members are managed instances of group triggers. Changes to the group trigger are pushed down to the members.
Currently, ManageIQ creates group triggers that raise events (not alerts) and polls Hawkular to catch the events and run configured actions, if needed.
Terminology relationships
When a user creates a ManageIQ alert, a Hawkular group trigger is created in the background. From then, any changes to the ManageIQ alert are replicated to the associated Hawkular group trigger until the alert is deleted, which causes the group trigger to also be deleted.
While ManageIQ alerts have a direct relation with Hawkular group triggers, ManageIQ alert profiles are a little bit more complicated. For each alert in a ManageIQ profile, one Hawkular group member is created for each infrastructure item in the profile.
This means that nothing is created in Hawkular if:
The alert profile is empty (has no alerts nor infrastructure items)
The alert profile has alerts but has no infrastructure items
The alert profile has infrastructure items but has no alerts
If a ManageIQ alert profile has one alert and has assigned multiple infrastructure items, then the relevant Hawkular group trigger will have as many members as infrastructure items are in the ManageIQ alert profile.
If a ManageIQ alert profile has multiple alerts and has assigned only one infrastructure item, then each Hawkular group trigger will have one member.
With these two examples, you should be able to guess what will happen if the ManageIQ alert profile has a set of alerts and a set of infrastructure items.
Creating ManageIQ alerts
Alerts are created by navigating to Control > Explorer > Alerts. In the alerts tree, select the All alerts folder. This will enable the Add a New Alert option under the Configuration button.
In the form to create/edit an alert, be sure to select Middleware server in the Based On field. Currently, this is the only infrastructure item supported by Hawkular. If you choose something else, the alert won’t be managed by Hawkular. All other options can be filled as desired.
When the alert is created, it will be available in the control explorer and will be available to be included in an alert profile. In the background, a Hawkular group trigger is also created.
Creating an alert profile
Alert profiles are created by navigating to Control > Explorer > Alert Profiles. In the alerts tree, select the Middleware server Alert Profiles item. This will enable the Add a Middleware Server Alert Profile option under the Configuration button.
In the form to create/edit an alert profile, write a description and select the desired alerts to evaluate. At least, one alert is required to be able to create the alert profile.
When the alert profile is created, it will be available in the control explorer which will also list the alerts contained in the profile. Nothing will be created in Hawkular because when the profile is created it is still not assigned to middleware servers.
Assigning middleware servers to an alert profile
To assign middleware servers to an alert profile, select the desired profile in the control explorer. In the toolbar, use the Edit assignments for this Alert Profile options under the Configuration button.
This will show the assignments page. An alert profile can be assigned to specific middleware servers or to all inventoried middleware servers (The Enterprise).
Once you have chosen the desired middleware servers (or the enterprise) and changes are saved, the view page of the alert will be displayed again.
In the background, Hawkular group members will be created to make effective the configuration and alerts should start triggering.
Viewing alerts in the timeline
The timeline of ManageIQ Hawkular provider will log events if the configuration of an alert is enabled to show timeline events. Hawkular’s provider timeline can be accessed through the summary page of the provider, under the Monitoring menu:
If the alert has enabled the standard Show on Timeline configuration, the options to query the events are:
Event type: Management Events
Category: Alarm/Status Change/Errors
Conclusion
ManageIQ and Hawkular, albeit being two independent projects, they can be connected together to complement each other features. In this post is discussed how the alerting integration works and how to configure a basic alert.
At the time of writing, ManageIQ supports only Middleware Servers as targets and, also, only a limited set of metrics is available to configure alerts. There is ongoing work to provide a wider range of metrics and this is expected to be available in following versions.
from Hawkular Blog
6 years, 6 months