Configuring hawkular services with collectd?
by Mikhail Ramendik
Hello,
I would like to try out hawkular services. I want to monitor the local
host for a start, so collectd is an obvious choice for collecting
metrics (CPU for a start).
I have successfully set up and started cassandra, hawkular-services
and grafana. I can see the hawkular services welcome message at
http://localhost:8080
I have also started ptrans and collectd. I am attaching their
configuration files. (The username and password for hawular-services:
myUsername/myPassword, as in the installation guide)
However, I do not seem to get any metrics in hawkular. The following command:
curl -u myUsername:myPassword -X GET
"http://localhost:8080/hawkular/metrics/gauges" -H "Hawkular-Tenant:
hawkular"
returns an empty string.
I would appreciate advice about fixing my setup so I can get CPU usage
in hawkular-services (and see it in Grafana).
Thanks!
--
Yours, Mikhail Ramendik
Unless explicitly stated, all opinions in my mail are my own and do
not reflect the views of any organization
5 years, 7 months
New Hawkular Blog Post: Canary Deployment in OpenShift using OpenTracing based Application Metrics
by Thomas Heute
New Hawkular blog post from noreply(a)hawkular.org (Gary Brown): http://ift.tt/2vWrA6Y
In a previous article we showed how OpenTracing instrumentation can be used to collect application metrics, in addition to (but independent from) reported tracing data, from services deployed within a cloud environment (e.g. Kubernetes or OpenShift).
In this article we will show how this information can be used to aid a Canary deployment strategy within OpenShift.
Figure 1: Error ratio per service and version
The updated example application
We will be using the same example as used in the previous article.
However since writing that article, the configuration of the tracer and Prometheus metrics support has been simplified. There is now no explicit configuration of either, with only some auto configuration of MetricLabel beans to identify some custom labels to be added to the Prometheus metrics, e.g.
Metrics configuration used in both services:
@Configuration
public class MetricsConfiguration {
@Bean
public MetricLabel transactionLabel() {
return new BaggageMetricLabel("transaction", "n/a"); (1)
}
@Bean
public MetricLabel versionLabel() {
return new ConstMetricLabel("version", System.getenv("VERSION")); (2)
}
}
1
This metric label identifies the business transaction associated with the metrics, which can be used to isolate the specific number of requests, duration and errors that occurred when the service was used within the particular business transaction
2
This metric label identifies the service version, which is especially useful in the Canary deployment use case being discussed in this article
The first step is to following the instructions in the example for deploying and using the services within OpenShift.
Once the ./genorders.sh script has been running for a while, to generate plenty of metrics for version 0.0.1 of the services, then deploy the new version of the services. This is achieved by:
updating the versions in the pom.xml files, within the simple/accountmgr and simple/ordermgr folders from 0.0.1 to 0.0.2
re-run the mvn clean install docker:build command from the simple folder
deploy the canary versions of the services using the command oc create -f services-canary-kubernetes.yml
As our services accountmgr and ordermgr determine the backing deployment based on the respective labels app: accountmgr and app: ordermgr, simply having a second deployment with these labels will make them serve requests in a round-robin manner.
This deployment script has been pre-configured with the 0.0.2 version, and to only start a single instance of the new version of the services. This may be desirable if you want to monitor the behaviour of the new service versions over a reasonable time period, but as we want to see results faster we will scale them up to see more activity. You can do this by expanding the deployment area for each service in the OpenShift web console and selecting the up arrow to scale up each service:
Figure 2: Scaling up canary deployment
Now we can monitor the Prometheus dashboard, using the following query, to see the error ratio per service and version:
sum(increase(span_count{error="true",span_kind="server"}[1m])) without (pod,instance,job,namespace,endpoint,transaction,error,operation,span_kind) / sum(increase(span_count{span_kind="server"}[1m])) without (pod,instance,job,namespace,endpoint,transaction,error,operation,span_kind)
The result of this query can be seen in Figure 1 at the top of the article. This chart shows that version 0.0.2 of the accountmgr service has not generated any errors, while the 0.0.2 of the ordermgr appears to be less error prone than version 0.0.1.
Based on these metrics, we could decide that the new versions of these services are better than the previous, and therefore update the main service deployments to use the new versions. In the OpenShift web console you can do this by clicking the three vertical dots in the upper right hand side of the deployment region and selecting Edit YAML from the menu. This will display an editor window where you can change the version from 0.0.1 to 0.0.2 in the YAML file.
Figure 3: Update the service version
After you save the YAML configuration file, in the web console you can see the service going through a "rolling update" as OpenShift incrementally changes each service instance over to the new version.
Figure 4: Rolling update
After the rolling update has completed for both the ordermgr and accountmgr services, then you can scale down or completely remove the canary version of each deployment.
An alternative to performing the rolling update would simply be to name the canary version something else (i.e. specific to the version being tested), and when it comes time to switch over, simply scale down the previous deployment version. This would be more straightforward, but wouldn’t show off the cool rolling update approach in the OpenShift web console :-)
Figure 5: Scaling down canary deployment
Although we have updated both services at the same time, this is not necessary. Generally microservices would be managed by separate teams and subject to their own deployment lifecycles.
Conclusion
This article has shown how application metrics, captured by instrumenting services using the OpenTracing API, can be used to support a simple Canary deployment strategy.
These metrics can similarly be used with other deployment strategies, such as A/B testing, which can be achieved using a weighted load balancing capability within OpenShift.
Links
OpenTracing: http://opentracing.io
Github repository with demo: http://ift.tt/2rX5MoK
OpenTracing java metrics: http://ift.tt/2rWUNvF
Kubernetes: https://kubernetes.io
OpenShift: https://openshift.io
Jaeger: http://ift.tt/2eOSqHE
Prometheus: https://prometheus.io
from Hawkular Blog
5 years, 7 months
New Hawkular Blog Post: Advanced Behaviour Detection with Nelson Rules
by Thomas Heute
New Hawkular blog post from noreply(a)hawkular.org (Lucas Ponce): http://ift.tt/2uIqytE
Modeling Conditions
Hawkular Alerting offers several types of Conditions for defining Triggers. Most of the Conditions deal with numeric data but String, Availability and Event data are also supported.
Modeling scenarios for detecting behaviours is highly dependent on the nature of the Domain being represented. The Domain may only require simple numeric threshold conditions to efficiently detect unexpected situations.
In other domains, it can be non-trivial to identify unusual metric variations that may lead to a problem. Simple thresholds are not expressive enough to detect metric patterns or trends that can identify potential problems.
Nelson Rules
Hawkular Alerting supports Conditions based on Nelson Rules to enable advanced detection on Numeric metrics.
These rules are based on the mean and the standard deviation of the samples and offer additional techniques for modeling complex scenarios.
For example,
...
"trigger":{
"id": "nelson-rule-trigger",
"name": "Nelson Rule Trigger",
"description": "An example Trigger that uses Nelson Rules Conditions.",
"enabled": true,
"actions":[]
},
"conditions":[
{
"type": "NELSON", (1)
"dataId": "metric-data-id",
"activeRules": ["Rule1","Rule2"], (2)
"sampleSize": 75 (3)
}
]
...
Mark this Condition as a NelsonRule
Define the Nelson Rules to activate (Rule1, Rule2, …, Rule8) for metric-data-id (all rules are activated by default)
Define the sampleSize (by default this value is set to 50)
Each rule represents a specific pattern as described below:
Rule 1
One sample is grossly out of control.
Rule 2
Some prolonged error has been detected.
Rule 3
An unusual trend has been detected.
Rule 4
The oscillation of a metric is beyond an expected amount of noise.
Note that the rule is concerned with directionality only. The position of the mean and the size of the standard deviation have no bearing.
Rule 5
There is a medium tendency for samples to be mediumly out of control.
The side of the mean for the third point is unspecified.
Rule 6
There is a strong tendency for samples to be out of control.
Rule 7
A greater variation would be expected.
Rule 7
Jumping from above to below whilst missing the first standard deviation band is rarely random.
Conclusion
Applying Nelson Rules in our scenario can help to detect potential "out of control" situations.
But as discussed, modeling scenarios are highly dependent of the nature of the Domain; applying Nelson Rules is a useful tool to help identify a problem. Although, the alerts are predictive and a Domain’s Analyst may need to evaluate the quality of the model.
from Hawkular Blog
5 years, 7 months
hawkular.org - Redux Initiative
by Stefan Negrea
Hello Everybody,
I would like to kickstart "Redux Initiative" for the Hawkular website. The
goal is to refocus the message and information presented on the website to
core active Hawkular projects and nothing more.
I created a document that outlines the plan ("How" section), the
modifications ("Changes" section), and a visual map of the initial cleanup
stage ("Visual Changes" section). This document was created with feedback
from Hawkular contributors and was triggered by a very insightful
interaction with a manager from Open Innovation Lab.
How to participate? The "How" section of the document outlines the big
plan. First step is to reach consensus on the changes. Please, read the
document, comment on the proposal, and participate in the discussions that
might spark here.
The document:
https://docs.google.com/a/redhat.com/document/d/1XsB_DzMjjG7QjJw-kX41PeuU...
Thank you,
Stefan Negrea
5 years, 7 months
New Hawkular Blog Post: Hawkular Alerts with Prometheus, ElasticSearch, Kafka
by Thomas Heute
New Hawkular blog post from noreply(a)hawkular.org (John Mazzitelli): http://ift.tt/2vYe1WG
Federated Alerts
Hawkular Alerts aims to be a federated alerting system. That is to say, it can fire alerts and send notifications that are triggered by data coming from a number of third-party external systems.
Thus, Hawkular Alerts is more than just an alerting system for use with Hawkular Metrics. In fact, Hawkular Alerts can be used independently of Hawkular Metrics. This means you do not even have to be using Hawkular Metrics to take advantage of the functionality provided by Hawkular Alerts.
This is a key differentiator between Hawkular Alerts and other alerting systems. Most alerting systems only alert on data coming from their respective storage systems (e.g. the Prometheus Alert Engine alerts only on Prometheus data). Hawkular Alerts, on the other hand, can trigger alerts based on data from various systems.
Alerts vs. Events
Before we begin, a quick clarification is in order. When it is said that Hawkular Alerts fires an "alert" it means some data came into Hawkular Alerts that matched some conditions which triggered the creation of an alert in Hawkular Alerts backend storage (which can then trigger additional actions such as sending emails or calling a webhook). An "alert" typically refers to a problem that has been detected, and someone should take action to fix it. An alert has a lifecycle attached to it - alerts are opened, then acknowledged by some user who will hopefully fix the problem, then resolved when the problem can be considered closed.
However, there can be conditions that occur that do not represent problems but nevertheless are events you want recorded. There is no lifecycle associated with events and no additional actions are triggered by events, but "events" are fired by Hawkular Alerts in the same general manner as "alerts" are.
In this document, when it is said that Hawkular Alerts can fire "alerts" based on data coming from external third-party systems such as Prometheus, ElasticSearch, and Kakfa, this also means events can be fired as well as alerts. What this means is you can record any event (not just a "problem", aka "alert") that can be gleaned from this data coming from external third-party systems.
See alerting philosophy for more.
Demo
There is a recorded demo found here that will illustrate what this document is describing. After you read this document, you should watch the demo to gain further clarity on what is being explained. The demo is the multiple-sources example which you can run yourself found here (note: at the time of writing, this example is only found in the next branch, to be merged in master soon).
Prometheus
Hawkular Alerts can take the results of Prometheus metric queries and use the queried data for triggers that can fire alerts.
This Hawkular Alerts trigger will fire an alert (and send an email) when a Prometheus metric indicates our store’s inventory of widgets is consistently low (as defined by the Prometheus query you see in the "expression" field of the condition):
"trigger":{
"id": "low-stock-prometheus-trigger",
"name": "Low Stock",
"description": "The number of widgets in stock is consistently low.",
"severity": "MEDIUM",
"enabled": true,
"tags": {
"prometheus": "Prometheus"
},
"actions":[
{
"actionPlugin": "email",
"actionId": "email-notify-owner"
}
]
},
"conditions":[
{
"type": "EXTERNAL",
"alerterId": "prometheus",
"dataId": "prometheus-dataid",
"expression": "rate(products_in_inventory{product=\"widget\"}[30s])<2"
}
]
Integration with Prometheus Alert Engine
As a side note, though not demostrated in the example, Hawkular Alerts also has an integration with Prometheus' own Alert Engine. This means the alerts generated by Prometheus itself can be forward to Hawkular Alerts which can, in turn, be used for additional processing, perhaps for use with data that is unavailable to Prometheus that can tell Hawkular Alerts to fire other alerts. For example, Hawkular Alerts can take Prometheus alerts as input and feed it back into other conditions that trigger on the Prometheus alert along with ElasticSearch logs.
ElasticSearch
Hawkular Alerts can examine logs stored in ElasticSearch and trigger alerts based on patterns that match within the ElasticSearch log messages.
This Hawkular Alerts trigger will fire an alert (and send an email) when ElasticSearch logs indicate sales are being lost due to inventory being out of stock of items (as defined by the condition which looks for a log category of "FATAL" which happens to mean a lost sale in the case of the store’s logs). Notice dampening is enabled on this trigger - this alert will only fire when the logs indicate lost sales every 3 times.
"trigger":{
"id": "lost-sale-elasticsearch-trigger",
"name": "Lost Sale",
"description": "A sale was lost due to inventory out of stock.",
"severity": "CRITICAL",
"enabled": true,
"tags": {
"Elasticsearch": "Localhost instance"
},
"context": {
"timestamp": "@timestamp",
"filter": "{\"match\":{\"category\":\"inventory\"}}",
"interval": "10s",
"index": "store",
"mapping": "level:category,@timestamp:ctime,message:text,category:dataId,index:tags"
},
"actions":[
{
"actionPlugin": "email",
"actionId": "email-notify-owner"
}
]
},
"dampenings": [
{
"triggerMode": "FIRING",
"type":"STRICT",
"evalTrueSetting": 3
}
],
"conditions":[
{
"type": "EVENT",
"dataId": "inventory",
"expression": "category == 'FATAL'"
}
]
Kafka
Hawkular Alerts can examine data retrieved from Kafka message streams and trigger alerts based that Kafka data.
This Hawkular Alerts trigger will fire an alert when data over a Kakfa topic indicates a large purchase was made to fill the store’s inventory (as defined by the condition which evaluates to true when any number over 17 is received on the Kafka topic):
"trigger":{
"id": "large-inventory-purchase-kafka-trigger",
"name": "Large Inventory Purchase",
"description": "A large purchase was made to restock inventory.",
"severity": "LOW",
"enabled": true,
"tags": {
"Kafka": "Localhost instance"
},
"context": {
"topic": "store",
"kafka.bootstrap.servers": "localhost:9092",
"kafka.group.id": "hawkular-alerting"
},
"actions":[ ]
},
"conditions":[
{
"type": "THRESHOLD",
"dataId": "store",
"operator": "GT",
"threshold": 17
}
]
But, Wait! There’s More!
The above only mentions the different ways Hawkular Metrics retrieves data for use in determining what alerts to fire. What is not covered here is the fact that Hawkular Alerts can stream data in the other direction as well - Hawkular Alerts can send alert and event data to things like an ElasticSearch server or a Kafka broker. There are additional examples (mentioned below) that can demonstrate this capability.
The point is Hawkular Alerts should be seen as a shared, common alerting engine that can be shared for use by multiple third-party systems and can be used as both a consumer and producer - as a consumer of the data from external third-party systems (which is used to fire alerts and events) and as a producer to send notifications of alerts and events to external third-party systems.
More Examples
Take a look at the Hawkular Alerts examples for more examples on using external systems as data to be used for triggering alerts. (note: at the time of writing, some examples are currently in the next branch such as the Kafka ones).
from Hawkular Blog
5 years, 7 months
A convention for metrics (short) name
by Joel Takvorian
Hi,
What would you say about having a convention of a special tag (let's say
"_name") that would point to a (short) intelligible name for a metric. That
convention wouldn't be mandatory in any case of course, but the UI could
check if that tag exists and use that name, instead of the full metric id,
for better display.
WDYT?
5 years, 7 months
