<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html><body>
<p>New Hawkular blog post from noreply@hawkular.org (John Mazzitelli): http://ift.tt/2vYe1WG<br><br></p>
<div class="sect1">
<h2 id="_federated_alerts">Federated Alerts</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Hawkular Alerts aims to be a federated alerting system. That is to say, it can fire alerts and send notifications that are triggered by data coming from a number of third-party external systems.</p>
</div>
<div class="paragraph">
<p>Thus, Hawkular Alerts is more than just an alerting system for use with Hawkular Metrics. In fact, Hawkular Alerts can be used independently of Hawkular Metrics. This means you do not even have to be using Hawkular Metrics to take advantage of the functionality provided by Hawkular Alerts.</p>
</div>
<div class="paragraph">
<p>This is a key differentiator between Hawkular Alerts and other alerting systems. Most alerting systems only alert on data coming from their respective storage systems (e.g. the Prometheus Alert Engine alerts only on Prometheus data). Hawkular Alerts, on the other hand, can trigger alerts based on data from various systems.</p>
</div>
<div class="sect2">
<h3 id="_alerts_vs_events">Alerts vs. Events</h3>
<div class="paragraph">
<p>Before we begin, a quick clarification is in order. When it is said that Hawkular Alerts fires an "alert" it means some data came into Hawkular Alerts that matched some conditions which triggered the creation of an alert in Hawkular Alerts backend storage (which can then trigger additional actions such as sending emails or calling a webhook). An "alert" typically refers to a problem that has been detected, and someone should take action to fix it. An alert has a lifecycle attached to it - alerts are <em>opened</em>, then <em>acknowledged</em> by some user who will hopefully fix the problem, then <em>resolved</em> when the problem can be considered closed.</p>
</div>
<div class="paragraph">
<p>However, there can be conditions that occur that do not represent problems but nevertheless are events you want recorded. There is no lifecycle associated with events and no additional actions are triggered by events, but "events" are fired by Hawkular Alerts in the same general manner as "alerts" are.</p>
</div>
<div class="paragraph">
<p>In this document, when it is said that Hawkular Alerts can fire "alerts" based on data coming from external third-party systems such as Prometheus, ElasticSearch, and Kakfa, this also means events can be fired as well as alerts. What this means is you can record any event (not just a "problem", aka "alert") that can be gleaned from this data coming from external third-party systems.</p>
</div>
<div class="paragraph">
<p>See <a href="http://ift.tt/2vqWK7R">alerting philosophy</a> for more.</p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_demo">Demo</h2>
<div class="sectionbody">
<div class="paragraph">
<p>There is a <a href="https://youtu.be/mM1mwJneKO4">recorded demo found here</a> that will illustrate what this document is describing. After you read this document, you should watch the demo to gain further clarity on what is being explained. The demo is the <code>multiple-sources</code> example which you can run yourself found <a href="http://ift.tt/2wC413g">here</a> (note: at the time of writing, this example is only found in the <a href="http://ift.tt/2vqLJ6C">next branch</a>, to be merged in master soon).</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_prometheus">Prometheus</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Hawkular Alerts can take the results of Prometheus metric queries and use the queried data for triggers that can fire alerts.</p>
</div>
<div class="paragraph">
<p>This Hawkular Alerts trigger will fire an alert (and send an email) when a Prometheus metric indicates our store’s inventory of widgets is consistently low (as defined by the Prometheus query you see in the "expression" field of the condition):</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="prettyprint highlight">
<code class="language-json" data-lang="json">"trigger":{
"id": "low-stock-prometheus-trigger",
"name": "Low Stock",
"description": "The number of widgets in stock is consistently low.",
"severity": "MEDIUM",
"enabled": true,
"tags": {
"prometheus": "Prometheus"
},
"actions":[
{
"actionPlugin": "email",
"actionId": "email-notify-owner"
}
]
},
"conditions":[
{
"type": "EXTERNAL",
"alerterId": "prometheus",
"dataId": "prometheus-dataid",
"expression": "rate(products_in_inventory{product=\"widget\"}[30s])<2"
}
]</code>
</pre>
</div>
</div>
<div class="sect2">
<h3 id="_integration_with_prometheus_alert_engine">Integration with Prometheus Alert Engine</h3>
<div class="paragraph">
<p>As a side note, though not demostrated in the example, Hawkular Alerts also has an integration with Prometheus' own Alert Engine. This means the alerts generated by Prometheus itself can be forward to Hawkular Alerts which can, in turn, be used for additional processing, perhaps for use with data that is unavailable to Prometheus that can tell Hawkular Alerts to fire other alerts. For example, Hawkular Alerts can take Prometheus alerts as input and feed it back into other conditions that trigger on the Prometheus alert along with ElasticSearch logs.</p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_elasticsearch">ElasticSearch</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Hawkular Alerts can examine logs stored in ElasticSearch and trigger alerts based on patterns that match within the ElasticSearch log messages.</p>
</div>
<div class="paragraph">
<p>This Hawkular Alerts trigger will fire an alert (and send an email) when ElasticSearch logs indicate sales are being lost due to inventory being out of stock of items (as defined by the condition which looks for a log category of "FATAL" which happens to mean a lost sale in the case of the store’s logs). Notice dampening is enabled on this trigger - this alert will only fire when the logs indicate lost sales every 3 times.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="prettyprint highlight">
<code class="language-json" data-lang="json">"trigger":{
"id": "lost-sale-elasticsearch-trigger",
"name": "Lost Sale",
"description": "A sale was lost due to inventory out of stock.",
"severity": "CRITICAL",
"enabled": true,
"tags": {
"Elasticsearch": "Localhost instance"
},
"context": {
"timestamp": "@timestamp",
"filter": "{\"match\":{\"category\":\"inventory\"}}",
"interval": "10s",
"index": "store",
"mapping": "level:category,@timestamp:ctime,message:text,category:dataId,index:tags"
},
"actions":[
{
"actionPlugin": "email",
"actionId": "email-notify-owner"
}
]
},
"dampenings": [
{
"triggerMode": "FIRING",
"type":"STRICT",
"evalTrueSetting": 3
}
],
"conditions":[
{
"type": "EVENT",
"dataId": "inventory",
"expression": "category == 'FATAL'"
}
]</code>
</pre>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_kafka">Kafka</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Hawkular Alerts can examine data retrieved from Kafka message streams and trigger alerts based that Kafka data.</p>
</div>
<div class="paragraph">
<p>This Hawkular Alerts trigger will fire an alert when data over a Kakfa topic indicates a large purchase was made to fill the store’s inventory (as defined by the condition which evaluates to true when any number over 17 is received on the Kafka topic):</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="prettyprint highlight">
<code class="language-json" data-lang="json">"trigger":{
"id": "large-inventory-purchase-kafka-trigger",
"name": "Large Inventory Purchase",
"description": "A large purchase was made to restock inventory.",
"severity": "LOW",
"enabled": true,
"tags": {
"Kafka": "Localhost instance"
},
"context": {
"topic": "store",
"kafka.bootstrap.servers": "localhost:9092",
"kafka.group.id": "hawkular-alerting"
},
"actions":[ ]
},
"conditions":[
{
"type": "THRESHOLD",
"dataId": "store",
"operator": "GT",
"threshold": 17
}
]</code>
</pre>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_but_wait_there_s_more">But, Wait! There’s More!</h2>
<div class="sectionbody">
<div class="paragraph">
<p>The above only mentions the different ways Hawkular Metrics retrieves data for use in determining what alerts to fire. What is not covered here is the fact that Hawkular Alerts can stream data in the other direction as well - Hawkular Alerts can send alert and event data <em>to</em> things like an ElasticSearch server or a Kafka broker. There are additional examples (mentioned below) that can demonstrate this capability.</p>
</div>
<div class="paragraph">
<p>The point is Hawkular Alerts should be seen as a shared, common alerting engine that can be shared for use by multiple third-party systems and can be used as both a consumer and producer - as a consumer of the data from external third-party systems (which is used to fire alerts and events) and as a producer to send notifications of alerts and events to external third-party systems.</p>
</div>
<div class="sect2">
<h3 id="_more_examples">More Examples</h3>
<div class="paragraph">
<p>Take a look at the Hawkular Alerts <a href="http://ift.tt/2m4Z8tb">examples</a> for more examples on using external systems as data to be used for triggering alerts. (note: at the time of writing, some examples are currently in the <a href="http://ift.tt/2vqwYQV">next branch</a> such as the Kafka ones).</p>
</div>
</div>
</div>
</div>
<br><br>
from Hawkular Blog
</body></html>