<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html><body>
<p>New Hawkular blog post from noreply@hawkular.org (John Mazzitelli): http://ift.tt/2xc8MUQ<br><br></p>
<div id="preamble">
<div class="sectionbody">
<div class="paragraph">
<p>Two recent blogs discuss how OpenTracing instrumentation can be used to collect application metrics:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="http://ift.tt/2rX1NbW" class="bare">http://ift.tt/2rX1NbW</a></p>
</li>
<li>
<p><a href="http://ift.tt/2vWrA6Y" class="bare">http://ift.tt/2vWrA6Y</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>A further interesting integration can be the addition of Hawkular Alerts to the environment.</p>
</div>
<div class="paragraph">
<p>As the previous <a href="http://ift.tt/2vYe1WG">blog</a> and <a href="https://www.youtube.com/watch?v=mM1mwJneKO4">demo</a> discuss, Hawkular Alerts is a generic, federated alerts system that can trigger events, alerts, and notifications from different, independent systems such as Prometheus, ElasticSearch, and Kafka.</p>
</div>
<div class="paragraph">
<p>Here we can combine the two. Let’s follow the directions for the OpenTracing demo (using the Jaeger implementation) and add Hawkular Alerts.</p>
</div>
<div class="paragraph">
<p>What this can show is OpenTracing application metrics triggering alerts when (as in this example) OpenTracing spans encounter a larger-than-expected error rates.</p>
</div>
<div class="paragraph">
<p>(Note: these instructions assume you are using Kubernetes / Minikube - see the Hawkular OpenTracing blogs linked above for more details on these instructions)</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_start_kubernetes">START KUBERNETES</h3>
<div class="paragraph">
<p>Here we start minikube giving it enough resources to run all of the pods necessary for this demo. We also start up a browser pointing to the Kubernetes dashboard, so you can follow the progress of the remaining instructions.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="prettyprint highlight">
<code>minikube start --cpus 4 --memory 8192
minikube dashboard</code>
</pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_deploy_prometheus">DEPLOY PROMETHEUS</h3>
<div class="listingblock">
<div class="content">
<pre class="prettyprint highlight">
<code>kubectl create -f http://ift.tt/2wHvySK
kubectl create -f http://ift.tt/2tgfO8q</code>
</pre>
</div>
</div>
<div class="paragraph">
<p>(Note: the last command might not work depending on your version - if you get errors, download a copy of prometheus-kubernetes.yml and edit it, changing “v1alpha1” to “v1”)</p>
</div>
</div>
<div class="sect2">
<h3 id="_deploy_jaeger">DEPLOY JAEGER</h3>
<div class="listingblock">
<div class="content">
<pre class="prettyprint highlight">
<code>kubectl create -f http://ift.tt/2tfYiRY</code>
</pre>
</div>
</div>
<div class="paragraph">
<p>The following will build and deploy the Jaeger example code that will produce the OpenTracing data for the demo:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="prettyprint highlight">
<code>mkdir -p ${HOME}/opentracing ; cd ${HOME}/opentracing
git clone git@github.com:objectiser/opentracing-prometheus-example.git
cd opentracing-prometheus-example/simple
eval $(minikube docker-env)
mvn clean install docker:build
kubectl create -f services-kubernetes.yml</code>
</pre>
</div>
</div>
<div class="paragraph">
<p>(Note: The last command might not work depending on your version - if you get errors, edit services-kubernetes.yml, changing “v1alpha1” to “v1”)</p>
</div>
</div>
<div class="sect2">
<h3 id="_deploy_hawkular_alerts_and_create_alert_trigger">DEPLOY HAWKULAR-ALERTS AND CREATE ALERT TRIGGER</h3>
<div class="paragraph">
<p>The following will deploy Hawkular Alerts and create the trigger definition that will trigger an alert when the Jaeger OpenTracing data indicates an error rate that is over 30%</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="prettyprint highlight">
<code>kubectl create -f http://ift.tt/2w86alD</code>
</pre>
</div>
</div>
<div class="paragraph">
<p>Next use <code>minikube service hawkular-alerts --url</code> to determine the Hawkular Alerts URL and point your browser to the path “/hawkular/alerts/ui” at that URL (i.e. <code><a href="http://host:port/hawkular/alerts/ui" class="bare">http://host:port/hawkular/alerts/ui</a></code>).</p>
</div>
<div class="paragraph">
<p>From the browser page running the Hawkular Alerts UI, enter a tenant name in the top right text field (“my-organization” for example) and click the “Change” button.</p>
</div>
<div class="paragraph">
<p>Navigate to the “Triggers” page (found in the left-hand nav menu).</p>
</div>
<div class="paragraph">
<p>Click the kabob menu icon at the top and select “New Trigger”.</p>
</div>
<div class="paragraph">
<p>In the text area, enter the following to define a new trigger that will trigger alerts when the Prometheus query shows that there is a 30% error rate or greater in the accountmgr or ordermgr servers:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="prettyprint highlight">
<code class="language-json" data-lang="json">{
"trigger":{
"id":"jaeger-prom-trigger",
"name":"High Error Rate",
"description":"Data indicates high error rate",
"severity":"HIGH",
"enabled":true,
"autoDisable":false,
"tags":{
"prometheus":"Test"
},
"context":{
"prometheus.url":"http://prometheus:9090"
}
},
"conditions":[
{
"type":"EXTERNAL",
"alerterId":"prometheus",
"dataId":"prometheus-test",
"expression":"(sum(increase(span_count{error=\"true\",span_kind=\"server\"}[1m])) without (pod,instance,job,namespace,endpoint,transaction,error,operation,span_kind) / sum(increase(span_count{span_kind=\"server\"}[1m])) without (pod,instance,job,namespace,endpoint,transaction,error,operation,span_kind)) > 0.3"
}
]
}</code>
</pre>
</div>
</div>
<hr>
<div id="new-trigger" class="imageblock">
<div class="content"><img src="http://ift.tt/2wHH7JY" alt="2017 09 06 new trigger"></div>
<div class="title">Figure 1: Create New Alert Trigger</div>
</div>
<div id="trigger" class="imageblock">
<div class="content"><img src="http://ift.tt/2w7V54i" alt="2017 09 06 trigger"></div>
<div class="title">Figure 2: Alert Trigger</div>
</div>
<hr>
<div class="paragraph">
<p>Now navigate back to the “Dashboard” page (again via the left-hand nav menu). From this Dashboard page, look for alerts when they are triggered. We’ll next start generating the data that will trigger these alerts.</p>
</div>
</div>
<div class="sect2">
<h3 id="_generate_some_sample_open_tracing_application_data">GENERATE SOME SAMPLE OPEN TRACING APPLICATION DATA</h3>
<div class="listingblock">
<div class="content">
<pre class="prettyprint highlight">
<code>export ORDERMGR=$(minikube service ordermgr --url)
${HOME}/opentracing/opentracing-prometheus-example/simple/genorders.sh</code>
</pre>
</div>
</div>
<div class="paragraph">
<p>Once the data starts to be collected, you will see alerts in the Hawkular Alerts UI as error rates become over 30% in the past minute (as per the Prometheus query).</p>
</div>
<hr>
<div id="dashboard" class="imageblock">
<div class="content"><img src="http://ift.tt/2wHmoFY" alt="2017 09 06 dashboard"></div>
<div class="title">Figure 3: Alerts Dashboard</div>
</div>
<div id="alert-list" class="imageblock">
<div class="content"><img src="http://ift.tt/2w8Geqa" alt="2017 09 06 alert list"></div>
<div class="title">Figure 4: Alert</div>
</div>
<hr>
<div class="paragraph">
<p>If you look at the alerts information in the Hawkular Alerts UI, you’ll see the conditions that triggered the alerts. For example, one such alert could look like this:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>
Time: 2017-09-01 17:41:17 -0400
External[prometheus]: prometheus-test[Event [tenantId=my-organization,
id=1a81471d-340d-4dba-abe9-5b991326dc80, ctime=1504302077288, category=prometheus,
dataId=prometheus-test, dataSource=<em>none</em>, text=[1.504302077286E9, <strong>0.3333333333333333</strong>],
context={<strong>service=ordermgr</strong>, version=<strong>0.0.1</strong>}, tags={}, trigger=null]] matches
[(sum(increase(span_count{error="true",span_kind="server"}[1m])) without
(pod,instance,job,namespace,endpoint,transaction,error,operation,span_kind) /
sum(increase(span_count{span_kind="server"}[1m])) without
(pod,instance,job,namespace,endpoint,transaction,error,operation,span_kind)) > 0.3]
</pre>
</div>
</div>
<div class="paragraph">
<p>Notice the “<strong>ordermgr</strong>” service (version "<strong>0.0.1</strong>") had an error rate of <strong>0.3333</strong> (33%) which caused the alert since it is above the allowed 30% threshold.</p>
</div>
<div class="paragraph">
<p>At this point, the Hawkular Alerts UI provides the ability for system admins to log notes about the issue, acknowledge the alert and mark the alert resolved if the underlying issue has been fixed. These lifecycle functions (also available as REST operations) are just part of the value add of Hawkular-Alerts.</p>
</div>
<div class="paragraph">
<p>You could do more complex things such as only trigger this alert if this Prometheus query generated results AND some other condition was true (say, ElasticSearch logs match a particular pattern, or if a Kafka topic had certain data). This demo merely scratches the surface, but does show how Hawkular Alerts can be used to work with OpenTracing to provide additional capabilities that may be found useful by system administrators and IT support personnel.</p>
</div>
</div>
<br><br>
from Hawkular Blog
</body></html>