<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html><body>
<p>New Hawkular blog post from noreply@hawkular.org (Gary Brown): http://ift.tt/2rX1NbW<br><br></p>
<div id="preamble">
<div class="sectionbody">
<div class="paragraph">
<p>This article will show how <a href="http://opentracing.io/">OpenTracing</a> instrumentation can be used to collect Application Metrics, in addition to (but independent from) reported tracing data, from services deployed within <a href="https://kubernetes.io/">Kubernetes</a>. These Application Metrics can then be displayed in your monitoring dashboard and used to trigger alerts.</p>
</div>
<div class="imageblock">
<div class="content"><img src="http://ift.tt/2rWEoHP" alt="2017 06 26 grafana error ratio"></div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_the_example_application">The example application</h2>
<div class="sectionbody">
<div class="paragraph">
<p>In a recent article we showed how a Spring Boot application could easily be <a href="http://ift.tt/2soBcsg">instrumented using OpenTracing</a>.</p>
</div>
<div class="paragraph">
<p>The <a href="http://ift.tt/2rX5MoK">example</a> we are going to use in this article uses the same approach to create two services, <em>ordermgr</em> and <em>accountmgr</em>.</p>
</div>
<div class="paragraph">
<p><em>accountmgr</em> presents a single REST endpoint (<code>/getAccount</code>) for internal use by <em>ordermgr</em>. The code for this endpoint is:</p>
</div>
<div class="listingblock">
<div class="title">Account Managers’s Controller:</div>
<div class="content">
<pre>
@RequestMapping("/account")
public String getAccount() throws InterruptedException {
Thread.sleep(1 + (long)(Math.random()*500)); <b>(1)</b>
if (Math.random() > 0.8) { <b>(2)</b>
throw new RuntimeException("Failed to find account");
}
return "Account details";
}
</pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><b>1</b></td>
<td>This line simply introduces a random delay, to make the collected metrics more interesting.</td>
</tr>
<tr>
<td><b>2</b></td>
<td>These three lines randomly cause an exception which will result in the span (associated with the REST endpoint invocation) being tagged as an error with associated log events identifying the error details.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p><em>ordermgr</em> presents three REST endpoints for use by an end user. These are:</p>
</div>
<div class="listingblock">
<div class="title">Order Manager’s Controller:</div>
<div class="content">
<pre>
@Autowired
private io.opentracing.Tracer tracer; <b>(1)</b>
@RequestMapping("/buy")
public String buy() throws InterruptedException {
Thread.sleep(1 + (long)(Math.random()*500)); <b>(2)</b>
tracer.activeSpan().setBaggageItem("transaction", "buy"); <b>(3)</b>
ResponseEntity<String> response = restTemplate.getForEntity(accountMgrUrl + "/account", String.class);
return "BUY + " + response.getBody();
}
@RequestMapping("/sell")
public String sell() throws InterruptedException {
Thread.sleep(1 + (long)(Math.random()*500)); <b>(2)</b>
tracer.activeSpan().setBaggageItem("transaction", "sell"); <b>(3)</b>
ResponseEntity<String> response = restTemplate.getForEntity(accountMgrUrl + "/account", String.class);
return "SELL + " + response.getBody();
}
@RequestMapping("/fail")
public String fail() throws InterruptedException {
Thread.sleep(1 + (long)(Math.random()*500)); <b>(2)</b>
ResponseEntity<String> response = restTemplate.getForEntity(accountMgrUrl + "/missing", String.class); <b>(4)</b>
return "FAIL + " + response.getBody();
}
</pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><b>1</b></td>
<td>The service injects the OpenTracing <code>Tracer</code> to enable access to the active span.</td>
</tr>
<tr>
<td><b>2</b></td>
<td>All three methods introduce a random delay.</td>
</tr>
<tr>
<td><b>3</b></td>
<td>The <code>buy</code> and <code>sell</code> methods additionally set a baggage item <code>transaction</code> with the name of the business transaction being performed (i.e. buy or sell). For those not familiar with OpenTracing, the <a href="http://ift.tt/2tfPHi1">baggage concept</a> allows information to be carried <em>in band</em> with the trace context between invoked services. We will show you how a baggage item can be used to isolate the metrics relevant only for a particular business transaction.</td>
</tr>
<tr>
<td><b>4</b></td>
<td>Invoking a non-existent endpoint on <em>accountmgr</em> will lead to an error being reported in the trace and metric data.</td>
</tr>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_adding_metrics_reporting_to_the_opentracing_instrumentation">Adding Metrics Reporting to the OpenTracing instrumentation</h2>
<div class="sectionbody">
<div class="paragraph">
<p>The OpenTracing API defines the concept of a <em>Span</em> which represents a unit of work performed by a service, e.g. to receive a service invocation, perform some internal task (e.g. accessing a database) or invoking an external service. They provide an ideal basis upon which to report metrics (count and duration) regarding these points within a service.</p>
</div>
<div class="paragraph">
<p>Therefore a new <a href="http://ift.tt/2rWUNvF">OpenTracing contrib project</a> has been established (initially just for Java) to intercept the finished spans, and create the relevant metrics. These metrics are then submitted to a <em>MetricsReporter</em> for recording - the initial implementation of this interface being for <a href="https://prometheus.io/">Prometheus</a>.</p>
</div>
<div class="paragraph">
<p>The first step is to expose an endpoint for collecting the Prometheus metrics. Each service has the following configuration:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>
@Configuration
@ConditionalOnClass(CollectorRegistry.class)
public class PrometheusConfiguration {
@Bean
@ConditionalOnMissingBean
CollectorRegistry metricRegistry() {
return CollectorRegistry.defaultRegistry;
}
@Bean
ServletRegistrationBean registerPrometheusExporterServlet(CollectorRegistry metricRegistry) {
return new ServletRegistrationBean(new MetricsServlet(metricRegistry), "/metrics");
}
}
</pre>
</div>
</div>
<div class="paragraph">
<p>This will allow the Prometheus metrics to be obtained from the service’s <code>/metrics</code> REST endpoint.</p>
</div>
<div class="paragraph">
<p>Each service then requires a configuration to obtain the <code>io.opentracing.Tracer</code>:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>
@Configuration
public class TracerConfiguration implements javax.servlet.ServletContextListener {
@Bean
public io.opentracing.Tracer tracer() {
return io.opentracing.contrib.metrics.Metrics.decorate(
io.opentracing.contrib.tracerresolver.TracerResolver.resolveTracer(),
PrometheusMetricsReporter.newMetricsReporter()
.withBaggageLabel("transaction","n/a")
.build());
}
@Override
public void contextInitialized(javax.servlet.ServletContextEvent sce) {
sce.getServletContext().setAttribute(io.opentracing.contrib.web.servlet.filter.TracingFilter.SKIP_PATTERN, Pattern.compile("/metrics"));
}
...
</pre>
</div>
</div>
<div class="paragraph">
<p>The first method uses the <a href="http://ift.tt/2tfWg4j">TracerResolver</a> to provide a vendor neutral approach for accessing a <code>Tracer</code>. This tracer is then enhanced with the metrics capability using a <code>PrometheusMetricsReporter</code>. This metrics reporter is further configured to add a special label related to the baggage key <code>transaction</code> (discussed later).</p>
</div>
<div class="paragraph">
<p>By default, the Servlet OpenTracing integration will trace all REST endpoints. Therefore in the second method above we add an attribute that will inform the instrumentation to ignore the <code>/metrics</code> endpoint. Otherwise we will have tracing data reported each time Prometheus reads the metrics for the service.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_deploying_on_kubernetes">Deploying on Kubernetes</h2>
<div class="sectionbody">
<div class="paragraph">
<p>The steps to set up an environment on Kubernetes is discussed in the <a href="http://ift.tt/2rX5MoK">example codebase</a>. A summary of the steps is:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Start <a href="http://ift.tt/2rXa6EP">minikube</a></p>
<div class="listingblock">
<div class="content">
<pre class="prettyprint highlight">
<code>minikube start
minikube dashboard</code>
</pre>
</div>
</div>
</li>
<li>
<p>Deploy Prometheus - using the <a href="http://ift.tt/2f5vLEC">Prometheus Operator</a> project to capture metrics from the services</p>
<div class="listingblock">
<div class="content">
<pre class="prettyprint highlight">
<code>kubectl create -f http://ift.tt/2rXa67N
# Wait until pods are green, then add configuration to locate service monitors based on label "team: frontend":
kubectl create -f http://ift.tt/2tgfO8q
# Wait until these pods are green, then get the URL from the following command and open in browser:
minikube service prometheus --url</code>
</pre>
</div>
</div>
</li>
<li>
<p>Deploy <a href="http://ift.tt/2eOSqHE">Jaeger</a> - an OpenTracing compatible tracing system</p>
<div class="listingblock">
<div class="content">
<pre class="prettyprint highlight">
<code>kubectl create -f http://ift.tt/2tfYiRY
# Once pods are green, then get the Jaeger dashboard URL from the following command and open in a browser
minikube service jaeger-all-in-one --url</code>
</pre>
</div>
</div>
</li>
<li>
<p>For this article, we also <a href="http://ift.tt/2rWOzMr">deployed Grafana</a> to display the metrics, although the Prometheus dashboard could be used. Once Grafana is installed:</p>
<div class="ulist">
<ul>
<li>
<p>Obtain the Prometheus server URL using <code>minikube service prometheus --url</code></p>
</li>
<li>
<p>Configure a new Datasource named <em>Prometheus</em> of type <code>Prometheus</code> and specify the URL obtained from the previous command</p>
</li>
<li>
<p>Download the example dashboard using the following command and import it into Grafana</p>
<div class="listingblock">
<div class="content">
<pre class="prettyprint highlight">
<code>wget http://ift.tt/2tfRq7b</code>
</pre>
</div>
</div>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Once they are all running, then the simple example with the two services can be deployed. For this you will need to clone the <a href="http://ift.tt/2rX5MoK">example code repo</a>, and follow <a href="http://ift.tt/2rXfxmW">these instructions</a>.</p>
</div>
<div class="paragraph">
<p>At this stage the Kubernetes dashboard would look like this:</p>
</div>
<div class="imageblock">
<div class="content"><img src="http://ift.tt/2tfMm2o" alt="2017 06 26 kubernetes dashboard"></div>
<div class="title">Figure 1: Kubernetes dashboard</div>
</div>
<div class="paragraph">
<p>The example code includes a script that loops, randomly invoking the three REST endpoints provided by <em>ordermgr</em>. Once some example requests have been created, you can view the tracing dashboard:</p>
</div>
<div class="imageblock">
<div class="content"><img src="http://www.hawkular.org//img/blog/2017/2017-06-26-traces.png" alt="2017 06 26 traces"></div>
<div class="title">Figure 2: Jaeger tracing dashboard</div>
</div>
<div class="paragraph">
<p>Then you can select a specific trace instance and see further details:</p>
</div>
<div class="imageblock">
<div class="content"><img src="http://ift.tt/2rX1luo" alt="2017 06 26 trace"></div>
<div class="title">Figure 3: Jaeger trace instance view</div>
</div>
<div class="paragraph">
<p>This shows that the trace instance has three spans, the first representing the receipt of the <code>/buy</code> request on <em>ordermgr</em>, the second where <em>ordermgr</em> is invoking <em>accountmgr</em>, and finally the <em>accountmgr</em> receiving the <code>/hello</code> request. In this particular trace instance, the <em>accountmgr</em> invocation has reported an error, indicated by the <code>error=true</code> tag.</p>
</div>
<div class="paragraph">
<p>Now we will look at the Grafana dashboard to see what metrics have been reported from the OpenTracing instrumentation within the two services:</p>
</div>
<div class="imageblock">
<div class="content"><img src="http://ift.tt/2tfK5Ei" alt="2017 06 26 grafana dashboard"></div>
<div class="title">Figure 4: Grafana dashboard</div>
</div>
<div class="paragraph">
<p>This dashboard includes three graphs, the first showing the number of spans created (i.e. span count) by our <code>sell()</code> method, and we can use it to track how many times this business operation has been executed. The second showing the average duration of the spans, and third showing the ratio between successful and erronous spans.</p>
</div>
<div class="paragraph">
<p>The metrics reported by Prometheus are based on a range of labels - a metric exists for each unique combination of those labels.</p>
</div>
<div class="paragraph">
<p>The standard labels included with the OpenTracing java-metrics project are: <code>operation</code>, <code>span.kind</code> and <code>error</code>.</p>
</div>
<div class="paragraph">
<p>With this particular example, we also included the <code>transaction</code> label.</p>
</div>
<div class="paragraph">
<p>However when the services are deployed to Kubernetes, the following additional labels are included for free: <code>pod</code>, <code>instance</code>, <code>service</code>, <code>job</code> and <code>namespace</code>.</p>
</div>
<div class="paragraph">
<p>In our example Prometheus queries, we have ignored most of the Kubernetes added labels (except <code>service</code>) so that the metrics are aggregated across the specific pods, namespaces, etc. However, having these labels available means it is possible to segment the metrics in whatever way is required to analyse the data.</p>
</div>
<div class="paragraph">
<p>When using the <code>java-metrics</code> project outside of Kubernetes, it is still possible to include the <code>service</code> label, however you would configure this when setting up the tracer.</p>
</div>
<div class="paragraph">
<p>We can also filter the data, to focus on specific areas of interest:</p>
</div>
<div class="imageblock">
<div class="content"><img src="http://ift.tt/2rWLBr8" alt="2017 06 26 grafana txn service"></div>
<div class="title">Figure 5: Customized Grafana graph focusing on metrics for transaction 'sell' and service 'accountmgr'</div>
</div>
<div class="paragraph">
<p>In this image we have filtered the metrics based on the <code>transaction='sell'</code> and <code>service='accountmgr'</code>. This is where using the metric label based on the baggage item <code>transaction</code> can be useful, to understand the usage of a particular shared service by a business transaction. With further work it would be possible to show the distribution of requests for a service across the various business transactions.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_video">Video</h2>
<div class="sectionbody">
<div class="videoblock">
<div class="content"></div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_conclusion">Conclusion</h2>
<div class="sectionbody">
<div class="paragraph">
<p>This article has shown how a service can be instrumented once (using OpenTracing) and generate both tracing and application metrics.</p>
</div>
<div class="paragraph">
<p>When deployed to a Kubernetes environment, the metrics also benefit from an additional set of labels automatically added by the infrastructure, describing the service, pod, namespace, etc. This makes it easy to isolate specific metrics of interest, or view high level aggregated metrics to gain an overview of your applications performance.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_links">Links</h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>OpenTracing: <a href="http://opentracing.io" class="bare">http://opentracing.io</a></p>
</li>
<li>
<p>Github repository with demo: <a href="http://ift.tt/2rX5MoK" class="bare">http://ift.tt/2rX5MoK</a></p>
</li>
<li>
<p>OpenTracing java metrics: <a href="http://ift.tt/2rWUNvF" class="bare">http://ift.tt/2rWUNvF</a></p>
</li>
<li>
<p>Kubernetes: <a href="https://kubernetes.io" class="bare">https://kubernetes.io</a></p>
</li>
<li>
<p>Jaeger: <a href="http://ift.tt/2eOSqHE" class="bare">http://ift.tt/2eOSqHE</a></p>
</li>
<li>
<p>Prometheus: <a href="https://prometheus.io" class="bare">https://prometheus.io</a></p>
</li>
</ul>
</div>
</div>
</div>
<br><br>
from Hawkular Blog
</body></html>