<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html><body>
<p>New Hawkular blog post from noreply@hawkular.org (Gary Brown): http://ift.tt/2vWrA6Y<br><br></p>
<div id="preamble">
<div class="sectionbody">
<div class="paragraph">
<p>In a <a href="http://ift.tt/2rX1NbW">previous article</a> we showed how <a href="http://opentracing.io/">OpenTracing</a> instrumentation can be used to collect application metrics, in addition to (but independent from) reported tracing data, from services deployed within a cloud environment (e.g. <a href="https://kubernetes.io/">Kubernetes</a> or <a href="https://openshift.io/">OpenShift</a>).</p>
</div>
<div class="paragraph">
<p>In this article we will show how this information can be used to aid a <a href="http://ift.tt/2ie6awO">Canary deployment strategy</a> within OpenShift.</p>
</div>
<div id="error-ratio-service-version" class="imageblock">
<div class="content"><img src="http://ift.tt/2vRqafz" alt="2017 08 18 canary service compare"></div>
<div class="title">Figure 1: Error ratio per service and version</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_the_updated_example_application">The updated example application</h2>
<div class="sectionbody">
<div class="paragraph">
<p>We will be using the same <a href="http://ift.tt/2rX5MoK">example</a> as used in the previous article.</p>
</div>
<div class="paragraph">
<p>However since writing that article, the configuration of the tracer and Prometheus metrics support has been simplified. There is now no explicit configuration of either, with only some auto configuration of <code>MetricLabel</code> beans to identify some custom labels to be added to the Prometheus metrics, e.g.</p>
</div>
<div class="listingblock">
<div class="title">Metrics configuration used in both services:</div>
<div class="content">
<pre>
@Configuration
public class MetricsConfiguration {
@Bean
public MetricLabel transactionLabel() {
return new BaggageMetricLabel("transaction", "n/a"); <b>(1)</b>
}
@Bean
public MetricLabel versionLabel() {
return new ConstMetricLabel("version", System.getenv("VERSION")); <b>(2)</b>
}
}
</pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><b>1</b></td>
<td>This metric label identifies the business transaction associated with the metrics, which can be used to isolate the specific number of requests, duration and errors that occurred when the service was used within the particular business transaction</td>
</tr>
<tr>
<td><b>2</b></td>
<td>This metric label identifies the service version, which is especially useful in the Canary deployment use case being discussed in this article</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>The first step is to following the instructions in the <a href="http://ift.tt/2rX5MoK">example</a> for deploying and using the services within OpenShift.</p>
</div>
<div class="paragraph">
<p>Once the <code>./genorders.sh</code> script has been running for a while, to generate plenty of metrics for version <code>0.0.1</code> of the services, then deploy the new version of the services. This is achieved by:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>updating the versions in the <code>pom.xml</code> files, within the <code>simple/accountmgr</code> and <code>simple/ordermgr</code> folders from <code>0.0.1</code> to <code>0.0.2</code></p>
</li>
<li>
<p>re-run the <code>mvn clean install docker:build</code> command from the <code>simple</code> folder</p>
</li>
<li>
<p>deploy the canary versions of the services using the command <code>oc create -f services-canary-kubernetes.yml</code></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>As our services <code>accountmgr</code> and <code>ordermgr</code> determine the backing deployment based on the respective labels <code>app: accountmgr</code> and <code>app: ordermgr</code>, simply having a second deployment with these labels will make them serve requests in a round-robin manner.</p>
</div>
<div class="paragraph">
<p>This deployment script has been pre-configured with the <code>0.0.2</code> version, and to only start a single instance of the new version of the services. This may be desirable if you want to monitor the behaviour of the new service versions over a reasonable time period, but as we want to see results faster we will scale them up to see more activity. You can do this by expanding the deployment area for each service in the OpenShift web console and selecting the up arrow to scale up each service:</p>
</div>
<div class="imageblock">
<div class="content"><img src="http://ift.tt/2vSeYPT" alt="2017 08 18 canary scale up"></div>
<div class="title">Figure 2: Scaling up canary deployment</div>
</div>
<div class="paragraph">
<p>Now we can monitor the Prometheus dashboard, using the following query, to see the error ratio per service and version:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="prettyprint highlight">
<code>sum(increase(span_count{error="true",span_kind="server"}[1m])) without (pod,instance,job,namespace,endpoint,transaction,error,operation,span_kind) / sum(increase(span_count{span_kind="server"}[1m])) without (pod,instance,job,namespace,endpoint,transaction,error,operation,span_kind)</code>
</pre>
</div>
</div>
<div class="paragraph">
<p>The result of this query can be seen in <a href="#error-ratio-service-version">Figure 1</a> at the top of the article. This chart shows that version <code>0.0.2</code> of the <code>accountmgr</code> service has not generated any errors, while the <code>0.0.2</code> of the <code>ordermgr</code> appears to be less error prone than version <code>0.0.1</code>.</p>
</div>
<div class="paragraph">
<p>Based on these metrics, we could decide that the new versions of these services are better than the previous, and therefore update the main service deployments to use the new versions. In the OpenShift web console you can do this by clicking the three vertical dots in the upper right hand side of the deployment region and selecting <em>Edit YAML</em> from the menu. This will display an editor window where you can change the version from <code>0.0.1</code> to <code>0.0.2</code> in the YAML file.</p>
</div>
<div class="imageblock">
<div class="content"><img src="http://ift.tt/2x54s6i" alt="2017 08 18 canary service update"></div>
<div class="title">Figure 3: Update the service version</div>
</div>
<div class="paragraph">
<p>After you save the YAML configuration file, in the web console you can see the service going through a "rolling update" as OpenShift incrementally changes each service instance over to the new version.</p>
</div>
<div class="imageblock">
<div class="content"><img src="http://ift.tt/2vRiLg5" alt="2017 08 18 canary rolling update"></div>
<div class="title">Figure 4: Rolling update</div>
</div>
<div class="paragraph">
<p>After the rolling update has completed for both the <code>ordermgr</code> and <code>accountmgr</code> services, then you can scale down or completely remove the canary version of each deployment.</p>
</div>
<div class="paragraph">
<p>An alternative to performing the rolling update would simply be to name the canary version something else (i.e. specific to the version being tested), and when it comes time to switch over, simply scale down the previous deployment version. This would be more straightforward, but wouldn’t show off the cool rolling update approach in the OpenShift web console :-)</p>
</div>
<div class="imageblock">
<div class="content"><img src="http://ift.tt/2x54sDk" alt="2017 08 18 canary scale down"></div>
<div class="title">Figure 5: Scaling down canary deployment</div>
</div>
<div class="admonitionblock note">
<table><tr>
<td class="icon"></td>
<td class="content">Although we have updated both services at the same time, this is not necessary. Generally microservices would be managed by separate teams and subject to their own deployment lifecycles.</td>
</tr></table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_conclusion">Conclusion</h2>
<div class="sectionbody">
<div class="paragraph">
<p>This article has shown how application metrics, captured by instrumenting services using the OpenTracing API, can be used to support a simple Canary deployment strategy.</p>
</div>
<div class="paragraph">
<p>These metrics can similarly be used with other deployment strategies, such as <a href="http://ift.tt/MmhAZX">A/B testing</a>, which can be achieved using a <a href="http://ift.tt/2x54tam">weighted load balancing capability</a> within OpenShift.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_links">Links</h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>OpenTracing: <a href="http://opentracing.io" class="bare">http://opentracing.io</a></p>
</li>
<li>
<p>Github repository with demo: <a href="http://ift.tt/2rX5MoK" class="bare">http://ift.tt/2rX5MoK</a></p>
</li>
<li>
<p>OpenTracing java metrics: <a href="http://ift.tt/2rWUNvF" class="bare">http://ift.tt/2rWUNvF</a></p>
</li>
<li>
<p>Kubernetes: <a href="https://kubernetes.io" class="bare">https://kubernetes.io</a></p>
</li>
<li>
<p>OpenShift: <a href="https://openshift.io" class="bare">https://openshift.io</a></p>
</li>
<li>
<p>Jaeger: <a href="http://ift.tt/2eOSqHE" class="bare">http://ift.tt/2eOSqHE</a></p>
</li>
<li>
<p>Prometheus: <a href="https://prometheus.io" class="bare">https://prometheus.io</a></p>
</li>
</ul>
</div>
</div>
</div>
<br><br>
from Hawkular Blog
</body></html>