[Hawkular-dev] Business transaction support with opentracing API

[Gary Brown]

Hi

The current release of Hawkular APM has introduced support for zipkin clients, so that existing zipkin instrumented applications can now report their span data to the APM server and have it represented in the UI.

Further work is being done in this release to ensure that it works well with polyglot examples, including the Red Hat helloworld MSA example.

This means that in a JVM only environment it is possible to use the non-instrusive byteman approach to instrument the applications, but if wanting to monitor a polyglot environment, then we also can now support zipkin instrumented clients.

However this means that for those zipkin instrumented applications, it will not be possible to take advantage of the "business transaction" capabilities that are available with the non-instrusive instrumentation approach, as shown here: https://vimeo.com/167739840

Looking ahead, we are also investigating providing support for the opentracing API (http://opentracing.io/) - which will provide end users with the ability to more easily switch between backend application tracing solutions.

The purpose of this email is to look at the possible approaches that could be used to introduce business transaction support for opentracing and potentially zipkin.

1) Only support as part of our own opentracing providers

The opentracing API provides the means to record 'log' events against the spans being reported: https://github.com/opentracing/opentracing-java/blob/master/opentracing-api/src/main/java/io/opentracing/Span.java#L75

As shown in this javascript example:

http.request(opts, function (res) {
    res.setEncoding('utf8');
    res.on('error', function (err) {
        span.logEvent('request_error', err);
        span.finish();
    });
    res.on('data', function (chunk) {
        span.logEvent('data_received', chunk);
    });
    res.on('end', function(err) {
        span.logEvent('request_end', err);
        span.finish();
    });
}).end();

the log event can be used to record data - in this case from the response.

Our opentracing provider could use this mechanism to process request or response data and extract relevant business properties.

The benefit of this approach is that the API used by the application is 'standard' - they just need to ensure the relevant business data is supplied as log events, and then define a business transaction config to process that data to extract the relevant information.

2) Provide additional module used by application above the zipkin/opentracing API

This would require the application to deal with a Hawkular-APM specific client library to preprocess their business messages and then associate relevant properties with the zipkin/opentracing spans.

This approach obviously does not provide a standard API to the application for this aspect, and as it can only deal with the spans, is potentially limited in the actions it can perform - i.e. possibly only extract business properties and representing them as tags (in opentracing) or binary annotations (in zipkin).


My current preference is option 1 as it encapsulates all of the functionality in our own opentracing providers (one per language supported), and gives us a bit more flexibility in terms of what the business txn config actions can perform.

Any feedback/suggestions welcome.


Regards
Gary