[Hawkular-dev] Proposing closer integration of APM and "Hawkular metrics" on Kubernetes / OpenShift

[Heiko W.Rupp]

Hi,

right now Hawkular metrics and Hawkular APM are going
relatively separate ways. This is in part due to the backend choice,
but probably also for other reasons.

I am proposing that we try to get the two closer together because at the 
end neither tracing data alone, not classic monitoring data can answer 
all the questions like:

APM
- why is my service XY slow (my be overload of underlying CPU)
- how much disk will my service need in two years
- how much network usage did my service have yesterday

Classic montoring
- which service will fail if I pull the plug here
- what are customers buying
- why is my service slow (may be come from a dependency)

I am proposing that we integrate the two over the UI - in the first 
scenario
here the key driver is the APM UI with its trace diagrams (red boxes).
A klick on such a box will then show related metrics from the
classic monitoring.
On the level of the individual pod, both APM and Classic 
'instrumentations'
are present. For JVM-based apps this is on one side the APM agent and/or
APM instrumentation ("OT-instrumentation") (*a) On the other side the
jolokia agent/agent bond (*b)

  ![](cid:60528B4C-9911-4942-9091-D66129836840 at redhat.com 
"hosa-apm1.png")

In this first scenario, APM and classic still have separate agents and 
connections to the
backends and different backend storage.

The 2nd scenario, assumes that it is possible to use only one agent 
binary that
does both APM and classic metric export. For classic metrics, Hosa will 
poll it with
P8s metrics. And on top APM trace data will also be made available for 
grab by
Hosa, which will then forward them to the APM server.

![](cid:C4FC8BD0-2735-4CF5-9EEA-30A5B176909B at redhat.com "hosa-apm2.png")


Thoughts?
   Heiko


*a) I propose to always deploy the APM agent to get a quick and easy 
coverage
of standard scenarios, so that the user only needs explicit 
instrumentation to
increase granularity and/or to cover cases the agent can't cover.
Also "manual" instrumentation should be able to use the agent's 
connection to
talk to the APM server.

*b) I think it would make sense to always use the Prometheus protocol 
(and Hosa
may learn how to use the more efficient binary protocol) as Jolokia/http 
is JVM/Jmx
specific, while P8s exporters also exist for other environments like 
Node or Ruby

-- 
Reg. Adresse: Red Hat GmbH, Technopark II, Haus C,
Werner-von-Siemens-Ring 14, D-85630 Grasbrunn
Handelsregister: Amtsgericht München HRB 153243
Geschäftsführer: Charles Cachera, Michael Cunningham, Michael O'Neill, 
Eric Shander
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170227/e6c47959/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hosa-apm1.png
Type: image/png
Size: 59889 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170227/e6c47959/attachment-0002.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hosa-apm2.png
Type: image/png
Size: 56574 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170227/e6c47959/attachment-0003.png