12:02 p.m.
Hi,
right now Hawkular metrics and Hawkular APM are going
relatively separate ways. This is in part due to the backend choice,
but probably also for other reasons.
I am proposing that we try to get the two closer together because at the
end neither tracing data alone, not classic monitoring data can answer
all the questions like:
APM
- why is my service XY slow (my be overload of underlying CPU)
- how much disk will my service need in two years
- how much network usage did my service have yesterday
Classic montoring
- which service will fail if I pull the plug here
- what are customers buying
- why is my service slow (may be come from a dependency)
I am proposing that we integrate the two over the UI - in the first
scenario
here the key driver is the APM UI with its trace diagrams (red boxes).
A klick on such a box will then show related metrics from the
classic monitoring.
On the level of the individual pod, both APM and Classic
'instrumentations'
are present. For JVM-based apps this is on one side the APM agent and/or
APM instrumentation ("OT-instrumentation") (*a) On the other side the
jolokia agent/agent bond (*b)
In this first scenario, APM and classic still have separate agents and
connections to the
backends and different backend storage.
The 2nd scenario, assumes that it is possible to use only one agent
binary that
does both APM and classic metric export. For classic metrics, Hosa will
poll it with
P8s metrics. And on top APM trace data will also be made available for
grab by
Hosa, which will then forward them to the APM server.
Thoughts?
Heiko
*a) I propose to always deploy the APM agent to get a quick and easy
coverage
of standard scenarios, so that the user only needs explicit
instrumentation to
increase granularity and/or to cover cases the agent can't cover.
Also "manual" instrumentation should be able to use the agent's
connection to
talk to the APM server.
*b) I think it would make sense to always use the Prometheus protocol
(and Hosa
may learn how to use the more efficient binary protocol) as Jolokia/http
is JVM/Jmx
specific, while P8s exporters also exist for other environments like
Node or Ruby
--
Reg. Adresse: Red Hat GmbH, Technopark II, Haus C,
Werner-von-Siemens-Ring 14, D-85630 Grasbrunn
Handelsregister: Amtsgericht München HRB 153243
Geschäftsführer: Charles Cachera, Michael Cunningham, Michael O'Neill,
Eric Shander
4:53 p.m.
I clicked 'send' a bit too quickly :)
On 27 Feb 2017, at 12:02, Heiko W.Rupp wrote:
I am proposing that we integrate the two over the UI - in the first
scenario
here the key driver is the APM UI with its trace diagrams (red boxes).
Of course the other way of the integration should be possible
as well. If for a pod we know it hosts a service that is traced
with Hawkular APM, it should be possible to get (to) the APM-view.
For the 1st scenario I quickly looked at Grafana, but it looks like
one can only deep-link to a dashboard (or perhaps panel in a
dashboard), but not by giving the name of an individual metric to
graph. One could dynamically set up a dashboard on the fly though,
which could even be interesting to see all the 'classic metrics' of
an APM trace in one Grafana dashboard.
5:15 p.m.
It would be worth experimenting with different ideas, but we should remember that Jaeger
is being built by Uber with the aim of making sure it helps their ops team easily identify
and focus in on the cause of problems. So I think we need to build upon this work as much
as possible.
Regards
Gary
----- Original Message -----
I clicked 'send' a bit too quickly :)
On 27 Feb 2017, at 12:02, Heiko W.Rupp wrote:
> I am proposing that we integrate the two over the UI - in the first
> scenario
> here the key driver is the APM UI with its trace diagrams (red boxes).
Of course the other way of the integration should be possible
as well. If for a pod we know it hosts a service that is traced
with Hawkular APM, it should be possible to get (to) the APM-view.
For the 1st scenario I quickly looked at Grafana, but it looks like
one can only deep-link to a dashboard (or perhaps panel in a
dashboard), but not by giving the name of an individual metric to
graph. One could dynamically set up a dashboard on the fly though,
which could even be interesting to see all the 'classic metrics' of
an APM trace in one Grafana dashboard.
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
5:58 p.m.
*b) I think it would make sense to always use the Prometheus protocol
(and
Hosa may learn how to use the more efficient binary protocol)
If you are referring to the Prometheus binary exposition format [1] (aka content type of
"application/vnd.google.protobuf; proto=io.prometheus.client.MetricFamily;
encoding=delimited" as opposed to "text/plain"), HOSA already supports this
- it can read any Prometheus endpoint that exposes metric data in either the text or
binary format.
See the source code [2] and a test [3] that shows this working.
[1] https://prometheus.io/docs/instrumenting/exposition_formats/
[2]
https://github.com/hawkular/hawkular-openshift-agent/blob/master/promethe...
[3]
https://github.com/hawkular/hawkular-openshift-agent/blob/master/promethe...
12:23 a.m.
From the outside looking in, this proposal seems to assume
monitoring/metrics are similar enough to APM traces that they can and should converge in a
single collection system. Whereas I would have expected the proposal was to combine the in
the presentation/UI *after* collection. Am I misunderstanding? Because I would have
thought the characteristics of these systems would be unique enough to stay separate, e.g.
the traffic rates for metrics vs traces would be different, the ‘schema’ of what needs to
be forwarded and stored is different, and for sure the APM traces need to be correlated
based on their trace IDs and otherwise filtered based on somewhat arbitrary tags supplied
in the instrumentation.
On Feb 27, 2017, at 8:58 AM, John Mazzitelli <mazz(a)redhat.com>
wrote:
> *b) I think it would make sense to always use the Prometheus protocol (and
> Hosa may learn how to use the more efficient binary protocol)
If you are referring to the Prometheus binary exposition format [1] (aka content type of
"application/vnd.google.protobuf; proto=io.prometheus.client.MetricFamily;
encoding=delimited" as opposed to "text/plain"), HOSA already supports this
- it can read any Prometheus endpoint that exposes metric data in either the text or
binary format.
See the source code [2] and a test [3] that shows this working.
[1] https://prometheus.io/docs/instrumenting/exposition_formats/
[2]
https://github.com/hawkular/hawkular-openshift-agent/blob/master/promethe...
[3]
https://github.com/hawkular/hawkular-openshift-agent/blob/master/promethe...
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
9:24 a.m.
Agree - the most important aspect of the proposal is the ability for a user to navigate
between the two types of information in the UI.
Having a common collection mechanism may simplify the infrastructure, but in this respect
the greater benefit would be convergence on Cassandra at the backend - which is something
we (APM team) will be investigating.
Regards
Gary
----- Original Message -----
From the outside looking in, this proposal seems to assume
monitoring/metrics
are similar enough to APM traces that they can and should converge in a
single collection system. Whereas I would have expected the proposal was to
combine the in the presentation/UI *after* collection. Am I
misunderstanding? Because I would have thought the characteristics of these
systems would be unique enough to stay separate, e.g. the traffic rates for
metrics vs traces would be different, the ‘schema’ of what needs to be
forwarded and stored is different, and for sure the APM traces need to be
correlated based on their trace IDs and otherwise filtered based on somewhat
arbitrary tags supplied in the instrumentation.
> On Feb 27, 2017, at 8:58 AM, John Mazzitelli <mazz(a)redhat.com> wrote:
>
>> *b) I think it would make sense to always use the Prometheus protocol (and
>> Hosa may learn how to use the more efficient binary protocol)
>
> If you are referring to the Prometheus binary exposition format [1] (aka
> content type of "application/vnd.google.protobuf;
> proto=io.prometheus.client.MetricFamily; encoding=delimited" as opposed to
> "text/plain"), HOSA already supports this - it can read any Prometheus
> endpoint that exposes metric data in either the text or binary format.
>
> See the source code [2] and a test [3] that shows this working.
>
> [1] https://prometheus.io/docs/instrumenting/exposition_formats/
> [2]
>
https://github.com/hawkular/hawkular-openshift-agent/blob/master/promethe...
> [3]
>
https://github.com/hawkular/hawkular-openshift-agent/blob/master/promethe...
>
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
3:35 p.m.
On Feb 28, 2017, at 3:24 AM, Gary Brown <gbrown(a)redhat.com>
wrote:
Agree - the most important aspect of the proposal is the ability for a user to navigate
between the two types of information in the UI.
Having a common collection mechanism may simplify the infrastructure, but in this respect
the greater benefit would be convergence on Cassandra at the backend - which is something
we (APM team) will be investigating.
I had spent some time not too long ago trying to get familiar enough with the APM code
base so that I could propose and hopefully do a small PoC with a Cassandra backend. I did
not make much progress and got busy with other tasks, but I do think it is very doable. I
would love to help out to the extent that I can. Please keep me in the loop as efforts on
this start to progress.
10:41 a.m.
On 28 Feb 2017, at 0:23, neil.okamoto+hawkular(a)gmail.com wrote:
From the outside looking in, this proposal seems to assume
monitoring/metrics are similar enough to APM traces that they can and
should converge in a single collection system. Whereas
Scenario 2 may be unclear here. I did not try to imply that this
may be the same collection system or that they are close enough,
but rather the same "forwarding" system.
Right now one would e.g. deploy Jolokia + DropWizard Metrics
to expose the classic metrics and also the APM agent for the
traces.
This means double configuration, potentially duplicated code
and a lot more places for errors.
I would have expected the proposal was to combine the in the
presentation/UI *after* collection. Am I misunderstanding?
No. And this was the main focus of the 1st scenario.
Because I would have thought the characteristics of these systems
would be unique enough to stay separate, e.g. the traffic rates for
metrics vs traces would be different, the ‘schema’ of what needs
to be forwarded and stored is different, and for sure the APM traces
need to be correlated based on their trace IDs and otherwise filtered
based on somewhat arbitrary tags supplied in the instrumentation.
Absolutely.
And still when you have a trace like the following
one wants to find out why it took 800ms and where the difference on
600ms was spent. Here looking at data from the underlying pod or even
node may be very helpful. Which is where the classical monitoring data
comes into play.
Similarly if you have a service scaled to 3 instances and you get a
larger variation in request time, the pod information may help you
pinpoint the variations
2614
days inactive
2615
days old
7 comments
5 participants
participants (5)
-
Gary Brown -
Heiko W.Rupp -
John Mazzitelli -
John Sanda -
neil.okamoto+hawkular＠gmail.com