3:42 a.m.
On Mon, Jan 9, 2017 at 10:05 AM, Gary Brown <gbrown(a)redhat.com> wrote:
----- Original Message -----
> Hello,
>
> let me start with terminology explanation:
> Span is one unit of work. It can be a method/REST endpoint invocation or
just
> any unit of work.
> List of spans form a trace which is currently tree like structure(every
span
> has its parent, except root span).
> Span captures its start and end, therefore it is possible to calculate a
> duration.
>
> By default all spans reported to the server are persisted in
elasticsearch.
>
> I think there are two different things in this email:
>
> 1. OpenTracing standard "kpi" tag. Tags are key-value pairs, so I guess
value
> would be a metric name (e.g. "user-registration": duration of user
> registration which consist of multiple service calls). I think that any
> duration is considered as KPI. Typically when users want to show
durations
> for a particular service they can use filtering by
tags(custom/standard). Is
> there really need for KPI tag?
This is about long term metric storage and analysis, rather than short
term user based filtering. So if we want to be able to identify the key
metrics that are of long term interest, it would need to either be done
through some configuration on the server, or as suggested in this proposal
by allowing the application to tag the relevant spans.
Long term storage wasn't mentioned in the original email. I ask for a
better explanation "KPI" tag. If there is a really need for it.
>
> 2. integration with h-metrics. What are the benefits for users?
Integration
> with grafana? Cannot be grafana directly connected to elastic APM is
using?
> All durations (span, trace) are stored in our backend and published to
JMS
> so an integration with other systems should be possible.
Long term storage of specific metrics. APM is collecting a large amount of
information, so it is likely that it will only be retained for a limited
time frame.
I think that tracing tool is mostly used for a real time analysis and
debugging or tool for solving performance problems and testing
"canary"/"blue-green" deployments, but long term storage can useful or
have
an access to aggregated results from older periods of time. Statistics
(average/median) of service response times. We should probably look at
zipkin ML and look if somebody was looking for this feature.
It would actually make sense to provide some kind of reports as APM does
higher level aggregations. Question is if there is an interest of this
feature. I'm not very familiar with grafana but from what I read it can
connect to elasticsearch so maybe it can be used with APM to show dasboards
and reports.
Regards
Gary
>
>
> Regards,
>
>
> On Sat, Jan 7, 2017 at 5:09 PM, John Sanda < jsanda(a)redhat.com > wrote:
>
>
> I want to make sure I understand the difference between trace and span. A
> trace captures the call between two services whether those are REST
> endpoints, EJBs, etc. Let’s say we want a trace of A calling B. Within
that
> A calls C which in turn calls D, etc. and then finally B is called.
Would a
> span measure the duration of the call between C and D?
>
> Are all spans captured/recorded by default?
>
> Would this be optional functionality? I am wondering because it obviously
> requires having Hawkular Metrics running which also means Cassandra.
>
> Under what tenant would you store these metrics?
>
> > On Jan 7, 2017, at 7:36 AM, Gary Brown < gbrown(a)redhat.com > wrote:
> >
> > Hi
> >
> > Wanted to discuss a proposal for recording some metric data captured
from
> > Hawkular APM in Hawkular Metrics.
> >
> > For those not familiar with Hawkular APM, it captures the end to end
trace
> > instance (think of it as a distributed call stack), for each
invocation of
> > an application. This trace can include information about the
> > communications between services, but can also include details about
> > internal components used within the services (e.g. EJBs, database
calls,
> > etc).
> >
> > First point is that if we were to record duration metrics for each
'span'
> > captured (i.e. scope within which a task is performed), for each
> > invocation of an application, then it would result in a large amount of
> > data that may be of no interest. So we need to find a way for end
> > users/developers to express which key points within an application
they do
> > want recorded as metrics.
> >
> > The proposal is to allow the application/services to define a
tag/property
> > on the spans of interest, e.g. 'kpi', that would indicate to the
server
> > that the duration value for the span should be stored in H-Metrics. The
> > value for the tag should define the name/description of the KPI.
> >
> > If considered a suitable solution, then we can also propose it as a
> > standard tag in the OpenTracing standard.
> >
> > There are a couple of metrics that we could record by default, first
being
> > the trace instance completion time, and the second possibly being the
> > individual service response times (although this could potentially
also be
> > governed by the 'kpi' tag).
> >
> > Thoughts?
> >
> > Regards
> > Gary
> >
> >
> > _______________________________________________
> > hawkular-dev mailing list
> > hawkular-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
>
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
>
>
> --
> Pavol Loffay
> Cell: +421948286055
> Red Hat
> Purkyňova 111 TPB-B
> 612 45 Brno
>
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
--
Pavol Loffay
Cell: +421948286055
Red Hat
Purkyňova 111 TPB-B
612 45 Brno