[Hawkular-dev] Application metrics from Jaeger clients

[John Sanda]

> On May 8, 2017, at 9:05 AM, Gary Brown <gbrown at redhat.com> wrote:
> 
> Hi John
> 
> ----- Original Message -----
>> In Prometheus metric names do not have to unique, but I believe the
>> combination of tags and the name must be unique. You cannot change the tags.
>> Doing so creates a new time series.
> 
> Ok, so what is being suggested would make sense, in that there would be a different timeseries for each combination of name and tags, e.g. jaeger-rpc.requests.myservice.fooendpoint.err.5xx.http.inbound - would be one possibility.
> 
> Is it then possible in h-metrics to aggregate across subgroups - e.g. jaeger-rpc.requests.myservice.fooendpoint.err.*.*.inbound - representing all types of inbound errors? or jaeger-rpc.requests.myservice.err - representing all failures on that service?

Absolutely. Let’s say you tag each metric with request_type = “inbound error”.  You do a GET /hawkular/metrics/gauges/stats?tags=request_type%3Dinbound%20error
> 
>> In Hawkular Metrics the metric name must
>> be unique for a given metric type and within a single tenant. For example,
>> you cannot have two jaeger-rcp.requests counters living within the same
>> tenant.
> 
> Not sure I understand - how would you get more than one? If for example there are two instances of that service running in different containers, and each reporting metrics for jaeger-rpc.requests.myservice.fooendpoint , is that ok? Will they be accumulated at the server?

If you use the same metric name for both, then data points will be stored in the same time series.

> 
> Regards
> Gary
> 
>> Metric tags are mutable in Hawkular Metrics. Changing the tags will
>> not result in a new time series.
>> 
>>> On May 8, 2017, at 5:47 AM, Gary Brown <gbrown at redhat.com> wrote:
>>> 
>>> Hi Metrics Experts!
>>> 
>>> Re:
>>> https://github.com/uber/jaeger-client-java/issues/172#issuecomment-299723621
>>> 
>>> This issue is concerned with supporting Prometheus endpoints within the
>>> Jaeger instrumented client applications, to capture Jaeger related metrics
>>> (e.g. number of spans reported/sampled/dropped, etc) but also application
>>> metrics - i.e. number of requests, errors and latency(duration) for
>>> different services/operations (endpoints).
>>> 
>>> As we will be interested in capturing and analysing these metrics within
>>> Hawkular Metrics, would be good if someone with relevant experience could
>>> get involved in the discussion to ensure the metrics are reported in the
>>> most appropriate way.
>>> 
>>> For example - is it a good idea to have generic metrics names
>>> (jaeger-rpc.requests - which I assume is a counter, and
>>> jaeger-rpc.latency), or a metric name per endpoint - e.g. I was thinking
>>> service.operation.direction?
>>> 
>>> Based on the referenced comment, I'm not sure how the tags would relate to
>>> the metric names - I thought the tags needed to be constant for a
>>> particular metric name, but it might be my misunderstanding of what they
>>> are proposing.
>>> 
>>> Would be good to discuss - possibly here first and then when better
>>> understood make a proposal on the github issue.
>>> 
>>> Regards
>>> Gary
>>> 
>>> 
>>> _______________________________________________
>>> hawkular-dev mailing list
>>> hawkular-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>> 
>> 
>> _______________________________________________
>> hawkular-dev mailing list
>> hawkular-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>> 
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org <mailto:hawkular-dev at lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/hawkular-dev <https://lists.jboss.org/mailman/listinfo/hawkular-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170508/42c85492/attachment-0001.html