On May 8, 2017, at 5:47 AM, Gary Brown <gbrown(a)redhat.com&gt; wrote: Hi Metrics Experts! Re: https://github.com/uber/jaeger-client-java/issues/172#issuecomment-299723621 This issue is concerned with supporting Prometheus endpoints within the Jaeger instrumented client applications, to capture Jaeger related metrics (e.g. number of spans reported/sampled/dropped, etc) but also application metrics - i.e. number of requests, errors and latency(duration) for different services/operations (endpoints). As we will be interested in capturing and analysing these metrics within Hawkular Metrics, would be good if someone with relevant experience could get involved in the discussion to ensure the metrics are reported in the most appropriate way. For example - is it a good idea to have generic metrics names (jaeger-rpc.requests - which I assume is a counter, and jaeger-rpc.latency), or a metric name per endpoint - e.g. I was thinking service.operation.direction? Based on the referenced comment, I'm not sure how the tags would relate to the metric names - I thought the tags needed to be constant for a particular metric name, but it might be my misunderstanding of what they are proposing. Would be good to discuss - possibly here first and then when better understood make a proposal on the github issue. Regards Gary _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev

On May 8, 2017, at 9:05 AM, Gary Brown <gbrown(a)redhat.com&gt; wrote: Hi John ----- Original Message ----- > In Prometheus metric names do not have to unique, but I believe the > combination of tags and the name must be unique. You cannot change the tags. > Doing so creates a new time series. Ok, so what is being suggested would make sense, in that there would be a different timeseries for each combination of name and tags, e.g. jaeger-rpc.requests.myservice.fooendpoint.err.5xx.http.inbound - would be one possibility. Is it then possible in h-metrics to aggregate across subgroups - e.g. jaeger-rpc.requests.myservice.fooendpoint.err.*.*.inbound - representing all types of inbound errors? or jaeger-rpc.requests.myservice.err - representing all failures on that service?

> In Hawkular Metrics the metric name must > be unique for a given metric type and within a single tenant. For example, > you cannot have two jaeger-rcp.requests counters living within the same > tenant. Not sure I understand - how would you get more than one? If for example there are two instances of that service running in different containers, and each reporting metrics for jaeger-rpc.requests.myservice.fooendpoint , is that ok? Will they be accumulated at the server?

Regards Gary > Metric tags are mutable in Hawkular Metrics. Changing the tags will > not result in a new time series. > >> On May 8, 2017, at 5:47 AM, Gary Brown <gbrown(a)redhat.com&gt; wrote: >> >> Hi Metrics Experts! >> >> Re: >> https://github.com/uber/jaeger-client-java/issues/172#issuecomment-299723621 >> >> This issue is concerned with supporting Prometheus endpoints within the >> Jaeger instrumented client applications, to capture Jaeger related metrics >> (e.g. number of spans reported/sampled/dropped, etc) but also application >> metrics - i.e. number of requests, errors and latency(duration) for >> different services/operations (endpoints). >> >> As we will be interested in capturing and analysing these metrics within >> Hawkular Metrics, would be good if someone with relevant experience could >> get involved in the discussion to ensure the metrics are reported in the >> most appropriate way. >> >> For example - is it a good idea to have generic metrics names >> (jaeger-rpc.requests - which I assume is a counter, and >> jaeger-rpc.latency), or a metric name per endpoint - e.g. I was thinking >> service.operation.direction? >> >> Based on the referenced comment, I'm not sure how the tags would relate to >> the metric names - I thought the tags needed to be constant for a >> particular metric name, but it might be my misunderstanding of what they >> are proposing. >> >> Would be good to discuss - possibly here first and then when better >> understood make a proposal on the github issue. >> >> Regards >> Gary >> >> >> _______________________________________________ >> hawkular-dev mailing list >> hawkular-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/hawkular-dev > > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org <mailto:hawkular-dev@lists.jboss.org> https://lists.jboss.org/mailman/listinfo/hawkular-dev <https://lists.jboss.org/mailman/listinfo/hawkular-dev>

On May 8, 2017, at 12:20 PM, Gary Brown <gbrown(a)redhat.com&gt; wrote: ----- Original Message ----- > > > > > On May 8, 2017, at 9:05 AM, Gary Brown < gbrown(a)redhat.com > wrote: > > Hi John > > ----- Original Message ----- > > > In Prometheus metric names do not have to unique, but I believe the > combination of tags and the name must be unique. You cannot change the tags. > Doing so creates a new time series. > > Ok, so what is being suggested would make sense, in that there would be a > different timeseries for each combination of name and tags, e.g. > jaeger-rpc.requests.myservice.fooendpoint.err.5xx.http.inbound - would be > one possibility. > > Is it then possible in h-metrics to aggregate across subgroups - e.g. > jaeger-rpc.requests.myservice.fooendpoint.err.*.*.inbound - representing all > types of inbound errors? or jaeger-rpc.requests.myservice.err - representing > all failures on that service? > > Absolutely. Let’s say you tag each metric with request_type = “inbound > error”. You do a GET > /hawkular/metrics/gauges/stats?tags=request_type%3Dinbound%20error > > > > > > > In Hawkular Metrics the metric name must > be unique for a given metric type and within a single tenant. For example, > you cannot have two jaeger-rcp.requests counters living within the same > tenant. > > Not sure I understand - how would you get more than one? If for example there > are two instances of that service running in different containers, and each > reporting metrics for jaeger-rpc.requests.myservice.fooendpoint , is that > ok? Will they be accumulated at the server? > > If you use the same metric name for both, then data points will be stored in > the same time series. Ok so for a particular service/operation metric X - if we want to be able to capture that metric individually for each service instance, we would also need to include the location in the set of tags? And then if we wanted to get an aggregated value across all of the instances, we would just specify the service and operation in the tags? Assuming my understanding is correct, we just need to get the hostname/ipaddr added to the list of tags they list?

Regards Gary > > > > > > Regards > Gary > > > > Metric tags are mutable in Hawkular Metrics. Changing the tags will > not result in a new time series. > > > > On May 8, 2017, at 5:47 AM, Gary Brown < gbrown(a)redhat.com > wrote: > > Hi Metrics Experts! > > Re: > https://github.com/uber/jaeger-client-java/issues/172#issuecomment-299723621 > > This issue is concerned with supporting Prometheus endpoints within the > Jaeger instrumented client applications, to capture Jaeger related metrics > (e.g. number of spans reported/sampled/dropped, etc) but also application > metrics - i.e. number of requests, errors and latency(duration) for > different services/operations (endpoints). > > As we will be interested in capturing and analysing these metrics within > Hawkular Metrics, would be good if someone with relevant experience could > get involved in the discussion to ensure the metrics are reported in the > most appropriate way. > > For example - is it a good idea to have generic metrics names > (jaeger-rpc.requests - which I assume is a counter, and > jaeger-rpc.latency), or a metric name per endpoint - e.g. I was thinking > service.operation.direction? > > Based on the referenced comment, I'm not sure how the tags would relate to > the metric names - I thought the tags needed to be constant for a > particular metric name, but it might be my misunderstanding of what they > are proposing. > > Would be good to discuss - possibly here first and then when better > understood make a proposal on the github issue. > > Regards > Gary > > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev(a)lists.jboss.org <mailto:hawkular-dev@lists.jboss.org> > https://lists.jboss.org/mailman/listinfo/hawkular-dev <https://lists.jboss.org/mailman/listinfo/hawkular-dev> > _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org <mailto:hawkular-dev@lists.jboss.org> https://lists.jboss.org/mailman/listinfo/hawkular-dev <https://lists.jboss.org/mailman/listinfo/hawkular-dev>