----- Original Message -----
On May 8, 2017, at 9:05 AM, Gary Brown < gbrown@redhat.com > wrote:
Hi John
----- Original Message -----
In Prometheus metric names do not have to unique, but I believe the
combination of tags and the name must be unique. You cannot change the tags.
Doing so creates a new time series.
Ok, so what is being suggested would make sense, in that there would be a
different timeseries for each combination of name and tags, e.g.
jaeger-rpc.requests.myservice.fooendpoint.err.5xx.http.inbound - would be
one possibility.
Is it then possible in h-metrics to aggregate across subgroups - e.g.
jaeger-rpc.requests.myservice.fooendpoint.err.*.*.inbound - representing all
types of inbound errors? or jaeger-rpc.requests.myservice.err - representing
all failures on that service?
Absolutely. Let’s say you tag each metric with request_type = “inbound
error”. You do a GET
/hawkular/metrics/gauges/stats?tags=request_type%3Dinbound%20error
In Hawkular Metrics the metric name must
be unique for a given metric type and within a single tenant. For example,
you cannot have two jaeger-rcp.requests counters living within the same
tenant.
Not sure I understand - how would you get more than one? If for example there
are two instances of that service running in different containers, and each
reporting metrics for jaeger-rpc.requests.myservice.fooendpoint , is that
ok? Will they be accumulated at the server?
If you use the same metric name for both, then data points will be stored in
the same time series.
Ok so for a particular service/operation metric X - if we want to be able to capture that metric individually for each service instance, we would also need to include the location in the set of tags? And then if we wanted to get an aggregated value across all of the instances, we would just specify the service and operation in the tags?Assuming my understanding is correct, we just need to get the hostname/ipaddr added to the list of tags they list?
If you want to capture the metric individually for each service instance, then you need to make sure that each corresponding metric has a unique name. If the location of each service is a unique hostname, then include that in the metric name. You will also want to include it in the metric tags. With the new tag query language you can do stuff like:
hostname = hostA
hostname IN [hostA, hostB, hostC]
hostname NOT IN [hostB, hostC, hostD]
As I understand things, specifying the service and the operation would return aggregated results across all instances.
RegardsGary
Regards
Gary
Metric tags are mutable in Hawkular Metrics. Changing the tags will
not result in a new time series.
On May 8, 2017, at 5:47 AM, Gary Brown < gbrown@redhat.com > wrote:
Hi Metrics Experts!
Re:
https://github.com/uber/jaeger-client-java/issues/172#issuecomment-299723621
This issue is concerned with supporting Prometheus endpoints within the
Jaeger instrumented client applications, to capture Jaeger related metrics
(e.g. number of spans reported/sampled/dropped, etc) but also application
metrics - i.e. number of requests, errors and latency(duration) for
different services/operations (endpoints).
As we will be interested in capturing and analysing these metrics within
Hawkular Metrics, would be good if someone with relevant experience could
get involved in the discussion to ensure the metrics are reported in the
most appropriate way.
For example - is it a good idea to have generic metrics names
(jaeger-rpc.requests - which I assume is a counter, and
jaeger-rpc.latency), or a metric name per endpoint - e.g. I was thinking
service.operation.direction?
Based on the referenced comment, I'm not sure how the tags would relate to
the metric names - I thought the tags needed to be constant for a
particular metric name, but it might be my misunderstanding of what they
are proposing.
Would be good to discuss - possibly here first and then when better
understood make a proposal on the github issue.
Regards
Gary
_______________________________________________
hawkular-dev mailing list
hawkular-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
_______________________________________________
hawkular-dev mailing list
hawkular-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
_______________________________________________
hawkular-dev mailing list
hawkular-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
_______________________________________________
hawkular-dev mailing list
hawkular-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
_______________________________________________hawkular-dev mailing listhawkular-dev@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/hawkular-dev