From: "John Mazzitelli" <mazz(a)redhat.com&gt; To: "Discussions around Hawkular development" <hawkular-dev(a)lists.jboss.org&gt; Sent: Friday, April 8, 2016 3:38:42 PM Subject: [Hawkular-dev] hawkular-agent, prometheus, openshift I'm trying to figure out how the Hawkular WildFly Agent needs to be enhanced to collect metrics from Prometheus (which is where a lot of Open Shift metrics are going). Here is how I originally understood the problem (which may be completely wrong): I am looking at this: https://prometheus.io/docs/querying/api/ So if Open Shift components are storing metrics in Prometheus, the Agent would need to query the data via something like: http://localhost:9090/api/v1/query?query=my_metric_name_seconds{label_one... The agent can take the data it gets and store it in Hawkular Metrics (using a different metric name and/or labels if we want). I am hoping Matt W. can clarify and if this is completely wrong, how does he see it working?

From: "John Mazzitelli" <mazz(a)redhat.com&gt; To: "Discussions around Hawkular development" <hawkular-dev(a)lists.jboss.org&gt; Sent: Friday, April 8, 2016 5:22:32 PM Subject: Re: [Hawkular-dev] hawkular-agent, prometheus, openshift > There are some endpoints used in OpenShift/Kubernetes that stores metrics > using the Prometheus protocol. These endpoint expose metrics in which > Prometheus (or something which understands the protocol) can then bring > into > their system. We are looking at potentially being able to read from these > endpoint types, not being able to read from a Prometheus server itself. Then it sounds more like we'd want to have some kind of plugin or component inside Hawkular-Metrics itself so H-Metrics can just pull metrics in using the Prometheus protocol (in addition to getting metrics pushed into it via its REST API). This could be an optional component to Hawkular-Metrics for those that don't need or want it. But it sounds like this is just another protocol to insert data into H-Metrics.

From: "John Mazzitelli" <mazz(a)redhat.com&gt; To: "Discussions around Hawkular development" <hawkular-dev(a)lists.jboss.org&gt; Sent: Friday, April 8, 2016 6:38:16 PM Subject: Re: [Hawkular-dev] hawkular-agent, prometheus, openshift > We need something to figure out where those endpoints are available from, > gather the metadata about the containers which are exposing those > endpoints, > convert the metrics from one type into something Hawkular Metrics can > understand, and then write those metrics (with metadata based tags) into > Hawkular Metrics. > > I think this needs to be part of a separate component. There is going to be > a > lot of code here dealing specifically with the OpenShift environment. > > If we are reading from a jolokia endpoint and we have an agent which knows > how to store jolokia style metrics in Hawkular-Metrics, then we should of > course try and reuse components. OK, I'll look more into the Prometheus protocol. That's the part I was missing. It sounds like essentially we are going to be pulling metrics directly from the components emitting the data (rather than pulling metrics from the Prometheus server itself which is what I originally thought). Now that I think about it, really this sounds just like JMX/Jolokia endpoints or WildFly's remote DMR interface). I just have to learn more about what these Prometheus endpoints look like and what the data looks like over that protocol.