From mazz at redhat.com  Fri Dec  1 17:54:52 2017
From: mazz at redhat.com (John Mazzitelli)
Date: Fri, 1 Dec 2017 17:54:52 -0500 (EST)
Subject: [Hawkular-dev] need discussion around how we are going to do domain
	support
In-Reply-To: <722629168.49381712.1512167769649.JavaMail.zimbra@redhat.com>
Message-ID: <1935475527.49398854.1512168892623.JavaMail.zimbra@redhat.com>

Spent time trying to figure out how to support WildFly domain mode with the new stuff. It's not going well. Next week I will need to have some discussions on how we want to do this.

The issues with domain mode (I'll try to be short):

1) host controllers do not emit JMX metrics for slave servers - just not implemented in WildFly. Need to go to slave servers for their own JMX metrics.
2) slave servers do not have managed interfaces available remotely (can't connect to slaves over remote management interface - so agent can't get any inventory from slaves - have to get all inventory from host controller)

This means our agent in the host controller needs to collect all inventory for both master host controller and slave servers.

But we need the slave servers to have a metrics endpoint because its only in the slaves where we can get JMX metrics (remember, host controller can't give us that, we must scrape the slave servers for the JMX metrics).

If we do not have our agent in slave server (why bother if we can't connect to it over DMR API to get inventory?), we still need to somehow get the P8s JMX Exporter installed in the slaves. But if we just use the raw jmx exporter agent, how do we tell our h-services server/P8s server about the new endpoint that needs to be scraped? And how do we get the jmx exporter yml config to install there?

So I suppose we should put our agent in a kind of "metrics only" mode in all slave servers so it can expose the JMX exporter and pull down the correct jmx exporter yml from h-services server and have it tell the server to add the scrape endpoint to p8s.

But because we aren't getting inventory from slave servers, how can our agent tell the server to add the scrape endpoint? Our current implementation only adds scrape endpoints to our P8s server when new agents go into inventory. Workaround: either have agent store a small inventory (just the agent resource itself) which triggers the new scrape endpoint addition, or add a inventory REST endpoint for the agent to call to add the new scrape endpoint manually.

OK, assume we have all this. How do we link the JMX metrics getting collected from one feed (the agent in the slave server) to the inventory from another feed (the agent in the host controller). Right now, it is assumed the inventory metadata from feed "A" is matched with metrics from the same feed. Now that we have to break that link (feed A's inventory refers to feed B's metrics) we need to figure out how to fix this.

There are other ancillary issues - like how do I get the correct metadata defined for host controller so it can match resources/metrics from the slaves. I assume that will be "implementation details."

I'm sure this sounds like gibberish, but that's how convoluted supporting domain mode is going to be.

(note: none of this attempts to solve how we are going to get MiQ UI to use this new inventory/metrics data - I have no idea if it will be trivial or be convoluted, too)

From lponce at redhat.com  Mon Dec  4 04:09:45 2017
From: lponce at redhat.com (Lucas Ponce)
Date: Mon, 4 Dec 2017 10:09:45 +0100
Subject: [Hawkular-dev] need discussion around how we are going to do
 domain support
In-Reply-To: <1935475527.49398854.1512168892623.JavaMail.zimbra@redhat.com>
References: <722629168.49381712.1512167769649.JavaMail.zimbra@redhat.com>
	<1935475527.49398854.1512168892623.JavaMail.zimbra@redhat.com>
Message-ID: <CAGUuq3XaJ7-p+oWiYdWKzO6SDfzN+YLcuTkFSZyFBawEAJr7Jw@mail.gmail.com>

On Fri, Dec 1, 2017 at 11:54 PM, John Mazzitelli <mazz at redhat.com> wrote:

> Spent time trying to figure out how to support WildFly domain mode with
> the new stuff. It's not going well. Next week I will need to have some
> discussions on how we want to do this.
>
> The issues with domain mode (I'll try to be short):
>
> 1) host controllers do not emit JMX metrics for slave servers - just not
> implemented in WildFly. Need to go to slave servers for their own JMX
> metrics.
> 2) slave servers do not have managed interfaces available remotely (can't
> connect to slaves over remote management interface - so agent can't get any
> inventory from slaves - have to get all inventory from host controller)
>
> This means our agent in the host controller needs to collect all inventory
> for both master host controller and slave servers.
>
> But we need the slave servers to have a metrics endpoint because its only
> in the slaves where we can get JMX metrics (remember, host controller can't
> give us that, we must scrape the slave servers for the JMX metrics).
>
> If we do not have our agent in slave server (why bother if we can't
> connect to it over DMR API to get inventory?), we still need to somehow get
> the P8s JMX Exporter installed in the slaves. But if we just use the raw
> jmx exporter agent, how do we tell our h-services server/P8s server about
> the new endpoint that needs to be scraped? And how do we get the jmx
> exporter yml config to install there?
>
> So I suppose we should put our agent in a kind of "metrics only" mode in
> all slave servers so it can expose the JMX exporter and pull down the
> correct jmx exporter yml from h-services server and have it tell the server
> to add the scrape endpoint to p8s.
>
> But because we aren't getting inventory from slave servers, how can our
> agent tell the server to add the scrape endpoint? Our current
> implementation only adds scrape endpoints to our P8s server when new agents
> go into inventory. Workaround: either have agent store a small inventory
> (just the agent resource itself) which triggers the new scrape endpoint
> addition, or add a inventory REST endpoint for the agent to call to add the
> new scrape endpoint manually.
>
> OK, assume we have all this. How do we link the JMX metrics getting
> collected from one feed (the agent in the slave server) to the inventory
> from another feed (the agent in the host controller). Right now, it is
> assumed the inventory metadata from feed "A" is matched with metrics from
> the same feed. Now that we have to break that link (feed A's inventory
> refers to feed B's metrics) we need to figure out how to fix this.
>
> There are other ancillary issues - like how do I get the correct metadata
> defined for host controller so it can match resources/metrics from the
> slaves. I assume that will be "implementation details."
>
> I'm sure this sounds like gibberish, but that's how convoluted supporting
> domain mode is going to be.
>

Mazz,

What about to promote the host controller agent as the responsible for
everything ?

- Host controller agent will be responsible to collect info of the domain
(slave servers) and write into the inventory (Domain, Server, etc).
- Slave server can have a "metrics-only" endpoint (perhaps same agent,
perhaps a *-domain.jar if we want to simplify things).
- Host controller agent can proxy slave metrics endpoint, so we can control
endpoints like /metrics-domain/<slave>/metrics and that is what inventory
uses to create the endpoint.
- Host controller will expose in a proxy way, the metrics endpoint for the
slaves.

This approach focus the complixity in the host controller, but let to not
have exceptions and corner use cases in the whole system, the benefit I see
is that from MiQ and Hawkular Services we can try to maintain and uniform
way.

So, as you point Host Controllers needs to know the topology of the domain,
and each slave should have a local /metrics which can be proxied from the
host controller main endpoint.
On this case, both Host Controllers and Slaves will share same feed,
because, they are using the same agent, and that won't break any design use
case.

Also, I think that this proxy has a minimal technical complexity, as the
host controller should have all info to expose that.


>
> (note: none of this attempts to solve how we are going to get MiQ UI to
> use this new inventory/metrics data - I have no idea if it will be trivial
> or be convoluted, too)
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20171204/5b123f48/attachment-0001.html 

From theute at redhat.com  Mon Dec  4 05:35:15 2017
From: theute at redhat.com (Thomas Heute)
Date: Mon, 4 Dec 2017 11:35:15 +0100
Subject: [Hawkular-dev] OpenShift Deployment
In-Reply-To: <CAPq8dBL9fwpYVi1u_c2uJ_1fT51QqbX+qs+-tbp-vbBtbmGssQ@mail.gmail.com>
References: <CAPq8dB+NBXzSEAKa019utbeJ=dhyTFiMSw_tnMJo=SdviizzTQ@mail.gmail.com>
	<CAGUuq3WTg-pLJL=jxUJyrq7Dgxz1-LeTJqzV2qTQPpNUawSHSw@mail.gmail.com>
	<CAGUuq3WTVqwpuKLzXTt9=u3dA7K2AjQvXgosERhgpVPRgfsc0A@mail.gmail.com>
	<CAPq8dBL9fwpYVi1u_c2uJ_1fT51QqbX+qs+-tbp-vbBtbmGssQ@mail.gmail.com>
Message-ID: <CA+aN=MokaPHO5xs9hHnsJ+8EeW5fRf+89VqLVGx_MM+_FFRqpA@mail.gmail.com>

+1

On Thu, Nov 30, 2017 at 2:56 PM, Matthew Wringe <mwringe at redhat.com> wrote:

> Yeah, I think we should go with the all-in-one pod approach for now. If we
> discover certain use cases that wont work properly we can re-evaluate.
>
> On Thu, Nov 30, 2017 at 5:15 AM, Lucas Ponce <lponce at redhat.com> wrote:
>
>>
>>
>> On Thu, Nov 30, 2017 at 10:55 AM, Lucas Ponce <lponce at redhat.com> wrote:
>>
>>>
>>>
>>> On Mon, Nov 27, 2017 at 4:38 PM, Matthew Wringe <mwringe at redhat.com>
>>> wrote:
>>>
>>>> With the changes that are now going to include Prometheus, how do we
>>>> want to deploy this in OpenShift?
>>>>
>>>> We can have a few options:
>>>>
>>>> ALL-IN-ONE CONTAINER
>>>> We put both Hawkular Services and Prometheus in the same container.
>>>>
>>>> Pros:
>>>> - easy to deploy in plain docker (but this doesn't appear to be a
>>>> usecase we are targetting anyways)
>>>> - shares the same network connection (even localhost) and ip address
>>>> (eg but both services are on the different ports).
>>>> - Does't require any special wiring of components.
>>>> - Can share the same volume mount
>>>> - version of components can't get out of sync.
>>>>
>>>> Cons:
>>>> - workflow doesn't work nicely. Docker containers are meant to only run
>>>> a single application and running two can cause problems. Eg lifecycle
>>>> events would become tricky and require some hacks to get around things.
>>>> - can't independently deploy things
>>>> - can't reuse or share any existing Prometheus docker containers.
>>>>
>>>> ALL-IN-ONE POD
>>>> Hawkular Services and Prometheus are in their own containers, but they
>>>> are both deployed within the same pod.
>>>>
>>>> Pros:
>>>> - shares the same network connection.
>>>> - bound to the same machine (useful if sharing the same hostpath pv)
>>>> and don' need to worry about external network configurations (eg firewalls
>>>> between OpenShift nodes)
>>>> - pvs can be shared or separate.
>>>> - lifecycle events will work properly.
>>>>
>>>> Cons:
>>>> - lifecycle hooks will mean that both containers will have to pass
>>>> before either one will enter the ready state. So if Prometheus is failing
>>>> for some reason, Hawkular Services will not be available under the service.
>>>> - cannot independently update one container. If we need to deploy a new
>>>> container we will need to bring down the whole pod.
>>>> - are stuck with a 1:1 ratio between Hawkular Services and Prometheus
>>>>
>>>>
>>> One technical requeriment is that Hawkular Services needs to now where
>>> is Prometheus server at initialization.
>>>
>>
>> One technical requeriment is that Hawkular Services needs to *know* where
>> is Prometheus server at initialization.
>>
>> [Sorry, typing fast]
>>
>> So, I guess that all-in-one pod will simplify things on this case.
>>>
>>> I would start with this architecture first and harden the basic
>>> scenarios.
>>>
>>>
>>>>
>>>> SEPARATE PODS
>>>> Hawkular Services and Prometheus have their own separate pods.
>>>>
>>>> Pros:
>>>> - can independently run components and each component has its own
>>>> separate lifecycle
>>>> - if in the future we want to cluster Hawkular Services. this will make
>>>> it a lot easier and will also allow for running an n:m ratio between
>>>> Hawkular Services and Prometheus
>>>> - probably the more 'correct' way to deploy things as we don't have a
>>>> strong requirement for Hawkular Services and Prometheus to run together.
>>>>
>>>> Cons:
>>>> - more complex wiring. We will need to have extra services and routes
>>>> created to handle this. This mean more things running and more chances for
>>>> things to go wrong. Also more things to configure
>>>> - reusing a PV between Hawkular Services and Prometheus could be more
>>>> challenging (especially if we are using hostpath pvs). Updating the
>>>> Prometheus scrape endpoint may require a new component and container.
>>>>
>>>> _______________________________________________
>>>> hawkular-dev mailing list
>>>> hawkular-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>>>>
>>>>
>>>
>>
>> _______________________________________________
>> hawkular-dev mailing list
>> hawkular-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>>
>>
>
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20171204/a236f499/attachment.html 

From mazz at redhat.com  Mon Dec  4 06:09:52 2017
From: mazz at redhat.com (John Mazzitelli)
Date: Mon, 4 Dec 2017 06:09:52 -0500 (EST)
Subject: [Hawkular-dev] need discussion around how we are going to do
 domain support
In-Reply-To: <CAGUuq3XaJ7-p+oWiYdWKzO6SDfzN+YLcuTkFSZyFBawEAJr7Jw@mail.gmail.com>
References: <722629168.49381712.1512167769649.JavaMail.zimbra@redhat.com>
	<1935475527.49398854.1512168892623.JavaMail.zimbra@redhat.com>
	<CAGUuq3XaJ7-p+oWiYdWKzO6SDfzN+YLcuTkFSZyFBawEAJr7Jw@mail.gmail.com>
Message-ID: <443296667.51267597.1512385792555.JavaMail.zimbra@redhat.com>

> > Spent time trying to figure out how to support WildFly domain mode with
> > the new stuff. It's not going well. Next week I will need to have some
> > discussions on how we want to do this.
> >
> > The issues with domain mode (I'll try to be short):
> >
> > 1) host controllers do not emit JMX metrics for slave servers - just not
> > implemented in WildFly. Need to go to slave servers for their own JMX
> > metrics.
> > 2) slave servers do not have managed interfaces available remotely (can't
> > connect to slaves over remote management interface - so agent can't get any
> > inventory from slaves - have to get all inventory from host controller)
> >
> > This means our agent in the host controller needs to collect all inventory
> > for both master host controller and slave servers.
> >
> > But we need the slave servers to have a metrics endpoint because its only
> > in the slaves where we can get JMX metrics (remember, host controller can't
> > give us that, we must scrape the slave servers for the JMX metrics).
> >
> > If we do not have our agent in slave server (why bother if we can't
> > connect to it over DMR API to get inventory?), we still need to somehow get
> > the P8s JMX Exporter installed in the slaves. But if we just use the raw
> > jmx exporter agent, how do we tell our h-services server/P8s server about
> > the new endpoint that needs to be scraped? And how do we get the jmx
> > exporter yml config to install there?
> >
> > So I suppose we should put our agent in a kind of "metrics only" mode in
> > all slave servers so it can expose the JMX exporter and pull down the
> > correct jmx exporter yml from h-services server and have it tell the server
> > to add the scrape endpoint to p8s.
> >
> > But because we aren't getting inventory from slave servers, how can our
> > agent tell the server to add the scrape endpoint? Our current
> > implementation only adds scrape endpoints to our P8s server when new agents
> > go into inventory. Workaround: either have agent store a small inventory
> > (just the agent resource itself) which triggers the new scrape endpoint
> > addition, or add a inventory REST endpoint for the agent to call to add the
> > new scrape endpoint manually.
> >
> > OK, assume we have all this. How do we link the JMX metrics getting
> > collected from one feed (the agent in the slave server) to the inventory
> > from another feed (the agent in the host controller). Right now, it is
> > assumed the inventory metadata from feed "A" is matched with metrics from
> > the same feed. Now that we have to break that link (feed A's inventory
> > refers to feed B's metrics) we need to figure out how to fix this.
> >
> > There are other ancillary issues - like how do I get the correct metadata
> > defined for host controller so it can match resources/metrics from the
> > slaves. I assume that will be "implementation details."
> >
> > I'm sure this sounds like gibberish, but that's how convoluted supporting
> > domain mode is going to be.
> >
> 
> Mazz,
> 
> What about to promote the host controller agent as the responsible for
> everything ?
> 
> - Host controller agent will be responsible to collect info of the domain
> (slave servers) and write into the inventory (Domain, Server, etc).
> - Slave server can have a "metrics-only" endpoint (perhaps same agent,
> perhaps a *-domain.jar if we want to simplify things).
> - Host controller agent can proxy slave metrics endpoint, so we can control
> endpoints like /metrics-domain/<slave>/metrics and that is what inventory
> uses to create the endpoint.
> - Host controller will expose in a proxy way, the metrics endpoint for the
> slaves.
> 
> This approach focus the complixity in the host controller, but let to not
> have exceptions and corner use cases in the whole system, the benefit I see
> is that from MiQ and Hawkular Services we can try to maintain and uniform
> way.
> 
> So, as you point Host Controllers needs to know the topology of the domain,
> and each slave should have a local /metrics which can be proxied from the
> host controller main endpoint.
> On this case, both Host Controllers and Slaves will share same feed,
> because, they are using the same agent, and that won't break any design use
> case.
> 
> Also, I think that this proxy has a minimal technical complexity, as the
> host controller should have all info to expose that.


This has some promise. Just need to think about how to configure this proxy and implement it without having to rip apart too much of the internals of the agent code.

Thinking out loud:

We add "proxies" section to the agent config's metrics-exporters:

metrics-exporter:
  enabled: true
  host: ${hawkular.agent.metrics.host:127.0.0.1}
  port: ${hawkular.agent.metrics.port:9779}
  config-dir: ${jboss.server.config.dir}
  config-file: WF10
  # HERE IS THE NEW SECTION
  proxies:
  - path: slave1/metrics
    host: 127.0.0.1
    port: 9780
  - path: slave2/metrics
    host: 127.0.0.1
    port: 9781
  - path: slave3/metrics
    host: 127.0.0.1
    port: 9782

We would have to enhance our jmx exporter wrapper code so it supports those extra endpoints (it would simply just pass-through request/responses to those different endpoints).

The slave agent configs would have their jmx exporter sections the same as always and bind to localhost:

metrics-exporter:
  enabled: true
  host: 127.0.0.1
  port: ${hawkular.agent.metrics.port:9779}
  config-dir: ${jboss.server.config.dir}
  config-file: WF10

The nice thing here is they only need to expose the metric data on the loopback address, 127.0.0.1, since the host controller agent will be on the same box - no need to open this up to the wider network.

The bad thing is the proxies section requires you to know how many slaves there are and what their ports are going to be. There is no way to know that ahead of time (this is all encoded in the host controller's config, and who knows what the user wants to use - you can start host controllers with --host-config option to specify a custom .xml). And besides, new slaves can be added to the host controller at any time.

So it seems there needs to be some kind of auto-discovery of proxies, not a fixed configuration of them. Maybe we have a section that tells the agent to check for metric endpoints to be proxied?

  proxies:
  - host: 127.0.0.1
    port-range: 9780-9880

But if we have auto-discover like that, what is the proxy /path that we need to tell Prometheus to scrape? We need to distinguish it somehow so P knows "to get slave #1, I scrape host:9779/slave1/metrics and to scrape slave #2 its host:9779/slave2/metrics the host controller its just host:9779/metrics ??

Maybe we tell the host controller agent to talk to the slave agent somehow? Over remote JMX is an interesting idea (the agent already has its own JMX MBean, as long as we can talk to it over remote JMX, we can have the host agent ask the slave agent for information about its metrics endpoint ("what is your host/port of /metrics endpoint?". We could possibly then associate it with the name of the slave and use that as the proxy path?

We could then add some kind of "flag" or metadata on the "Domain WildFly Server" resource to say, "this resource has an agent whose metrics endpoint we want to proxy - go ask this agent for its details". So when the host agent runs its normal inventory scan, we can add special code to say, "when you hit a resource that has this special proxy metadata, you need to proxy that resource's agent's metrics endpoint." Because the Domain WildFly Server is a DMR resource - there is no associated JMX Server information with it. We would need to configure that somehow.

  - name: Domain WildFly Server
    resource-name-template: "%-"
    path: "/server=*"
    parents:
    - Domain Host
    proxy-metrics: true

Oh, BTW, because the agent's remote JMX client is the jolokia client, we need to install a jolokia -javaagent in the slave server along with our agent. We'd need to add some additional information there so the host agent knows all the details so it can connect to the JMX server where the resource is hosted. How does it know the jolokia URL of the slave server?

See how this snowballs? :-/

From mazz at redhat.com  Mon Dec  4 06:44:31 2017
From: mazz at redhat.com (John Mazzitelli)
Date: Mon, 4 Dec 2017 06:44:31 -0500 (EST)
Subject: [Hawkular-dev] need discussion around how we are going to do
 domain support
In-Reply-To: <443296667.51267597.1512385792555.JavaMail.zimbra@redhat.com>
References: <722629168.49381712.1512167769649.JavaMail.zimbra@redhat.com>
	<1935475527.49398854.1512168892623.JavaMail.zimbra@redhat.com>
	<CAGUuq3XaJ7-p+oWiYdWKzO6SDfzN+YLcuTkFSZyFBawEAJr7Jw@mail.gmail.com>
	<443296667.51267597.1512385792555.JavaMail.zimbra@redhat.com>
Message-ID: <1967471492.51281903.1512387871758.JavaMail.zimbra@redhat.com>



----- Original Message -----
> > > Spent time trying to figure out how to support WildFly domain mode with
> > > the new stuff. It's not going well. Next week I will need to have some
> > > discussions on how we want to do this.
> > >
> > > The issues with domain mode (I'll try to be short):
> > >
> > > 1) host controllers do not emit JMX metrics for slave servers - just not
> > > implemented in WildFly. Need to go to slave servers for their own JMX
> > > metrics.
> > > 2) slave servers do not have managed interfaces available remotely (can't
> > > connect to slaves over remote management interface - so agent can't get
> > > any
> > > inventory from slaves - have to get all inventory from host controller)
> > >
> > > This means our agent in the host controller needs to collect all
> > > inventory
> > > for both master host controller and slave servers.
> > >
> > > But we need the slave servers to have a metrics endpoint because its only
> > > in the slaves where we can get JMX metrics (remember, host controller
> > > can't
> > > give us that, we must scrape the slave servers for the JMX metrics).
> > >
> > > If we do not have our agent in slave server (why bother if we can't
> > > connect to it over DMR API to get inventory?), we still need to somehow
> > > get
> > > the P8s JMX Exporter installed in the slaves. But if we just use the raw
> > > jmx exporter agent, how do we tell our h-services server/P8s server about
> > > the new endpoint that needs to be scraped? And how do we get the jmx
> > > exporter yml config to install there?
> > >
> > > So I suppose we should put our agent in a kind of "metrics only" mode in
> > > all slave servers so it can expose the JMX exporter and pull down the
> > > correct jmx exporter yml from h-services server and have it tell the
> > > server
> > > to add the scrape endpoint to p8s.
> > >
> > > But because we aren't getting inventory from slave servers, how can our
> > > agent tell the server to add the scrape endpoint? Our current
> > > implementation only adds scrape endpoints to our P8s server when new
> > > agents
> > > go into inventory. Workaround: either have agent store a small inventory
> > > (just the agent resource itself) which triggers the new scrape endpoint
> > > addition, or add a inventory REST endpoint for the agent to call to add
> > > the
> > > new scrape endpoint manually.
> > >
> > > OK, assume we have all this. How do we link the JMX metrics getting
> > > collected from one feed (the agent in the slave server) to the inventory
> > > from another feed (the agent in the host controller). Right now, it is
> > > assumed the inventory metadata from feed "A" is matched with metrics from
> > > the same feed. Now that we have to break that link (feed A's inventory
> > > refers to feed B's metrics) we need to figure out how to fix this.
> > >
> > > There are other ancillary issues - like how do I get the correct metadata
> > > defined for host controller so it can match resources/metrics from the
> > > slaves. I assume that will be "implementation details."
> > >
> > > I'm sure this sounds like gibberish, but that's how convoluted supporting
> > > domain mode is going to be.
> > >
> > 
> > Mazz,
> > 
> > What about to promote the host controller agent as the responsible for
> > everything ?
> > 
> > - Host controller agent will be responsible to collect info of the domain
> > (slave servers) and write into the inventory (Domain, Server, etc).
> > - Slave server can have a "metrics-only" endpoint (perhaps same agent,
> > perhaps a *-domain.jar if we want to simplify things).
> > - Host controller agent can proxy slave metrics endpoint, so we can control
> > endpoints like /metrics-domain/<slave>/metrics and that is what inventory
> > uses to create the endpoint.
> > - Host controller will expose in a proxy way, the metrics endpoint for the
> > slaves.
> > 
> > This approach focus the complixity in the host controller, but let to not
> > have exceptions and corner use cases in the whole system, the benefit I see
> > is that from MiQ and Hawkular Services we can try to maintain and uniform
> > way.
> > 
> > So, as you point Host Controllers needs to know the topology of the domain,
> > and each slave should have a local /metrics which can be proxied from the
> > host controller main endpoint.
> > On this case, both Host Controllers and Slaves will share same feed,
> > because, they are using the same agent, and that won't break any design use
> > case.
> > 
> > Also, I think that this proxy has a minimal technical complexity, as the
> > host controller should have all info to expose that.
> 
> 
> This has some promise. Just need to think about how to configure this proxy
> and implement it without having to rip apart too much of the internals of
> the agent code.
> 
> Thinking out loud:
> 
> We add "proxies" section to the agent config's metrics-exporters:
> 
> metrics-exporter:
>   enabled: true
>   host: ${hawkular.agent.metrics.host:127.0.0.1}
>   port: ${hawkular.agent.metrics.port:9779}
>   config-dir: ${jboss.server.config.dir}
>   config-file: WF10
>   # HERE IS THE NEW SECTION
>   proxies:
>   - path: slave1/metrics
>     host: 127.0.0.1
>     port: 9780
>   - path: slave2/metrics
>     host: 127.0.0.1
>     port: 9781
>   - path: slave3/metrics
>     host: 127.0.0.1
>     port: 9782
> 
> We would have to enhance our jmx exporter wrapper code so it supports those
> extra endpoints (it would simply just pass-through request/responses to
> those different endpoints).
> 
> The slave agent configs would have their jmx exporter sections the same as
> always and bind to localhost:
> 
> metrics-exporter:
>   enabled: true
>   host: 127.0.0.1
>   port: ${hawkular.agent.metrics.port:9779}
>   config-dir: ${jboss.server.config.dir}
>   config-file: WF10
> 
> The nice thing here is they only need to expose the metric data on the
> loopback address, 127.0.0.1, since the host controller agent will be on the
> same box - no need to open this up to the wider network.
> 
> The bad thing is the proxies section requires you to know how many slaves
> there are and what their ports are going to be. There is no way to know that
> ahead of time (this is all encoded in the host controller's config, and who
> knows what the user wants to use - you can start host controllers with
> --host-config option to specify a custom .xml). And besides, new slaves can
> be added to the host controller at any time.
> 
> So it seems there needs to be some kind of auto-discovery of proxies, not a
> fixed configuration of them. Maybe we have a section that tells the agent to
> check for metric endpoints to be proxied?
> 
>   proxies:
>   - host: 127.0.0.1
>     port-range: 9780-9880
> 
> But if we have auto-discover like that, what is the proxy /path that we need
> to tell Prometheus to scrape? We need to distinguish it somehow so P knows
> "to get slave #1, I scrape host:9779/slave1/metrics and to scrape slave #2
> its host:9779/slave2/metrics the host controller its just host:9779/metrics
> ??
> 
> Maybe we tell the host controller agent to talk to the slave agent somehow?
> Over remote JMX is an interesting idea (the agent already has its own JMX
> MBean, as long as we can talk to it over remote JMX, we can have the host
> agent ask the slave agent for information about its metrics endpoint ("what
> is your host/port of /metrics endpoint?". We could possibly then associate
> it with the name of the slave and use that as the proxy path?
> 
> We could then add some kind of "flag" or metadata on the "Domain WildFly
> Server" resource to say, "this resource has an agent whose metrics endpoint
> we want to proxy - go ask this agent for its details". So when the host
> agent runs its normal inventory scan, we can add special code to say, "when
> you hit a resource that has this special proxy metadata, you need to proxy
> that resource's agent's metrics endpoint." Because the Domain WildFly Server
> is a DMR resource - there is no associated JMX Server information with it.
> We would need to configure that somehow.
> 
>   - name: Domain WildFly Server
>     resource-name-template: "%-"
>     path: "/server=*"
>     parents:
>     - Domain Host
>     proxy-metrics: true
> 
> Oh, BTW, because the agent's remote JMX client is the jolokia client, we need
> to install a jolokia -javaagent in the slave server along with our agent.
> We'd need to add some additional information there so the host agent knows
> all the details so it can connect to the JMX server where the resource is
> hosted. How does it know the jolokia URL of the slave server?


Maybe we forget all this remote JMX talk from host agent to slave agent and just use the file system? We know the slaves are on the same box as the host controller - so perhaps we tell each slave, "write some information the host agent needs so it can proxy your metrics endpoint". This has the added benefit of having that auto-discovery I said we'd need. The host controller simply proxies whatever information it detects on the filesystem (maybe each slave writes a file whose name is the name of the slave (so all the files are unique) and in each file is information like the host and port where the /metrics endpoint is. Any file the host agent finds it will proxy. The agent therefore would just need to be told, "look in this directory - as you see slaves writing their proxy files, start proxying their info."

The question then becomes, how does the agent tell the h-server, "let P know that it needs to scrape this endpoint"? I think we are back to the "have the agent manually tell the h-server to tell P to register a metrics endpoint" - what is provided by this commit in my local branch: https://github.com/jmazzitelli/hawkular-commons/commit/ae8d316486be7ba738f63b666c99a4f5a2e61f60


From lponce at redhat.com  Mon Dec  4 07:10:12 2017
From: lponce at redhat.com (Lucas Ponce)
Date: Mon, 4 Dec 2017 13:10:12 +0100
Subject: [Hawkular-dev] need discussion around how we are going to do
 domain support
In-Reply-To: <443296667.51267597.1512385792555.JavaMail.zimbra@redhat.com>
References: <722629168.49381712.1512167769649.JavaMail.zimbra@redhat.com>
	<1935475527.49398854.1512168892623.JavaMail.zimbra@redhat.com>
	<CAGUuq3XaJ7-p+oWiYdWKzO6SDfzN+YLcuTkFSZyFBawEAJr7Jw@mail.gmail.com>
	<443296667.51267597.1512385792555.JavaMail.zimbra@redhat.com>
Message-ID: <CAGUuq3V9hRGZW9V0koOyTB1M6mOE+rdOCnrzsNVG=8+FkWT6Nw@mail.gmail.com>

On Mon, Dec 4, 2017 at 12:09 PM, John Mazzitelli <mazz at redhat.com> wrote:

> > > Spent time trying to figure out how to support WildFly domain mode with
> > > the new stuff. It's not going well. Next week I will need to have some
> > > discussions on how we want to do this.
> > >
> > > The issues with domain mode (I'll try to be short):
> > >
> > > 1) host controllers do not emit JMX metrics for slave servers - just
> not
> > > implemented in WildFly. Need to go to slave servers for their own JMX
> > > metrics.
> > > 2) slave servers do not have managed interfaces available remotely
> (can't
> > > connect to slaves over remote management interface - so agent can't
> get any
> > > inventory from slaves - have to get all inventory from host controller)
> > >
> > > This means our agent in the host controller needs to collect all
> inventory
> > > for both master host controller and slave servers.
> > >
> > > But we need the slave servers to have a metrics endpoint because its
> only
> > > in the slaves where we can get JMX metrics (remember, host controller
> can't
> > > give us that, we must scrape the slave servers for the JMX metrics).
> > >
> > > If we do not have our agent in slave server (why bother if we can't
> > > connect to it over DMR API to get inventory?), we still need to
> somehow get
> > > the P8s JMX Exporter installed in the slaves. But if we just use the
> raw
> > > jmx exporter agent, how do we tell our h-services server/P8s server
> about
> > > the new endpoint that needs to be scraped? And how do we get the jmx
> > > exporter yml config to install there?
> > >
> > > So I suppose we should put our agent in a kind of "metrics only" mode
> in
> > > all slave servers so it can expose the JMX exporter and pull down the
> > > correct jmx exporter yml from h-services server and have it tell the
> server
> > > to add the scrape endpoint to p8s.
> > >
> > > But because we aren't getting inventory from slave servers, how can our
> > > agent tell the server to add the scrape endpoint? Our current
> > > implementation only adds scrape endpoints to our P8s server when new
> agents
> > > go into inventory. Workaround: either have agent store a small
> inventory
> > > (just the agent resource itself) which triggers the new scrape endpoint
> > > addition, or add a inventory REST endpoint for the agent to call to
> add the
> > > new scrape endpoint manually.
> > >
> > > OK, assume we have all this. How do we link the JMX metrics getting
> > > collected from one feed (the agent in the slave server) to the
> inventory
> > > from another feed (the agent in the host controller). Right now, it is
> > > assumed the inventory metadata from feed "A" is matched with metrics
> from
> > > the same feed. Now that we have to break that link (feed A's inventory
> > > refers to feed B's metrics) we need to figure out how to fix this.
> > >
> > > There are other ancillary issues - like how do I get the correct
> metadata
> > > defined for host controller so it can match resources/metrics from the
> > > slaves. I assume that will be "implementation details."
> > >
> > > I'm sure this sounds like gibberish, but that's how convoluted
> supporting
> > > domain mode is going to be.
> > >
> >
> > Mazz,
> >
> > What about to promote the host controller agent as the responsible for
> > everything ?
> >
> > - Host controller agent will be responsible to collect info of the domain
> > (slave servers) and write into the inventory (Domain, Server, etc).
> > - Slave server can have a "metrics-only" endpoint (perhaps same agent,
> > perhaps a *-domain.jar if we want to simplify things).
> > - Host controller agent can proxy slave metrics endpoint, so we can
> control
> > endpoints like /metrics-domain/<slave>/metrics and that is what
> inventory
> > uses to create the endpoint.
> > - Host controller will expose in a proxy way, the metrics endpoint for
> the
> > slaves.
> >
> > This approach focus the complixity in the host controller, but let to not
> > have exceptions and corner use cases in the whole system, the benefit I
> see
> > is that from MiQ and Hawkular Services we can try to maintain and uniform
> > way.
> >
> > So, as you point Host Controllers needs to know the topology of the
> domain,
> > and each slave should have a local /metrics which can be proxied from the
> > host controller main endpoint.
> > On this case, both Host Controllers and Slaves will share same feed,
> > because, they are using the same agent, and that won't break any design
> use
> > case.
> >
> > Also, I think that this proxy has a minimal technical complexity, as the
> > host controller should have all info to expose that.
>
>
> This has some promise. Just need to think about how to configure this
> proxy and implement it without having to rip apart too much of the
> internals of the agent code.
>
> Thinking out loud:
>
> We add "proxies" section to the agent config's metrics-exporters:
>
> metrics-exporter:
>   enabled: true
>   host: ${hawkular.agent.metrics.host:127.0.0.1}
>   port: ${hawkular.agent.metrics.port:9779}
>   config-dir: ${jboss.server.config.dir}
>   config-file: WF10
>   # HERE IS THE NEW SECTION
>   proxies:
>   - path: slave1/metrics
>     host: 127.0.0.1
>     port: 9780
>   - path: slave2/metrics
>     host: 127.0.0.1
>     port: 9781
>   - path: slave3/metrics
>     host: 127.0.0.1
>     port: 9782
>
> We would have to enhance our jmx exporter wrapper code so it supports
> those extra endpoints (it would simply just pass-through request/responses
> to those different endpoints).
>
> The slave agent configs would have their jmx exporter sections the same as
> always and bind to localhost:
>
> metrics-exporter:
>   enabled: true
>   host: 127.0.0.1
>   port: ${hawkular.agent.metrics.port:9779}
>   config-dir: ${jboss.server.config.dir}
>   config-file: WF10
>
> The nice thing here is they only need to expose the metric data on the
> loopback address, 127.0.0.1, since the host controller agent will be on the
> same box - no need to open this up to the wider network.
>
> The bad thing is the proxies section requires you to know how many slaves
> there are and what their ports are going to be. There is no way to know
> that ahead of time (this is all encoded in the host controller's config,
> and who knows what the user wants to use - you can start host controllers
> with --host-config option to specify a custom .xml). And besides, new
> slaves can be added to the host controller at any time.
>

Wait a minute. Metrics aside, isn't this a similar problem for the pure
inventory ?

If user starts a new slave sever dynamically, HostController agent will
detect that and add it to the inventory. Or is this process works
differently ?

If that is the case, is there any metadata we can use to pass this info ?

I guess next questions are also related to this.




> So it seems there needs to be some kind of auto-discovery of proxies, not
> a fixed configuration of them. Maybe we have a section that tells the agent
> to check for metric endpoints to be proxied?
>
>   proxies:
>   - host: 127.0.0.1
>     port-range: 9780-9880
>
> But if we have auto-discover like that, what is the proxy /path that we
> need to tell Prometheus to scrape? We need to distinguish it somehow so P
> knows "to get slave #1, I scrape host:9779/slave1/metrics and to scrape
> slave #2 its host:9779/slave2/metrics the host controller its just
> host:9779/metrics ??
>
> Maybe we tell the host controller agent to talk to the slave agent
> somehow? Over remote JMX is an interesting idea (the agent already has its
> own JMX MBean, as long as we can talk to it over remote JMX, we can have
> the host agent ask the slave agent for information about its metrics
> endpoint ("what is your host/port of /metrics endpoint?". We could possibly
> then associate it with the name of the slave and use that as the proxy path?
>
> We could then add some kind of "flag" or metadata on the "Domain WildFly
> Server" resource to say, "this resource has an agent whose metrics endpoint
> we want to proxy - go ask this agent for its details". So when the host
> agent runs its normal inventory scan, we can add special code to say, "when
> you hit a resource that has this special proxy metadata, you need to proxy
> that resource's agent's metrics endpoint." Because the Domain WildFly
> Server is a DMR resource - there is no associated JMX Server information
> with it. We would need to configure that somehow.
>
>   - name: Domain WildFly Server
>     resource-name-template: "%-"
>     path: "/server=*"
>     parents:
>     - Domain Host
>     proxy-metrics: true
>
> Oh, BTW, because the agent's remote JMX client is the jolokia client, we
> need to install a jolokia -javaagent in the slave server along with our
> agent. We'd need to add some additional information there so the host agent
> knows all the details so it can connect to the JMX server where the
> resource is hosted. How does it know the jolokia URL of the slave server?
>
> See how this snowballs? :-/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20171204/8bb17432/attachment.html 

From lponce at redhat.com  Mon Dec  4 07:26:38 2017
From: lponce at redhat.com (Lucas Ponce)
Date: Mon, 4 Dec 2017 13:26:38 +0100
Subject: [Hawkular-dev] need discussion around how we are going to do
 domain support
In-Reply-To: <1967471492.51281903.1512387871758.JavaMail.zimbra@redhat.com>
References: <722629168.49381712.1512167769649.JavaMail.zimbra@redhat.com>
	<1935475527.49398854.1512168892623.JavaMail.zimbra@redhat.com>
	<CAGUuq3XaJ7-p+oWiYdWKzO6SDfzN+YLcuTkFSZyFBawEAJr7Jw@mail.gmail.com>
	<443296667.51267597.1512385792555.JavaMail.zimbra@redhat.com>
	<1967471492.51281903.1512387871758.JavaMail.zimbra@redhat.com>
Message-ID: <CAGUuq3VWvn9kFdAyYT3ABf-ZTp48kba1quKDXkNo9J+R0J60Rw@mail.gmail.com>

On Mon, Dec 4, 2017 at 12:44 PM, John Mazzitelli <mazz at redhat.com> wrote:

>
>
> ----- Original Message -----
> > > > Spent time trying to figure out how to support WildFly domain mode
> with
> > > > the new stuff. It's not going well. Next week I will need to have
> some
> > > > discussions on how we want to do this.
> > > >
> > > > The issues with domain mode (I'll try to be short):
> > > >
> > > > 1) host controllers do not emit JMX metrics for slave servers - just
> not
> > > > implemented in WildFly. Need to go to slave servers for their own JMX
> > > > metrics.
> > > > 2) slave servers do not have managed interfaces available remotely
> (can't
> > > > connect to slaves over remote management interface - so agent can't
> get
> > > > any
> > > > inventory from slaves - have to get all inventory from host
> controller)
> > > >
> > > > This means our agent in the host controller needs to collect all
> > > > inventory
> > > > for both master host controller and slave servers.
> > > >
> > > > But we need the slave servers to have a metrics endpoint because its
> only
> > > > in the slaves where we can get JMX metrics (remember, host controller
> > > > can't
> > > > give us that, we must scrape the slave servers for the JMX metrics).
> > > >
> > > > If we do not have our agent in slave server (why bother if we can't
> > > > connect to it over DMR API to get inventory?), we still need to
> somehow
> > > > get
> > > > the P8s JMX Exporter installed in the slaves. But if we just use the
> raw
> > > > jmx exporter agent, how do we tell our h-services server/P8s server
> about
> > > > the new endpoint that needs to be scraped? And how do we get the jmx
> > > > exporter yml config to install there?
> > > >
> > > > So I suppose we should put our agent in a kind of "metrics only"
> mode in
> > > > all slave servers so it can expose the JMX exporter and pull down the
> > > > correct jmx exporter yml from h-services server and have it tell the
> > > > server
> > > > to add the scrape endpoint to p8s.
> > > >
> > > > But because we aren't getting inventory from slave servers, how can
> our
> > > > agent tell the server to add the scrape endpoint? Our current
> > > > implementation only adds scrape endpoints to our P8s server when new
> > > > agents
> > > > go into inventory. Workaround: either have agent store a small
> inventory
> > > > (just the agent resource itself) which triggers the new scrape
> endpoint
> > > > addition, or add a inventory REST endpoint for the agent to call to
> add
> > > > the
> > > > new scrape endpoint manually.
> > > >
> > > > OK, assume we have all this. How do we link the JMX metrics getting
> > > > collected from one feed (the agent in the slave server) to the
> inventory
> > > > from another feed (the agent in the host controller). Right now, it
> is
> > > > assumed the inventory metadata from feed "A" is matched with metrics
> from
> > > > the same feed. Now that we have to break that link (feed A's
> inventory
> > > > refers to feed B's metrics) we need to figure out how to fix this.
> > > >
> > > > There are other ancillary issues - like how do I get the correct
> metadata
> > > > defined for host controller so it can match resources/metrics from
> the
> > > > slaves. I assume that will be "implementation details."
> > > >
> > > > I'm sure this sounds like gibberish, but that's how convoluted
> supporting
> > > > domain mode is going to be.
> > > >
> > >
> > > Mazz,
> > >
> > > What about to promote the host controller agent as the responsible for
> > > everything ?
> > >
> > > - Host controller agent will be responsible to collect info of the
> domain
> > > (slave servers) and write into the inventory (Domain, Server, etc).
> > > - Slave server can have a "metrics-only" endpoint (perhaps same agent,
> > > perhaps a *-domain.jar if we want to simplify things).
> > > - Host controller agent can proxy slave metrics endpoint, so we can
> control
> > > endpoints like /metrics-domain/<slave>/metrics and that is what
> inventory
> > > uses to create the endpoint.
> > > - Host controller will expose in a proxy way, the metrics endpoint for
> the
> > > slaves.
> > >
> > > This approach focus the complixity in the host controller, but let to
> not
> > > have exceptions and corner use cases in the whole system, the benefit
> I see
> > > is that from MiQ and Hawkular Services we can try to maintain and
> uniform
> > > way.
> > >
> > > So, as you point Host Controllers needs to know the topology of the
> domain,
> > > and each slave should have a local /metrics which can be proxied from
> the
> > > host controller main endpoint.
> > > On this case, both Host Controllers and Slaves will share same feed,
> > > because, they are using the same agent, and that won't break any
> design use
> > > case.
> > >
> > > Also, I think that this proxy has a minimal technical complexity, as
> the
> > > host controller should have all info to expose that.
> >
> >
> > This has some promise. Just need to think about how to configure this
> proxy
> > and implement it without having to rip apart too much of the internals of
> > the agent code.
> >
> > Thinking out loud:
> >
> > We add "proxies" section to the agent config's metrics-exporters:
> >
> > metrics-exporter:
> >   enabled: true
> >   host: ${hawkular.agent.metrics.host:127.0.0.1}
> >   port: ${hawkular.agent.metrics.port:9779}
> >   config-dir: ${jboss.server.config.dir}
> >   config-file: WF10
> >   # HERE IS THE NEW SECTION
> >   proxies:
> >   - path: slave1/metrics
> >     host: 127.0.0.1
> >     port: 9780
> >   - path: slave2/metrics
> >     host: 127.0.0.1
> >     port: 9781
> >   - path: slave3/metrics
> >     host: 127.0.0.1
> >     port: 9782
> >
> > We would have to enhance our jmx exporter wrapper code so it supports
> those
> > extra endpoints (it would simply just pass-through request/responses to
> > those different endpoints).
> >
> > The slave agent configs would have their jmx exporter sections the same
> as
> > always and bind to localhost:
> >
> > metrics-exporter:
> >   enabled: true
> >   host: 127.0.0.1
> >   port: ${hawkular.agent.metrics.port:9779}
> >   config-dir: ${jboss.server.config.dir}
> >   config-file: WF10
> >
> > The nice thing here is they only need to expose the metric data on the
> > loopback address, 127.0.0.1, since the host controller agent will be on
> the
> > same box - no need to open this up to the wider network.
> >
> > The bad thing is the proxies section requires you to know how many slaves
> > there are and what their ports are going to be. There is no way to know
> that
> > ahead of time (this is all encoded in the host controller's config, and
> who
> > knows what the user wants to use - you can start host controllers with
> > --host-config option to specify a custom .xml). And besides, new slaves
> can
> > be added to the host controller at any time.
> >
> > So it seems there needs to be some kind of auto-discovery of proxies,
> not a
> > fixed configuration of them. Maybe we have a section that tells the
> agent to
> > check for metric endpoints to be proxied?
> >
> >   proxies:
> >   - host: 127.0.0.1
> >     port-range: 9780-9880
> >
> > But if we have auto-discover like that, what is the proxy /path that we
> need
> > to tell Prometheus to scrape? We need to distinguish it somehow so P
> knows
> > "to get slave #1, I scrape host:9779/slave1/metrics and to scrape slave
> #2
> > its host:9779/slave2/metrics the host controller its just
> host:9779/metrics
> > ??
> >
> > Maybe we tell the host controller agent to talk to the slave agent
> somehow?
> > Over remote JMX is an interesting idea (the agent already has its own JMX
> > MBean, as long as we can talk to it over remote JMX, we can have the host
> > agent ask the slave agent for information about its metrics endpoint
> ("what
> > is your host/port of /metrics endpoint?". We could possibly then
> associate
> > it with the name of the slave and use that as the proxy path?
> >
> > We could then add some kind of "flag" or metadata on the "Domain WildFly
> > Server" resource to say, "this resource has an agent whose metrics
> endpoint
> > we want to proxy - go ask this agent for its details". So when the host
> > agent runs its normal inventory scan, we can add special code to say,
> "when
> > you hit a resource that has this special proxy metadata, you need to
> proxy
> > that resource's agent's metrics endpoint." Because the Domain WildFly
> Server
> > is a DMR resource - there is no associated JMX Server information with
> it.
> > We would need to configure that somehow.
> >
> >   - name: Domain WildFly Server
> >     resource-name-template: "%-"
> >     path: "/server=*"
> >     parents:
> >     - Domain Host
> >     proxy-metrics: true
> >
> > Oh, BTW, because the agent's remote JMX client is the jolokia client, we
> need
> > to install a jolokia -javaagent in the slave server along with our agent.
> > We'd need to add some additional information there so the host agent
> knows
> > all the details so it can connect to the JMX server where the resource is
> > hosted. How does it know the jolokia URL of the slave server?
>
>
> Maybe we forget all this remote JMX talk from host agent to slave agent
> and just use the file system? We know the slaves are on the same box as the
> host controller - so perhaps we tell each slave, "write some information
> the host agent needs so it can proxy your metrics endpoint". This has the
> added benefit of having that auto-discovery I said we'd need. The host
> controller simply proxies whatever information it detects on the filesystem
> (maybe each slave writes a file whose name is the name of the slave (so all
> the files are unique) and in each file is information like the host and
> port where the /metrics endpoint is. Any file the host agent finds it will
> proxy. The agent therefore would just need to be told, "look in this
> directory - as you see slaves writing their proxy files, start proxying
> their info."
>
>
Using filesystem to share info between slaves sounds ok for this use case (
domain/tmp, or similar)


> The question then becomes, how does the agent tell the h-server, "let P
> know that it needs to scrape this endpoint"? I think we are back to the
> "have the agent manually tell the h-server to tell P to register a metrics
> endpoint" - what is provided by this commit in my local branch:
> https://github.com/jmazzitelli/hawkular-commons/commit/
> ae8d316486be7ba738f63b666c99a4f5a2e61f60
>
>
Why don't just upload the agent with the config of the new slaves ?

Today, when an agent is added/modified (is the same use case in practical
terms), the p8s are generated.

So, if we know we have a modification, just add the slave endpoints into
the agent config, and just let the inventory re-create them.




> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20171204/886c7fb5/attachment-0001.html 

From mazz at redhat.com  Mon Dec  4 08:00:07 2017
From: mazz at redhat.com (John Mazzitelli)
Date: Mon, 4 Dec 2017 08:00:07 -0500 (EST)
Subject: [Hawkular-dev] need discussion around how we are going to do
 domain support
In-Reply-To: <CAGUuq3VWvn9kFdAyYT3ABf-ZTp48kba1quKDXkNo9J+R0J60Rw@mail.gmail.com>
References: <722629168.49381712.1512167769649.JavaMail.zimbra@redhat.com>
	<1935475527.49398854.1512168892623.JavaMail.zimbra@redhat.com>
	<CAGUuq3XaJ7-p+oWiYdWKzO6SDfzN+YLcuTkFSZyFBawEAJr7Jw@mail.gmail.com>
	<443296667.51267597.1512385792555.JavaMail.zimbra@redhat.com>
	<1967471492.51281903.1512387871758.JavaMail.zimbra@redhat.com>
	<CAGUuq3VWvn9kFdAyYT3ABf-ZTp48kba1quKDXkNo9J+R0J60Rw@mail.gmail.com>
Message-ID: <965644111.51326703.1512392407093.JavaMail.zimbra@redhat.com>

> Why don't just upload the agent with the config of the new slaves ?
> 
> Today, when an agent is added/modified (is the same use case in practical
> terms), the p8s are generated.
> 
> So, if we know we have a modification, just add the slave endpoints into
> the agent config, and just let the inventory re-create them.

Hmm.. that's possible. We would have to change the way the "Metrics Endpoint" resource config property is processed. Today it is assumed to be a single value ("host:port")... we would need it to be a comma-separate list (e.g. "host:8787,host:8787/slave1/metrics,host:8787/slave2/metrics" or something like that. The server would then have to parse it out and create multiple endpoints. Note that it would also be nice to DELETE files if they did exist before but now no longer do exist.

Like if the agent originally was

"host:8787,host:8787/slave1/metrics,host:8787/slave2/metrics"

but is now

"host:8787,host:8787/slave1/metrics"

the slave2 file should be deleted so P doesn't try to scrape it anymore.

From lponce at redhat.com  Mon Dec  4 08:06:31 2017
From: lponce at redhat.com (Lucas Ponce)
Date: Mon, 4 Dec 2017 14:06:31 +0100
Subject: [Hawkular-dev] need discussion around how we are going to do
 domain support
In-Reply-To: <965644111.51326703.1512392407093.JavaMail.zimbra@redhat.com>
References: <722629168.49381712.1512167769649.JavaMail.zimbra@redhat.com>
	<1935475527.49398854.1512168892623.JavaMail.zimbra@redhat.com>
	<CAGUuq3XaJ7-p+oWiYdWKzO6SDfzN+YLcuTkFSZyFBawEAJr7Jw@mail.gmail.com>
	<443296667.51267597.1512385792555.JavaMail.zimbra@redhat.com>
	<1967471492.51281903.1512387871758.JavaMail.zimbra@redhat.com>
	<CAGUuq3VWvn9kFdAyYT3ABf-ZTp48kba1quKDXkNo9J+R0J60Rw@mail.gmail.com>
	<965644111.51326703.1512392407093.JavaMail.zimbra@redhat.com>
Message-ID: <CAGUuq3W0kQWN+_HTwMNXUpecQAnYc2-w-pdyPHiGrt7xX=WWvw@mail.gmail.com>

On Mon, Dec 4, 2017 at 2:00 PM, John Mazzitelli <mazz at redhat.com> wrote:

> > Why don't just upload the agent with the config of the new slaves ?
> >
> > Today, when an agent is added/modified (is the same use case in practical
> > terms), the p8s are generated.
> >
> > So, if we know we have a modification, just add the slave endpoints into
> > the agent config, and just let the inventory re-create them.
>
> Hmm.. that's possible. We would have to change the way the "Metrics
> Endpoint" resource config property is processed. Today it is assumed to be
> a single value ("host:port")... we would need it to be a comma-separate
> list (e.g. "host:8787,host:8787/slave1/metrics,host:8787/slave2/metrics"
> or something like that. The server would then have to parse it out and
> create multiple endpoints. Note that it would also be nice to DELETE files
> if they did exist before but now no longer do exist.
>
> Like if the agent originally was
>
> "host:8787,host:8787/slave1/metrics,host:8787/slave2/metrics"
>
> but is now
>
> "host:8787,host:8787/slave1/metrics"
>
> the slave2 file should be deleted so P doesn't try to scrape it anymore.
>

It's doable, let's collect requeriments about if we would need additional
info perform those changes once we have a picture more or less drafted.

I mean, if we need to have a map to easy identify slave or something
similar (well, this is more implementation detail).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20171204/71807d24/attachment.html 

From mazz at redhat.com  Mon Dec  4 12:08:25 2017
From: mazz at redhat.com (John Mazzitelli)
Date: Mon, 4 Dec 2017 12:08:25 -0500 (EST)
Subject: [Hawkular-dev] need discussion around how we are going to do
 domain support
In-Reply-To: <965644111.51326703.1512392407093.JavaMail.zimbra@redhat.com>
References: <722629168.49381712.1512167769649.JavaMail.zimbra@redhat.com>
	<1935475527.49398854.1512168892623.JavaMail.zimbra@redhat.com>
	<CAGUuq3XaJ7-p+oWiYdWKzO6SDfzN+YLcuTkFSZyFBawEAJr7Jw@mail.gmail.com>
	<443296667.51267597.1512385792555.JavaMail.zimbra@redhat.com>
	<1967471492.51281903.1512387871758.JavaMail.zimbra@redhat.com>
	<CAGUuq3VWvn9kFdAyYT3ABf-ZTp48kba1quKDXkNo9J+R0J60Rw@mail.gmail.com>
	<965644111.51326703.1512392407093.JavaMail.zimbra@redhat.com>
Message-ID: <63021232.51549606.1512407305566.JavaMail.zimbra@redhat.com>

The current plan is to have a metrics-only agent run in the slave server. It will write a file indicating what its jmx exporter endpoint is. The host agent will probe a directory where the slave agents write their files - when new files are discovered, the agent will change its resource configuration "Metrics Endpoint" thus triggering an inventory update. When the server sees this update, it will write out new P scrape endpoint configs so P knows to scrape the slave endpoints, too.

Requires a change to the agent as well as server side.

From mwringe at redhat.com  Tue Dec  5 15:11:23 2017
From: mwringe at redhat.com (Matthew Wringe)
Date: Tue, 5 Dec 2017 15:11:23 -0500
Subject: [Hawkular-dev] OpenShift Deployment
In-Reply-To: <CA+aN=MokaPHO5xs9hHnsJ+8EeW5fRf+89VqLVGx_MM+_FFRqpA@mail.gmail.com>
References: <CAPq8dB+NBXzSEAKa019utbeJ=dhyTFiMSw_tnMJo=SdviizzTQ@mail.gmail.com>
	<CAGUuq3WTg-pLJL=jxUJyrq7Dgxz1-LeTJqzV2qTQPpNUawSHSw@mail.gmail.com>
	<CAGUuq3WTVqwpuKLzXTt9=u3dA7K2AjQvXgosERhgpVPRgfsc0A@mail.gmail.com>
	<CAPq8dBL9fwpYVi1u_c2uJ_1fT51QqbX+qs+-tbp-vbBtbmGssQ@mail.gmail.com>
	<CA+aN=MokaPHO5xs9hHnsJ+8EeW5fRf+89VqLVGx_MM+_FFRqpA@mail.gmail.com>
Message-ID: <CAPq8dB+0S82KauZ_y06aAnAKO6RVFe-MaNfo3=d3YYYy+i7Nsg@mail.gmail.com>

Going with the all-in-one pod approach, there is also the question of how
many volumes we want to use.

We can have everything in one volume with multiple directories (eg /p8s,
/hawkular-inventory, etc). This would make it easier to install as we only
have to have one directory to deal with.

Of we could have multiple volumes, one for Hawkular Services and one for
Prometheus. This might give us some more flexibility in the future, but I
can't see why we would need this extra complexity at the moment.

Any thoughts on this? Or maybe some perspective I am missing?

On Mon, Dec 4, 2017 at 5:35 AM, Thomas Heute <theute at redhat.com> wrote:

> +1
>
> On Thu, Nov 30, 2017 at 2:56 PM, Matthew Wringe <mwringe at redhat.com>
> wrote:
>
>> Yeah, I think we should go with the all-in-one pod approach for now. If
>> we discover certain use cases that wont work properly we can re-evaluate.
>>
>> On Thu, Nov 30, 2017 at 5:15 AM, Lucas Ponce <lponce at redhat.com> wrote:
>>
>>>
>>>
>>> On Thu, Nov 30, 2017 at 10:55 AM, Lucas Ponce <lponce at redhat.com> wrote:
>>>
>>>>
>>>>
>>>> On Mon, Nov 27, 2017 at 4:38 PM, Matthew Wringe <mwringe at redhat.com>
>>>> wrote:
>>>>
>>>>> With the changes that are now going to include Prometheus, how do we
>>>>> want to deploy this in OpenShift?
>>>>>
>>>>> We can have a few options:
>>>>>
>>>>> ALL-IN-ONE CONTAINER
>>>>> We put both Hawkular Services and Prometheus in the same container.
>>>>>
>>>>> Pros:
>>>>> - easy to deploy in plain docker (but this doesn't appear to be a
>>>>> usecase we are targetting anyways)
>>>>> - shares the same network connection (even localhost) and ip address
>>>>> (eg but both services are on the different ports).
>>>>> - Does't require any special wiring of components.
>>>>> - Can share the same volume mount
>>>>> - version of components can't get out of sync.
>>>>>
>>>>> Cons:
>>>>> - workflow doesn't work nicely. Docker containers are meant to only
>>>>> run a single application and running two can cause problems. Eg lifecycle
>>>>> events would become tricky and require some hacks to get around things.
>>>>> - can't independently deploy things
>>>>> - can't reuse or share any existing Prometheus docker containers.
>>>>>
>>>>> ALL-IN-ONE POD
>>>>> Hawkular Services and Prometheus are in their own containers, but they
>>>>> are both deployed within the same pod.
>>>>>
>>>>> Pros:
>>>>> - shares the same network connection.
>>>>> - bound to the same machine (useful if sharing the same hostpath pv)
>>>>> and don' need to worry about external network configurations (eg firewalls
>>>>> between OpenShift nodes)
>>>>> - pvs can be shared or separate.
>>>>> - lifecycle events will work properly.
>>>>>
>>>>> Cons:
>>>>> - lifecycle hooks will mean that both containers will have to pass
>>>>> before either one will enter the ready state. So if Prometheus is failing
>>>>> for some reason, Hawkular Services will not be available under the service.
>>>>> - cannot independently update one container. If we need to deploy a
>>>>> new container we will need to bring down the whole pod.
>>>>> - are stuck with a 1:1 ratio between Hawkular Services and Prometheus
>>>>>
>>>>>
>>>> One technical requeriment is that Hawkular Services needs to now where
>>>> is Prometheus server at initialization.
>>>>
>>>
>>> One technical requeriment is that Hawkular Services needs to *know*
>>> where is Prometheus server at initialization.
>>>
>>> [Sorry, typing fast]
>>>
>>> So, I guess that all-in-one pod will simplify things on this case.
>>>>
>>>> I would start with this architecture first and harden the basic
>>>> scenarios.
>>>>
>>>>
>>>>>
>>>>> SEPARATE PODS
>>>>> Hawkular Services and Prometheus have their own separate pods.
>>>>>
>>>>> Pros:
>>>>> - can independently run components and each component has its own
>>>>> separate lifecycle
>>>>> - if in the future we want to cluster Hawkular Services. this will
>>>>> make it a lot easier and will also allow for running an n:m ratio between
>>>>> Hawkular Services and Prometheus
>>>>> - probably the more 'correct' way to deploy things as we don't have a
>>>>> strong requirement for Hawkular Services and Prometheus to run together.
>>>>>
>>>>> Cons:
>>>>> - more complex wiring. We will need to have extra services and routes
>>>>> created to handle this. This mean more things running and more chances for
>>>>> things to go wrong. Also more things to configure
>>>>> - reusing a PV between Hawkular Services and Prometheus could be more
>>>>> challenging (especially if we are using hostpath pvs). Updating the
>>>>> Prometheus scrape endpoint may require a new component and container.
>>>>>
>>>>> _______________________________________________
>>>>> hawkular-dev mailing list
>>>>> hawkular-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>>>>>
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> hawkular-dev mailing list
>>> hawkular-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>>>
>>>
>>
>> _______________________________________________
>> hawkular-dev mailing list
>> hawkular-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>>
>>
>
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20171205/e3587064/attachment-0001.html 

From pgier at redhat.com  Tue Dec  5 16:09:32 2017
From: pgier at redhat.com (Paul Gier)
Date: Tue, 5 Dec 2017 15:09:32 -0600
Subject: [Hawkular-dev] OpenShift Deployment
In-Reply-To: <CAPq8dB+0S82KauZ_y06aAnAKO6RVFe-MaNfo3=d3YYYy+i7Nsg@mail.gmail.com>
References: <CAPq8dB+NBXzSEAKa019utbeJ=dhyTFiMSw_tnMJo=SdviizzTQ@mail.gmail.com>
	<CAGUuq3WTg-pLJL=jxUJyrq7Dgxz1-LeTJqzV2qTQPpNUawSHSw@mail.gmail.com>
	<CAGUuq3WTVqwpuKLzXTt9=u3dA7K2AjQvXgosERhgpVPRgfsc0A@mail.gmail.com>
	<CAPq8dBL9fwpYVi1u_c2uJ_1fT51QqbX+qs+-tbp-vbBtbmGssQ@mail.gmail.com>
	<CA+aN=MokaPHO5xs9hHnsJ+8EeW5fRf+89VqLVGx_MM+_FFRqpA@mail.gmail.com>
	<CAPq8dB+0S82KauZ_y06aAnAKO6RVFe-MaNfo3=d3YYYy+i7Nsg@mail.gmail.com>
Message-ID: <CAAe91u1j4MoPYmGERwUa8+LheOYLupTwkJSCpZCpCqS-7eazZQ@mail.gmail.com>

If we use a single shared volume, I'd be a little worried about Prometheus
data filling up all the space and causing problems in the hawkular-services
container.  We've seen some runaway Prometheus storage issues in open shift.

On Dec 5, 2017 2:13 PM, "Matthew Wringe" <mwringe at redhat.com> wrote:

Going with the all-in-one pod approach, there is also the question of how
many volumes we want to use.

We can have everything in one volume with multiple directories (eg /p8s,
/hawkular-inventory, etc). This would make it easier to install as we only
have to have one directory to deal with.

Of we could have multiple volumes, one for Hawkular Services and one for
Prometheus. This might give us some more flexibility in the future, but I
can't see why we would need this extra complexity at the moment.

Any thoughts on this? Or maybe some perspective I am missing?

On Mon, Dec 4, 2017 at 5:35 AM, Thomas Heute <theute at redhat.com> wrote:

> +1
>
> On Thu, Nov 30, 2017 at 2:56 PM, Matthew Wringe <mwringe at redhat.com>
> wrote:
>
>> Yeah, I think we should go with the all-in-one pod approach for now. If
>> we discover certain use cases that wont work properly we can re-evaluate.
>>
>> On Thu, Nov 30, 2017 at 5:15 AM, Lucas Ponce <lponce at redhat.com> wrote:
>>
>>>
>>>
>>> On Thu, Nov 30, 2017 at 10:55 AM, Lucas Ponce <lponce at redhat.com> wrote:
>>>
>>>>
>>>>
>>>> On Mon, Nov 27, 2017 at 4:38 PM, Matthew Wringe <mwringe at redhat.com>
>>>> wrote:
>>>>
>>>>> With the changes that are now going to include Prometheus, how do we
>>>>> want to deploy this in OpenShift?
>>>>>
>>>>> We can have a few options:
>>>>>
>>>>> ALL-IN-ONE CONTAINER
>>>>> We put both Hawkular Services and Prometheus in the same container.
>>>>>
>>>>> Pros:
>>>>> - easy to deploy in plain docker (but this doesn't appear to be a
>>>>> usecase we are targetting anyways)
>>>>> - shares the same network connection (even localhost) and ip address
>>>>> (eg but both services are on the different ports).
>>>>> - Does't require any special wiring of components.
>>>>> - Can share the same volume mount
>>>>> - version of components can't get out of sync.
>>>>>
>>>>> Cons:
>>>>> - workflow doesn't work nicely. Docker containers are meant to only
>>>>> run a single application and running two can cause problems. Eg lifecycle
>>>>> events would become tricky and require some hacks to get around things.
>>>>> - can't independently deploy things
>>>>> - can't reuse or share any existing Prometheus docker containers.
>>>>>
>>>>> ALL-IN-ONE POD
>>>>> Hawkular Services and Prometheus are in their own containers, but they
>>>>> are both deployed within the same pod.
>>>>>
>>>>> Pros:
>>>>> - shares the same network connection.
>>>>> - bound to the same machine (useful if sharing the same hostpath pv)
>>>>> and don' need to worry about external network configurations (eg firewalls
>>>>> between OpenShift nodes)
>>>>> - pvs can be shared or separate.
>>>>> - lifecycle events will work properly.
>>>>>
>>>>> Cons:
>>>>> - lifecycle hooks will mean that both containers will have to pass
>>>>> before either one will enter the ready state. So if Prometheus is failing
>>>>> for some reason, Hawkular Services will not be available under the service.
>>>>> - cannot independently update one container. If we need to deploy a
>>>>> new container we will need to bring down the whole pod.
>>>>> - are stuck with a 1:1 ratio between Hawkular Services and Prometheus
>>>>>
>>>>>
>>>> One technical requeriment is that Hawkular Services needs to now where
>>>> is Prometheus server at initialization.
>>>>
>>>
>>> One technical requeriment is that Hawkular Services needs to *know*
>>> where is Prometheus server at initialization.
>>>
>>> [Sorry, typing fast]
>>>
>>> So, I guess that all-in-one pod will simplify things on this case.
>>>>
>>>> I would start with this architecture first and harden the basic
>>>> scenarios.
>>>>
>>>>
>>>>>
>>>>> SEPARATE PODS
>>>>> Hawkular Services and Prometheus have their own separate pods.
>>>>>
>>>>> Pros:
>>>>> - can independently run components and each component has its own
>>>>> separate lifecycle
>>>>> - if in the future we want to cluster Hawkular Services. this will
>>>>> make it a lot easier and will also allow for running an n:m ratio between
>>>>> Hawkular Services and Prometheus
>>>>> - probably the more 'correct' way to deploy things as we don't have a
>>>>> strong requirement for Hawkular Services and Prometheus to run together.
>>>>>
>>>>> Cons:
>>>>> - more complex wiring. We will need to have extra services and routes
>>>>> created to handle this. This mean more things running and more chances for
>>>>> things to go wrong. Also more things to configure
>>>>> - reusing a PV between Hawkular Services and Prometheus could be more
>>>>> challenging (especially if we are using hostpath pvs). Updating the
>>>>> Prometheus scrape endpoint may require a new component and container.
>>>>>
>>>>> _______________________________________________
>>>>> hawkular-dev mailing list
>>>>> hawkular-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>>>>>
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> hawkular-dev mailing list
>>> hawkular-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>>>
>>>
>>
>> _______________________________________________
>> hawkular-dev mailing list
>> hawkular-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>>
>>
>
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
>

_______________________________________________
hawkular-dev mailing list
hawkular-dev at lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20171205/2f02f3cc/attachment.html 

From mwringe at redhat.com  Tue Dec  5 16:27:26 2017
From: mwringe at redhat.com (Matthew Wringe)
Date: Tue, 5 Dec 2017 16:27:26 -0500
Subject: [Hawkular-dev] OpenShift Deployment
In-Reply-To: <CAAe91u1j4MoPYmGERwUa8+LheOYLupTwkJSCpZCpCqS-7eazZQ@mail.gmail.com>
References: <CAPq8dB+NBXzSEAKa019utbeJ=dhyTFiMSw_tnMJo=SdviizzTQ@mail.gmail.com>
	<CAGUuq3WTg-pLJL=jxUJyrq7Dgxz1-LeTJqzV2qTQPpNUawSHSw@mail.gmail.com>
	<CAGUuq3WTVqwpuKLzXTt9=u3dA7K2AjQvXgosERhgpVPRgfsc0A@mail.gmail.com>
	<CAPq8dBL9fwpYVi1u_c2uJ_1fT51QqbX+qs+-tbp-vbBtbmGssQ@mail.gmail.com>
	<CA+aN=MokaPHO5xs9hHnsJ+8EeW5fRf+89VqLVGx_MM+_FFRqpA@mail.gmail.com>
	<CAPq8dB+0S82KauZ_y06aAnAKO6RVFe-MaNfo3=d3YYYy+i7Nsg@mail.gmail.com>
	<CAAe91u1j4MoPYmGERwUa8+LheOYLupTwkJSCpZCpCqS-7eazZQ@mail.gmail.com>
Message-ID: <CAPq8dB+rm3_c-or_NLWPyujQ7+Yncnxmr+PPU33AAmb5n1dGjQ@mail.gmail.com>

On Tue, Dec 5, 2017 at 4:09 PM, Paul Gier <pgier at redhat.com> wrote:

> If we use a single shared volume, I'd be a little worried about Prometheus
> data filling up all the space and causing problems in the hawkular-services
> container.  We've seen some runaway Prometheus storage issues in open shift.
>

One component eating all the space of the other is something we might want
to be worried about.

Prometheus considers its metrics storage to not be durable and that is our
stance with Hawkular Services as well.

The only thing we can really do is to give recommendations to users for how
much storage they should need. Since we are tightly coupling Hawkular
Services with Prometheus, it might be easier to just keep them together and
give the recommendation for both running.


>
>
> On Dec 5, 2017 2:13 PM, "Matthew Wringe" <mwringe at redhat.com> wrote:
>
> Going with the all-in-one pod approach, there is also the question of how
> many volumes we want to use.
>
> We can have everything in one volume with multiple directories (eg /p8s,
> /hawkular-inventory, etc). This would make it easier to install as we only
> have to have one directory to deal with.
>
> Of we could have multiple volumes, one for Hawkular Services and one for
> Prometheus. This might give us some more flexibility in the future, but I
> can't see why we would need this extra complexity at the moment.
>
> Any thoughts on this? Or maybe some perspective I am missing?
>
> On Mon, Dec 4, 2017 at 5:35 AM, Thomas Heute <theute at redhat.com> wrote:
>
>> +1
>>
>> On Thu, Nov 30, 2017 at 2:56 PM, Matthew Wringe <mwringe at redhat.com>
>> wrote:
>>
>>> Yeah, I think we should go with the all-in-one pod approach for now. If
>>> we discover certain use cases that wont work properly we can re-evaluate.
>>>
>>> On Thu, Nov 30, 2017 at 5:15 AM, Lucas Ponce <lponce at redhat.com> wrote:
>>>
>>>>
>>>>
>>>> On Thu, Nov 30, 2017 at 10:55 AM, Lucas Ponce <lponce at redhat.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Mon, Nov 27, 2017 at 4:38 PM, Matthew Wringe <mwringe at redhat.com>
>>>>> wrote:
>>>>>
>>>>>> With the changes that are now going to include Prometheus, how do we
>>>>>> want to deploy this in OpenShift?
>>>>>>
>>>>>> We can have a few options:
>>>>>>
>>>>>> ALL-IN-ONE CONTAINER
>>>>>> We put both Hawkular Services and Prometheus in the same container.
>>>>>>
>>>>>> Pros:
>>>>>> - easy to deploy in plain docker (but this doesn't appear to be a
>>>>>> usecase we are targetting anyways)
>>>>>> - shares the same network connection (even localhost) and ip address
>>>>>> (eg but both services are on the different ports).
>>>>>> - Does't require any special wiring of components.
>>>>>> - Can share the same volume mount
>>>>>> - version of components can't get out of sync.
>>>>>>
>>>>>> Cons:
>>>>>> - workflow doesn't work nicely. Docker containers are meant to only
>>>>>> run a single application and running two can cause problems. Eg lifecycle
>>>>>> events would become tricky and require some hacks to get around things.
>>>>>> - can't independently deploy things
>>>>>> - can't reuse or share any existing Prometheus docker containers.
>>>>>>
>>>>>> ALL-IN-ONE POD
>>>>>> Hawkular Services and Prometheus are in their own containers, but
>>>>>> they are both deployed within the same pod.
>>>>>>
>>>>>> Pros:
>>>>>> - shares the same network connection.
>>>>>> - bound to the same machine (useful if sharing the same hostpath pv)
>>>>>> and don' need to worry about external network configurations (eg firewalls
>>>>>> between OpenShift nodes)
>>>>>> - pvs can be shared or separate.
>>>>>> - lifecycle events will work properly.
>>>>>>
>>>>>> Cons:
>>>>>> - lifecycle hooks will mean that both containers will have to pass
>>>>>> before either one will enter the ready state. So if Prometheus is failing
>>>>>> for some reason, Hawkular Services will not be available under the service.
>>>>>> - cannot independently update one container. If we need to deploy a
>>>>>> new container we will need to bring down the whole pod.
>>>>>> - are stuck with a 1:1 ratio between Hawkular Services and Prometheus
>>>>>>
>>>>>>
>>>>> One technical requeriment is that Hawkular Services needs to now where
>>>>> is Prometheus server at initialization.
>>>>>
>>>>
>>>> One technical requeriment is that Hawkular Services needs to *know*
>>>> where is Prometheus server at initialization.
>>>>
>>>> [Sorry, typing fast]
>>>>
>>>> So, I guess that all-in-one pod will simplify things on this case.
>>>>>
>>>>> I would start with this architecture first and harden the basic
>>>>> scenarios.
>>>>>
>>>>>
>>>>>>
>>>>>> SEPARATE PODS
>>>>>> Hawkular Services and Prometheus have their own separate pods.
>>>>>>
>>>>>> Pros:
>>>>>> - can independently run components and each component has its own
>>>>>> separate lifecycle
>>>>>> - if in the future we want to cluster Hawkular Services. this will
>>>>>> make it a lot easier and will also allow for running an n:m ratio between
>>>>>> Hawkular Services and Prometheus
>>>>>> - probably the more 'correct' way to deploy things as we don't have a
>>>>>> strong requirement for Hawkular Services and Prometheus to run together.
>>>>>>
>>>>>> Cons:
>>>>>> - more complex wiring. We will need to have extra services and routes
>>>>>> created to handle this. This mean more things running and more chances for
>>>>>> things to go wrong. Also more things to configure
>>>>>> - reusing a PV between Hawkular Services and Prometheus could be more
>>>>>> challenging (especially if we are using hostpath pvs). Updating the
>>>>>> Prometheus scrape endpoint may require a new component and container.
>>>>>>
>>>>>> _______________________________________________
>>>>>> hawkular-dev mailing list
>>>>>> hawkular-dev at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> hawkular-dev mailing list
>>>> hawkular-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> hawkular-dev mailing list
>>> hawkular-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>>>
>>>
>>
>> _______________________________________________
>> hawkular-dev mailing list
>> hawkular-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>>
>>
>
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
>
>
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20171205/c87ef70d/attachment-0001.html 

From mazz at redhat.com  Tue Dec  5 17:57:06 2017
From: mazz at redhat.com (John Mazzitelli)
Date: Tue, 5 Dec 2017 17:57:06 -0500 (EST)
Subject: [Hawkular-dev] OpenShift Deployment
In-Reply-To: <CAPq8dB+rm3_c-or_NLWPyujQ7+Yncnxmr+PPU33AAmb5n1dGjQ@mail.gmail.com>
References: <CAPq8dB+NBXzSEAKa019utbeJ=dhyTFiMSw_tnMJo=SdviizzTQ@mail.gmail.com>
	<CAGUuq3WTg-pLJL=jxUJyrq7Dgxz1-LeTJqzV2qTQPpNUawSHSw@mail.gmail.com>
	<CAGUuq3WTVqwpuKLzXTt9=u3dA7K2AjQvXgosERhgpVPRgfsc0A@mail.gmail.com>
	<CAPq8dBL9fwpYVi1u_c2uJ_1fT51QqbX+qs+-tbp-vbBtbmGssQ@mail.gmail.com>
	<CA+aN=MokaPHO5xs9hHnsJ+8EeW5fRf+89VqLVGx_MM+_FFRqpA@mail.gmail.com>
	<CAPq8dB+0S82KauZ_y06aAnAKO6RVFe-MaNfo3=d3YYYy+i7Nsg@mail.gmail.com>
	<CAAe91u1j4MoPYmGERwUa8+LheOYLupTwkJSCpZCpCqS-7eazZQ@mail.gmail.com>
	<CAPq8dB+rm3_c-or_NLWPyujQ7+Yncnxmr+PPU33AAmb5n1dGjQ@mail.gmail.com>
Message-ID: <1997213574.52706312.1512514626918.JavaMail.zimbra@redhat.com>

For now, at this point in time, I vote for simplicity above things like scalability issues. Let's make it simple and easy for people to try this stuff out. Worry about Prometheus filling up a volume when we actually have people using this at a scale where that actually will be a real problem.

$0.02 deposited.

----- Original Message -----
> On Tue, Dec 5, 2017 at 4:09 PM, Paul Gier < pgier at redhat.com > wrote:
> 
> 
> 
> If we use a single shared volume, I'd be a little worried about Prometheus
> data filling up all the space and causing problems in the hawkular-services
> container. We've seen some runaway Prometheus storage issues in open shift.
> 
> One component eating all the space of the other is something we might want to
> be worried about.
> 
> Prometheus considers its metrics storage to not be durable and that is our
> stance with Hawkular Services as well.
> 
> The only thing we can really do is to give recommendations to users for how
> much storage they should need. Since we are tightly coupling Hawkular
> Services with Prometheus, it might be easier to just keep them together and
> give the recommendation for both running.
> 
> 
> 
> 
> 
> On Dec 5, 2017 2:13 PM, "Matthew Wringe" < mwringe at redhat.com > wrote:
> 
> 
> 
> Going with the all-in-one pod approach, there is also the question of how
> many volumes we want to use.
> 
> We can have everything in one volume with multiple directories (eg /p8s,
> /hawkular-inventory, etc). This would make it easier to install as we only
> have to have one directory to deal with.
> 
> Of we could have multiple volumes, one for Hawkular Services and one for
> Prometheus. This might give us some more flexibility in the future, but I
> can't see why we would need this extra complexity at the moment.
> 
> Any thoughts on this? Or maybe some perspective I am missing?
> 
> On Mon, Dec 4, 2017 at 5:35 AM, Thomas Heute < theute at redhat.com > wrote:
> 
> 
> 
> +1
> 
> On Thu, Nov 30, 2017 at 2:56 PM, Matthew Wringe < mwringe at redhat.com > wrote:
> 
> 
> 
> Yeah, I think we should go with the all-in-one pod approach for now. If we
> discover certain use cases that wont work properly we can re-evaluate.
> 
> On Thu, Nov 30, 2017 at 5:15 AM, Lucas Ponce < lponce at redhat.com > wrote:
> 
> 
> 
> 
> 
> On Thu, Nov 30, 2017 at 10:55 AM, Lucas Ponce < lponce at redhat.com > wrote:
> 
> 
> 
> 
> 
> On Mon, Nov 27, 2017 at 4:38 PM, Matthew Wringe < mwringe at redhat.com > wrote:
> 
> 
> 
> With the changes that are now going to include Prometheus, how do we want to
> deploy this in OpenShift?
> 
> We can have a few options:
> 
> ALL-IN-ONE CONTAINER
> We put both Hawkular Services and Prometheus in the same container.
> 
> Pros:
> - easy to deploy in plain docker (but this doesn't appear to be a usecase we
> are targetting anyways)
> - shares the same network connection (even localhost) and ip address (eg but
> both services are on the different ports).
> - Does't require any special wiring of components.
> - Can share the same volume mount
> - version of components can't get out of sync.
> 
> Cons:
> - workflow doesn't work nicely. Docker containers are meant to only run a
> single application and running two can cause problems. Eg lifecycle events
> would become tricky and require some hacks to get around things.
> - can't independently deploy things
> - can't reuse or share any existing Prometheus docker containers.
> 
> ALL-IN-ONE POD
> Hawkular Services and Prometheus are in their own containers, but they are
> both deployed within the same pod.
> 
> Pros:
> - shares the same network connection.
> - bound to the same machine (useful if sharing the same hostpath pv) and don'
> need to worry about external network configurations (eg firewalls between
> OpenShift nodes)
> - pvs can be shared or separate.
> - lifecycle events will work properly.
> 
> Cons:
> - lifecycle hooks will mean that both containers will have to pass before
> either one will enter the ready state. So if Prometheus is failing for some
> reason, Hawkular Services will not be available under the service.
> - cannot independently update one container. If we need to deploy a new
> container we will need to bring down the whole pod.
> - are stuck with a 1:1 ratio between Hawkular Services and Prometheus
> 
> 
> One technical requeriment is that Hawkular Services needs to now where is
> Prometheus server at initialization.
> One technical requeriment is that Hawkular Services needs to *know* where is
> Prometheus server at initialization.
> 
> [Sorry, typing fast]
> 
> 
> 
> 
> So, I guess that all-in-one pod will simplify things on this case.
> 
> I would start with this architecture first and harden the basic scenarios.
> 
> 
> 
> 
> 
> SEPARATE PODS
> Hawkular Services and Prometheus have their own separate pods.
> 
> Pros:
> - can independently run components and each component has its own separate
> lifecycle
> - if in the future we want to cluster Hawkular Services. this will make it a
> lot easier and will also allow for running an n:m ratio between Hawkular
> Services and Prometheus
> - probably the more 'correct' way to deploy things as we don't have a strong
> requirement for Hawkular Services and Prometheus to run together.
> 
> Cons:
> - more complex wiring. We will need to have extra services and routes created
> to handle this. This mean more things running and more chances for things to
> go wrong. Also more things to configure
> - reusing a PV between Hawkular Services and Prometheus could be more
> challenging (especially if we are using hostpath pvs). Updating the
> Prometheus scrape endpoint may require a new component and container.
> 

From vnguyen at redhat.com  Wed Dec  6 15:17:42 2017
From: vnguyen at redhat.com (Viet Nguyen)
Date: Wed, 6 Dec 2017 15:17:42 -0500 (EST)
Subject: [Hawkular-dev] OpenShift Deployment
In-Reply-To: <CAPq8dB+NBXzSEAKa019utbeJ=dhyTFiMSw_tnMJo=SdviizzTQ@mail.gmail.com>
References: <CAPq8dB+NBXzSEAKa019utbeJ=dhyTFiMSw_tnMJo=SdviizzTQ@mail.gmail.com>
Message-ID: <545952777.25428143.1512591462504.JavaMail.zimbra@redhat.com>

Hi all,

TL;DR: Technically it's ALL-IN-ONE POD but make HawkularService an option aka "sidecar" container

Would our team also look at providing Prometheus (without HS) as the defacto choice for OpenShift?

What I'm proposing is still technically an ALL-IN-ONE pod option. However, instead of looking at (Prometheus + HS) as a monolithic solution we can position HS as an enhancement to the plain vanilla Prometheus.  This add-on sidecar[1] approach can satisfy both Middleware users and non-middleware community users who may not necessarily need HawkularServices.  Let's say I want to use library X and X only comes with X+Y (which will cost me CPU and RAM resources) I may be less inclined to use the library.

[1] more on "sidecar" containers
http://blog.kubernetes.io/2015/06/the-distributed-system-toolkit-patterns.html


Viet




----- Original Message -----
From: "Matthew Wringe" <mwringe at redhat.com>
To: "Discussions around Hawkular development" <hawkular-dev at lists.jboss.org>
Sent: Monday, November 27, 2017 7:38:50 AM
Subject: [Hawkular-dev] OpenShift Deployment

With the changes that are now going to include Prometheus, how do we want to deploy this in OpenShift? 

We can have a few options: 

ALL-IN-ONE CONTAINER 
We put both Hawkular Services and Prometheus in the same container. 

Pros: 
- easy to deploy in plain docker (but this doesn't appear to be a usecase we are targetting anyways) 
- shares the same network connection (even localhost) and ip address (eg but both services are on the different ports). 
- Does't require any special wiring of components. 
- Can share the same volume mount 
- version of components can't get out of sync. 

Cons: 
- workflow doesn't work nicely. Docker containers are meant to only run a single application and running two can cause problems. Eg lifecycle events would become tricky and require some hacks to get around things. 
- can't independently deploy things 
- can't reuse or share any existing Prometheus docker containers. 

ALL-IN-ONE POD 
Hawkular Services and Prometheus are in their own containers, but they are both deployed within the same pod. 

Pros: 
- shares the same network connection. 
- bound to the same machine (useful if sharing the same hostpath pv) and don' need to worry about external network configurations (eg firewalls between OpenShift nodes) 
- pvs can be shared or separate. 
- lifecycle events will work properly. 

Cons: 
- lifecycle hooks will mean that both containers will have to pass before either one will enter the ready state. So if Prometheus is failing for some reason, Hawkular Services will not be available under the service. 
- cannot independently update one container. If we need to deploy a new container we will need to bring down the whole pod. 
- are stuck with a 1:1 ratio between Hawkular Services and Prometheus 


SEPARATE PODS 
Hawkular Services and Prometheus have their own separate pods. 

Pros: 
- can independently run components and each component has its own separate lifecycle 
- if in the future we want to cluster Hawkular Services. this will make it a lot easier and will also allow for running an n:m ratio between Hawkular Services and Prometheus 
- probably the more 'correct' way to deploy things as we don't have a strong requirement for Hawkular Services and Prometheus to run together. 

Cons: 
- more complex wiring. We will need to have extra services and routes created to handle this. This mean more things running and more chances for things to go wrong. Also more things to configure 
- reusing a PV between Hawkular Services and Prometheus could be more challenging (especially if we are using hostpath pvs). Updating the Prometheus scrape endpoint may require a new component and container. 

_______________________________________________
hawkular-dev mailing list
hawkular-dev at lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev

From mazz at redhat.com  Wed Dec  6 19:19:55 2017
From: mazz at redhat.com (John Mazzitelli)
Date: Wed, 6 Dec 2017 19:19:55 -0500 (EST)
Subject: [Hawkular-dev] need discussion around how we are going to do
 domain support
In-Reply-To: <63021232.51549606.1512407305566.JavaMail.zimbra@redhat.com>
References: <722629168.49381712.1512167769649.JavaMail.zimbra@redhat.com>
	<1935475527.49398854.1512168892623.JavaMail.zimbra@redhat.com>
	<CAGUuq3XaJ7-p+oWiYdWKzO6SDfzN+YLcuTkFSZyFBawEAJr7Jw@mail.gmail.com>
	<443296667.51267597.1512385792555.JavaMail.zimbra@redhat.com>
	<1967471492.51281903.1512387871758.JavaMail.zimbra@redhat.com>
	<CAGUuq3VWvn9kFdAyYT3ABf-ZTp48kba1quKDXkNo9J+R0J60Rw@mail.gmail.com>
	<965644111.51326703.1512392407093.JavaMail.zimbra@redhat.com>
	<63021232.51549606.1512407305566.JavaMail.zimbra@redhat.com>
Message-ID: <1651982661.53171374.1512605995711.JavaMail.zimbra@redhat.com>

> The current plan is to have a metrics-only agent run in the slave server. It
> will write a file indicating what its jmx exporter endpoint is. The host
> agent will probe a directory where the slave agents write their files - when
> new files are discovered, the agent will change its resource configuration
> "Metrics Endpoint" thus triggering an inventory update. When the server sees
> this update, it will write out new P scrape endpoint configs so P knows to
> scrape the slave endpoints, too.

I think I have domain mode working; at least, things look sane in the inventory UI and Prometheus.

https://github.com/hawkular/hawkular-commons/pull/202

https://github.com/hawkular/hawkular-agent/pull/423

If you build with those, then build hawkular-services with commons and agent SNAPSHOT deps (to pick up the new stuff), you should be able to see it working.

Run your new h-services build and prometheus from within your h-services directory:

hawkular-services/dist/target/hawkular-services-dist-1.0.0.Final-SNAPSHOT/bin/standalone.sh -b 0.0.0.0 -Dhawkular.agent.enabled=false
hawkular-services/dist/src/test/scripts/run-prometheus.sh

Then run a wildfly+agent in domain mode - your h-agent directory has a distro that mvn builds so that you can test. I run it on another box so my ports don't conflict with the h-services server:

cd hawkular-agent/hawkular-javaagent-wildfly-dist/target/hawkular-javaagent-wildfly-dist-2.0.0.Alpha1-SNAPSHOT/bin
export HAWKULAR_USER=jdoe
export HAWKULAR_PASSWORD=password
./domain.sh --host-config=host-hawkular.xml -Dhawkular.rest.url=http://your-h-services-server:8080 -Dhawkular.agent.metrics.host=your-local-host

Then just go to the inventory UI and poke around.

I'm sure there are things still not right (missing metrics, perhaps) but I suspect the rest of the work is just getting the inventory and metric configuration files to match what we want to collect.

From mwringe at redhat.com  Thu Dec  7 09:09:34 2017
From: mwringe at redhat.com (Matthew Wringe)
Date: Thu, 7 Dec 2017 09:09:34 -0500
Subject: [Hawkular-dev] OpenShift Deployment
In-Reply-To: <545952777.25428143.1512591462504.JavaMail.zimbra@redhat.com>
References: <CAPq8dB+NBXzSEAKa019utbeJ=dhyTFiMSw_tnMJo=SdviizzTQ@mail.gmail.com>
	<545952777.25428143.1512591462504.JavaMail.zimbra@redhat.com>
Message-ID: <CAPq8dBKdOwb3wunaXf_BEtSZdkR3=LRELXy+3UhoyA4O37TqKw@mail.gmail.com>

On Wed, Dec 6, 2017 at 3:17 PM, Viet Nguyen <vnguyen at redhat.com> wrote:

> Hi all,
>
> TL;DR: Technically it's ALL-IN-ONE POD but make HawkularService an option
> aka "sidecar" container
>

I believe the 'sidecar' container concept is more used for when you have a
main container and then a small container to add in some functionality (eg
auth proxy, metric agent, etc)

In our case it would just be a pod with two containers which are tightly
coupled.


>
> Would our team also look at providing Prometheus (without HS) as the
> defacto choice for OpenShift?
>

For Hawkular Services, we will have our own Prometheus which is private to
our needs. Someone will not be able to use our pod and optionally only use
Prometheus from it.


>
> What I'm proposing is still technically an ALL-IN-ONE pod option. However,
> instead of looking at (Prometheus + HS) as a monolithic solution we can
> position HS as an enhancement to the plain vanilla Prometheus.  This add-on
> sidecar[1] approach can satisfy both Middleware users and non-middleware
> community users who may not necessarily need HawkularServices.  Let's say I
> want to use library X and X only comes with X+Y (which will cost me CPU and
> RAM resources) I may be less inclined to use the library.
>

We are not entertaining this idea and conceptually its closer to having
Hawkular Services as the main container and p8s as the side car container.

If someone wants middleware monitoring, they have to use our pod with
Hawkular Services and p8s. Its important that we control how our own p8s
instance works and to prevent someone from modifying it to their own
purposes (it would be too difficult to handle all the different scenarios
here).

If someone wants p8s for something other than middleware monitoring, then
they will have to use a different p8s pod.


>
> [1] more on "sidecar" containers
> http://blog.kubernetes.io/2015/06/the-distributed-
> system-toolkit-patterns.html
>
>
> Viet
>
>
>
>
> ----- Original Message -----
> From: "Matthew Wringe" <mwringe at redhat.com>
> To: "Discussions around Hawkular development" <
> hawkular-dev at lists.jboss.org>
> Sent: Monday, November 27, 2017 7:38:50 AM
> Subject: [Hawkular-dev] OpenShift Deployment
>
> With the changes that are now going to include Prometheus, how do we want
> to deploy this in OpenShift?
>
> We can have a few options:
>
> ALL-IN-ONE CONTAINER
> We put both Hawkular Services and Prometheus in the same container.
>
> Pros:
> - easy to deploy in plain docker (but this doesn't appear to be a usecase
> we are targetting anyways)
> - shares the same network connection (even localhost) and ip address (eg
> but both services are on the different ports).
> - Does't require any special wiring of components.
> - Can share the same volume mount
> - version of components can't get out of sync.
>
> Cons:
> - workflow doesn't work nicely. Docker containers are meant to only run a
> single application and running two can cause problems. Eg lifecycle events
> would become tricky and require some hacks to get around things.
> - can't independently deploy things
> - can't reuse or share any existing Prometheus docker containers.
>
> ALL-IN-ONE POD
> Hawkular Services and Prometheus are in their own containers, but they are
> both deployed within the same pod.
>
> Pros:
> - shares the same network connection.
> - bound to the same machine (useful if sharing the same hostpath pv) and
> don' need to worry about external network configurations (eg firewalls
> between OpenShift nodes)
> - pvs can be shared or separate.
> - lifecycle events will work properly.
>
> Cons:
> - lifecycle hooks will mean that both containers will have to pass before
> either one will enter the ready state. So if Prometheus is failing for some
> reason, Hawkular Services will not be available under the service.
> - cannot independently update one container. If we need to deploy a new
> container we will need to bring down the whole pod.
> - are stuck with a 1:1 ratio between Hawkular Services and Prometheus
>
>
> SEPARATE PODS
> Hawkular Services and Prometheus have their own separate pods.
>
> Pros:
> - can independently run components and each component has its own separate
> lifecycle
> - if in the future we want to cluster Hawkular Services. this will make it
> a lot easier and will also allow for running an n:m ratio between Hawkular
> Services and Prometheus
> - probably the more 'correct' way to deploy things as we don't have a
> strong requirement for Hawkular Services and Prometheus to run together.
>
> Cons:
> - more complex wiring. We will need to have extra services and routes
> created to handle this. This mean more things running and more chances for
> things to go wrong. Also more things to configure
> - reusing a PV between Hawkular Services and Prometheus could be more
> challenging (especially if we are using hostpath pvs). Updating the
> Prometheus scrape endpoint may require a new component and container.
>
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20171207/23acf521/attachment-0001.html 

From vnguyen at redhat.com  Fri Dec  8 13:38:23 2017
From: vnguyen at redhat.com (Viet Nguyen)
Date: Fri, 8 Dec 2017 13:38:23 -0500 (EST)
Subject: [Hawkular-dev] OpenShift Deployment
In-Reply-To: <CAPq8dBKdOwb3wunaXf_BEtSZdkR3=LRELXy+3UhoyA4O37TqKw@mail.gmail.com>
References: <CAPq8dB+NBXzSEAKa019utbeJ=dhyTFiMSw_tnMJo=SdviizzTQ@mail.gmail.com>
	<545952777.25428143.1512591462504.JavaMail.zimbra@redhat.com>
	<CAPq8dBKdOwb3wunaXf_BEtSZdkR3=LRELXy+3UhoyA4O37TqKw@mail.gmail.com>
Message-ID: <493022633.25957055.1512758303968.JavaMail.zimbra@redhat.com>

Clarification:  I'm not suggesting an Assemble-Your-Own solution whereby the users independently install Prometheus then Hawkular Services.

Rather what I'm proposing is somewhat analogous to Fedora model.  We want to be the community leader in Prometheus on OpenShift and on top of that our expertise in Middleware monitoring brings values to Middleware users by providing our Prometheus++ solution (core Prometheus + Middleware monitoring enhancement). 

>If someone wants p8s for something other than middleware 
>monitoring, then they will have to use a different p8s pod. 

Let me flip that inside out if I may.  If someone wants to contribute to P8S other than Middleware monitoring, then they will have to go to a different project.  With that I feel we may be missing a huge opportunity here.  A large community of P8s users will go elsewhere to contribute.  We'll be on our own island.

I'm bringing up the sidecar approach because naturally it imposes loose-coupling between P8s and HS even when the two are in the same pod.  



Viet



----- Original Message -----
From: "Matthew Wringe" <mwringe at redhat.com>
To: "Discussions around Hawkular development" <hawkular-dev at lists.jboss.org>
Sent: Thursday, December 7, 2017 6:09:34 AM
Subject: Re: [Hawkular-dev] OpenShift Deployment

On Wed, Dec 6, 2017 at 3:17 PM, Viet Nguyen < vnguyen at redhat.com > wrote: 


>Hi all, 

>TL;DR: Technically it's ALL-IN-ONE POD but make HawkularService an option aka "sidecar" container 

I believe the 'sidecar' container concept is more used for when you have a main container and then a small container to add in some functionality (eg auth proxy, metric agent, etc) 

In our case it would just be a pod with two containers which are tightly coupled. 



>Would our team also look at providing Prometheus (without HS) as 
>the defacto choice for OpenShift? 

For Hawkular Services, we will have our own Prometheus which is private to our needs. Someone will not be able to use our pod and optionally only use Prometheus from it. 



>What I'm proposing is still technically an ALL-IN-ONE pod option.
> However, instead of looking at (Prometheus + HS) as a monolithic
>solution we can position HS as an enhancement to the plain vanilla 
>Prometheus. This add-on sidecar[1] approach can satisfy both 
>Middleware users and non-middleware community users who may not 
>necessarily need HawkularServices. Let's say I want to use library >X and X only comes with X+Y (which will cost me CPU and RAM 
>resources) I may be less inclined to use the library. 

We are not entertaining this idea and conceptually its closer to having Hawkular Services as the main container and p8s as the side car container. 

If someone wants middleware monitoring, they have to use our pod with Hawkular Services and p8s. Its important that we control how our own p8s instance works and to prevent someone from modifying it to their own purposes (it would be too difficult to handle all the different scenarios here). 

If someone wants p8s for something other than middleware monitoring, then they will have to use a different p8s pod. 



[1] more on "sidecar" containers 
http://blog.kubernetes.io/2015/06/the-distributed-system-toolkit-patterns.html 


Viet 




----- Original Message ----- 
From: "Matthew Wringe" < mwringe at redhat.com > 
To: "Discussions around Hawkular development" < hawkular-dev at lists.jboss.org > 
Sent: Monday, November 27, 2017 7:38:50 AM 
Subject: [Hawkular-dev] OpenShift Deployment 

With the changes that are now going to include Prometheus, how do we want to deploy this in OpenShift? 

We can have a few options: 

ALL-IN-ONE CONTAINER 
We put both Hawkular Services and Prometheus in the same container. 

Pros: 
- easy to deploy in plain docker (but this doesn't appear to be a usecase we are targetting anyways) 
- shares the same network connection (even localhost) and ip address (eg but both services are on the different ports). 
- Does't require any special wiring of components. 
- Can share the same volume mount 
- version of components can't get out of sync. 

Cons: 
- workflow doesn't work nicely. Docker containers are meant to only run a single application and running two can cause problems. Eg lifecycle events would become tricky and require some hacks to get around things. 
- can't independently deploy things 
- can't reuse or share any existing Prometheus docker containers. 

ALL-IN-ONE POD 
Hawkular Services and Prometheus are in their own containers, but they are both deployed within the same pod. 

Pros: 
- shares the same network connection. 
- bound to the same machine (useful if sharing the same hostpath pv) and don' need to worry about external network configurations (eg firewalls between OpenShift nodes) 
- pvs can be shared or separate. 
- lifecycle events will work properly. 

Cons: 
- lifecycle hooks will mean that both containers will have to pass before either one will enter the ready state. So if Prometheus is failing for some reason, Hawkular Services will not be available under the service. 
- cannot independently update one container. If we need to deploy a new container we will need to bring down the whole pod. 
- are stuck with a 1:1 ratio between Hawkular Services and Prometheus 


SEPARATE PODS 
Hawkular Services and Prometheus have their own separate pods. 

Pros: 
- can independently run components and each component has its own separate lifecycle 
- if in the future we want to cluster Hawkular Services. this will make it a lot easier and will also allow for running an n:m ratio between Hawkular Services and Prometheus 
- probably the more 'correct' way to deploy things as we don't have a strong requirement for Hawkular Services and Prometheus to run together. 

Cons: 
- more complex wiring. We will need to have extra services and routes created to handle this. This mean more things running and more chances for things to go wrong. Also more things to configure 
- reusing a PV between Hawkular Services and Prometheus could be more challenging (especially if we are using hostpath pvs). Updating the Prometheus scrape endpoint may require a new component and container. 

From mazz at redhat.com  Fri Dec  8 15:20:06 2017
From: mazz at redhat.com (John Mazzitelli)
Date: Fri, 8 Dec 2017 15:20:06 -0500 (EST)
Subject: [Hawkular-dev] starting wildfly+agent using new script
In-Reply-To: <2029263219.53586993.1512764264332.JavaMail.zimbra@redhat.com>
Message-ID: <1245749766.53587292.1512764406534.JavaMail.zimbra@redhat.com>

Once my PR [1] is merged, we'll have a script to run the wildfly+agent distro easier.

Get the wildfly+agent (build from here [2]) and run the start-with-hawkular.sh script passing in a few arguments.

Usage is:

Usage: ./start-with-hawkular.sh
  -d|--domain - to run in domain mode as opposed to standalone mode
  -s|--server <server-name> - the host where the Hawkular Server is running
  -u|--username <username> - the username to use when logging into the Hawkular Server
  -p|--password <password> - the username credentials
  [-h|--metrics-host <host/IP>] - the hostname to bind the metrics exporter

* -u and -p override environment variables HAWKULAR_USER and HAWKULAR_PASSWORD.
  You can set those environment variables as an alternative to using -u and -p.
* -h will default to '<your machine here>' if not specified.

--John Mazz

[1] https://github.com/hawkular/hawkular-agent/pull/423
[2] https://github.com/hawkular/hawkular-agent/tree/master/hawkular-javaagent-wildfly-dist

From mwringe at redhat.com  Fri Dec  8 15:54:14 2017
From: mwringe at redhat.com (Matthew Wringe)
Date: Fri, 8 Dec 2017 15:54:14 -0500
Subject: [Hawkular-dev] OpenShift Deployment
In-Reply-To: <493022633.25957055.1512758303968.JavaMail.zimbra@redhat.com>
References: <CAPq8dB+NBXzSEAKa019utbeJ=dhyTFiMSw_tnMJo=SdviizzTQ@mail.gmail.com>
	<545952777.25428143.1512591462504.JavaMail.zimbra@redhat.com>
	<CAPq8dBKdOwb3wunaXf_BEtSZdkR3=LRELXy+3UhoyA4O37TqKw@mail.gmail.com>
	<493022633.25957055.1512758303968.JavaMail.zimbra@redhat.com>
Message-ID: <CAPq8dBLHSq=LM=GU7smpA5gt+gvgk6LLaJFV_CSvyMEh1v9h_w@mail.gmail.com>

On Fri, Dec 8, 2017 at 1:38 PM, Viet Nguyen <vnguyen at redhat.com> wrote:

> Clarification:  I'm not suggesting an Assemble-Your-Own solution whereby
> the users independently install Prometheus then Hawkular Services.
>
> Rather what I'm proposing is somewhat analogous to Fedora model.  We want
> to be the community leader in Prometheus on OpenShift


No we don't. This is not the goal of Hawkular Services and has never been
our goal.

Hawkular Services is about being able to monitor and manage middleware
servers. Thats our purpose.

We are a consumer of Prometheus and using it as our metrics solution. In
the past this was done using Hawkular Metrics. And in the future we might
switch to some other metrics solution.

We are currently using Prometheus as a tool. That is it.

There are other teams working on the Prometheus experience in OpenShift,
and we will be working with them. We will be reusing as much as possible
their containers, scripts and configurations. Its these teams whose goal is
the community leadership of running Prometheus on OpenShift, not us.



> and on top of that our expertise in Middleware monitoring brings values to
> Middleware users by providing our Prometheus++ solution (core Prometheus +
> Middleware monitoring enhancement).
>
> >If someone wants p8s for something other than middleware
> >monitoring, then they will have to use a different p8s pod.
>
> Let me flip that inside out if I may.  If someone wants to contribute to
> P8S other than Middleware monitoring, then they will have to go to a
> different project.


Yes of course, they will go to the OpenShift team working directly with
Prometheus. Just as we are.


> With that I feel we may be missing a huge opportunity here.  A large
> community of P8s users will go elsewhere to contribute.  We'll be on our
> own island.
>
> I'm bringing up the sidecar approach because naturally it imposes
> loose-coupling between P8s and HS even when the two are in the same pod.


>
>
> Viet
>
>
>
> ----- Original Message -----
> From: "Matthew Wringe" <mwringe at redhat.com>
> To: "Discussions around Hawkular development" <
> hawkular-dev at lists.jboss.org>
> Sent: Thursday, December 7, 2017 6:09:34 AM
> Subject: Re: [Hawkular-dev] OpenShift Deployment
>
> On Wed, Dec 6, 2017 at 3:17 PM, Viet Nguyen < vnguyen at redhat.com > wrote:
>
>
> >Hi all,
>
> >TL;DR: Technically it's ALL-IN-ONE POD but make HawkularService an option
> aka "sidecar" container
>
> I believe the 'sidecar' container concept is more used for when you have a
> main container and then a small container to add in some functionality (eg
> auth proxy, metric agent, etc)
>
> In our case it would just be a pod with two containers which are tightly
> coupled.
>
>
>
> >Would our team also look at providing Prometheus (without HS) as
> >the defacto choice for OpenShift?
>
> For Hawkular Services, we will have our own Prometheus which is private to
> our needs. Someone will not be able to use our pod and optionally only use
> Prometheus from it.
>
>
>
> >What I'm proposing is still technically an ALL-IN-ONE pod option.
> > However, instead of looking at (Prometheus + HS) as a monolithic
> >solution we can position HS as an enhancement to the plain vanilla
> >Prometheus. This add-on sidecar[1] approach can satisfy both
> >Middleware users and non-middleware community users who may not
> >necessarily need HawkularServices. Let's say I want to use library >X and
> X only comes with X+Y (which will cost me CPU and RAM
> >resources) I may be less inclined to use the library.
>
> We are not entertaining this idea and conceptually its closer to having
> Hawkular Services as the main container and p8s as the side car container.
>
> If someone wants middleware monitoring, they have to use our pod with
> Hawkular Services and p8s. Its important that we control how our own p8s
> instance works and to prevent someone from modifying it to their own
> purposes (it would be too difficult to handle all the different scenarios
> here).
>
> If someone wants p8s for something other than middleware monitoring, then
> they will have to use a different p8s pod.
>
>
>
> [1] more on "sidecar" containers
> http://blog.kubernetes.io/2015/06/the-distributed-
> system-toolkit-patterns.html
>
>
> Viet
>
>
>
>
> ----- Original Message -----
> From: "Matthew Wringe" < mwringe at redhat.com >
> To: "Discussions around Hawkular development" <
> hawkular-dev at lists.jboss.org >
> Sent: Monday, November 27, 2017 7:38:50 AM
> Subject: [Hawkular-dev] OpenShift Deployment
>
> With the changes that are now going to include Prometheus, how do we want
> to deploy this in OpenShift?
>
> We can have a few options:
>
> ALL-IN-ONE CONTAINER
> We put both Hawkular Services and Prometheus in the same container.
>
> Pros:
> - easy to deploy in plain docker (but this doesn't appear to be a usecase
> we are targetting anyways)
> - shares the same network connection (even localhost) and ip address (eg
> but both services are on the different ports).
> - Does't require any special wiring of components.
> - Can share the same volume mount
> - version of components can't get out of sync.
>
> Cons:
> - workflow doesn't work nicely. Docker containers are meant to only run a
> single application and running two can cause problems. Eg lifecycle events
> would become tricky and require some hacks to get around things.
> - can't independently deploy things
> - can't reuse or share any existing Prometheus docker containers.
>
> ALL-IN-ONE POD
> Hawkular Services and Prometheus are in their own containers, but they are
> both deployed within the same pod.
>
> Pros:
> - shares the same network connection.
> - bound to the same machine (useful if sharing the same hostpath pv) and
> don' need to worry about external network configurations (eg firewalls
> between OpenShift nodes)
> - pvs can be shared or separate.
> - lifecycle events will work properly.
>
> Cons:
> - lifecycle hooks will mean that both containers will have to pass before
> either one will enter the ready state. So if Prometheus is failing for some
> reason, Hawkular Services will not be available under the service.
> - cannot independently update one container. If we need to deploy a new
> container we will need to bring down the whole pod.
> - are stuck with a 1:1 ratio between Hawkular Services and Prometheus
>
>
> SEPARATE PODS
> Hawkular Services and Prometheus have their own separate pods.
>
> Pros:
> - can independently run components and each component has its own separate
> lifecycle
> - if in the future we want to cluster Hawkular Services. this will make it
> a lot easier and will also allow for running an n:m ratio between Hawkular
> Services and Prometheus
> - probably the more 'correct' way to deploy things as we don't have a
> strong requirement for Hawkular Services and Prometheus to run together.
>
> Cons:
> - more complex wiring. We will need to have extra services and routes
> created to handle this. This mean more things running and more chances for
> things to go wrong. Also more things to configure
> - reusing a PV between Hawkular Services and Prometheus could be more
> challenging (especially if we are using hostpath pvs). Updating the
> Prometheus scrape endpoint may require a new component and container.
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20171208/5aaabd54/attachment-0001.html 

From theute at redhat.com  Tue Dec 12 01:54:12 2017
From: theute at redhat.com (Thomas Heute)
Date: Tue, 12 Dec 2017 07:54:12 +0100
Subject: [Hawkular-dev] OpenShift Deployment
In-Reply-To: <CAPq8dBLHSq=LM=GU7smpA5gt+gvgk6LLaJFV_CSvyMEh1v9h_w@mail.gmail.com>
References: <CAPq8dB+NBXzSEAKa019utbeJ=dhyTFiMSw_tnMJo=SdviizzTQ@mail.gmail.com>
	<545952777.25428143.1512591462504.JavaMail.zimbra@redhat.com>
	<CAPq8dBKdOwb3wunaXf_BEtSZdkR3=LRELXy+3UhoyA4O37TqKw@mail.gmail.com>
	<493022633.25957055.1512758303968.JavaMail.zimbra@redhat.com>
	<CAPq8dBLHSq=LM=GU7smpA5gt+gvgk6LLaJFV_CSvyMEh1v9h_w@mail.gmail.com>
Message-ID: <CA+aN=MqtddGHCFo2J31QPh6do+dkys5Hw4xXXvngzN-QOU-Shw@mail.gmail.com>

I see a lot of confusion on that thread.

Prometheus is a quite an internal detail for Hawkular Services/CloudForms

We happen to also have a team working on Prometheus directly working with
various consumers such as OpenShift and Fuse.

On Fri, Dec 8, 2017 at 9:54 PM, Matthew Wringe <mwringe at redhat.com> wrote:

> On Fri, Dec 8, 2017 at 1:38 PM, Viet Nguyen <vnguyen at redhat.com> wrote:
>
>> Clarification:  I'm not suggesting an Assemble-Your-Own solution whereby
>> the users independently install Prometheus then Hawkular Services.
>>
>> Rather what I'm proposing is somewhat analogous to Fedora model.  We want
>> to be the community leader in Prometheus on OpenShift
>
>
> No we don't. This is not the goal of Hawkular Services and has never been
> our goal.
>
> Hawkular Services is about being able to monitor and manage middleware
> servers. Thats our purpose.
>
> We are a consumer of Prometheus and using it as our metrics solution. In
> the past this was done using Hawkular Metrics. And in the future we might
> switch to some other metrics solution.
>
> We are currently using Prometheus as a tool. That is it.
>
> There are other teams working on the Prometheus experience in OpenShift,
> and we will be working with them. We will be reusing as much as possible
> their containers, scripts and configurations. Its these teams whose goal is
> the community leadership of running Prometheus on OpenShift, not us.
>
>
>
>> and on top of that our expertise in Middleware monitoring brings values
>> to Middleware users by providing our Prometheus++ solution (core Prometheus
>> + Middleware monitoring enhancement).
>>
>> >If someone wants p8s for something other than middleware
>> >monitoring, then they will have to use a different p8s pod.
>>
>> Let me flip that inside out if I may.  If someone wants to contribute to
>> P8S other than Middleware monitoring, then they will have to go to a
>> different project.
>
>
> Yes of course, they will go to the OpenShift team working directly with
> Prometheus. Just as we are.
>
>
>> With that I feel we may be missing a huge opportunity here.  A large
>> community of P8s users will go elsewhere to contribute.  We'll be on our
>> own island.
>>
>> I'm bringing up the sidecar approach because naturally it imposes
>> loose-coupling between P8s and HS even when the two are in the same pod.
>
>
>>
>>
>> Viet
>>
>>
>>
>> ----- Original Message -----
>> From: "Matthew Wringe" <mwringe at redhat.com>
>> To: "Discussions around Hawkular development" <
>> hawkular-dev at lists.jboss.org>
>> Sent: Thursday, December 7, 2017 6:09:34 AM
>> Subject: Re: [Hawkular-dev] OpenShift Deployment
>>
>> On Wed, Dec 6, 2017 at 3:17 PM, Viet Nguyen < vnguyen at redhat.com > wrote:
>>
>>
>> >Hi all,
>>
>> >TL;DR: Technically it's ALL-IN-ONE POD but make HawkularService an
>> option aka "sidecar" container
>>
>> I believe the 'sidecar' container concept is more used for when you have
>> a main container and then a small container to add in some functionality
>> (eg auth proxy, metric agent, etc)
>>
>> In our case it would just be a pod with two containers which are tightly
>> coupled.
>>
>>
>>
>> >Would our team also look at providing Prometheus (without HS) as
>> >the defacto choice for OpenShift?
>>
>> For Hawkular Services, we will have our own Prometheus which is private
>> to our needs. Someone will not be able to use our pod and optionally only
>> use Prometheus from it.
>>
>>
>>
>> >What I'm proposing is still technically an ALL-IN-ONE pod option.
>> > However, instead of looking at (Prometheus + HS) as a monolithic
>> >solution we can position HS as an enhancement to the plain vanilla
>> >Prometheus. This add-on sidecar[1] approach can satisfy both
>> >Middleware users and non-middleware community users who may not
>> >necessarily need HawkularServices. Let's say I want to use library >X
>> and X only comes with X+Y (which will cost me CPU and RAM
>> >resources) I may be less inclined to use the library.
>>
>> We are not entertaining this idea and conceptually its closer to having
>> Hawkular Services as the main container and p8s as the side car container.
>>
>> If someone wants middleware monitoring, they have to use our pod with
>> Hawkular Services and p8s. Its important that we control how our own p8s
>> instance works and to prevent someone from modifying it to their own
>> purposes (it would be too difficult to handle all the different scenarios
>> here).
>>
>> If someone wants p8s for something other than middleware monitoring, then
>> they will have to use a different p8s pod.
>>
>>
>>
>> [1] more on "sidecar" containers
>> http://blog.kubernetes.io/2015/06/the-distributed-system-
>> toolkit-patterns.html
>>
>>
>> Viet
>>
>>
>>
>>
>> ----- Original Message -----
>> From: "Matthew Wringe" < mwringe at redhat.com >
>> To: "Discussions around Hawkular development" <
>> hawkular-dev at lists.jboss.org >
>> Sent: Monday, November 27, 2017 7:38:50 AM
>> Subject: [Hawkular-dev] OpenShift Deployment
>>
>> With the changes that are now going to include Prometheus, how do we want
>> to deploy this in OpenShift?
>>
>> We can have a few options:
>>
>> ALL-IN-ONE CONTAINER
>> We put both Hawkular Services and Prometheus in the same container.
>>
>> Pros:
>> - easy to deploy in plain docker (but this doesn't appear to be a usecase
>> we are targetting anyways)
>> - shares the same network connection (even localhost) and ip address (eg
>> but both services are on the different ports).
>> - Does't require any special wiring of components.
>> - Can share the same volume mount
>> - version of components can't get out of sync.
>>
>> Cons:
>> - workflow doesn't work nicely. Docker containers are meant to only run a
>> single application and running two can cause problems. Eg lifecycle events
>> would become tricky and require some hacks to get around things.
>> - can't independently deploy things
>> - can't reuse or share any existing Prometheus docker containers.
>>
>> ALL-IN-ONE POD
>> Hawkular Services and Prometheus are in their own containers, but they
>> are both deployed within the same pod.
>>
>> Pros:
>> - shares the same network connection.
>> - bound to the same machine (useful if sharing the same hostpath pv) and
>> don' need to worry about external network configurations (eg firewalls
>> between OpenShift nodes)
>> - pvs can be shared or separate.
>> - lifecycle events will work properly.
>>
>> Cons:
>> - lifecycle hooks will mean that both containers will have to pass before
>> either one will enter the ready state. So if Prometheus is failing for some
>> reason, Hawkular Services will not be available under the service.
>> - cannot independently update one container. If we need to deploy a new
>> container we will need to bring down the whole pod.
>> - are stuck with a 1:1 ratio between Hawkular Services and Prometheus
>>
>>
>> SEPARATE PODS
>> Hawkular Services and Prometheus have their own separate pods.
>>
>> Pros:
>> - can independently run components and each component has its own
>> separate lifecycle
>> - if in the future we want to cluster Hawkular Services. this will make
>> it a lot easier and will also allow for running an n:m ratio between
>> Hawkular Services and Prometheus
>> - probably the more 'correct' way to deploy things as we don't have a
>> strong requirement for Hawkular Services and Prometheus to run together.
>>
>> Cons:
>> - more complex wiring. We will need to have extra services and routes
>> created to handle this. This mean more things running and more chances for
>> things to go wrong. Also more things to configure
>> - reusing a PV between Hawkular Services and Prometheus could be more
>> challenging (especially if we are using hostpath pvs). Updating the
>> Prometheus scrape endpoint may require a new component and container.
>> _______________________________________________
>> hawkular-dev mailing list
>> hawkular-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>>
>
>
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20171212/fb280b70/attachment.html 

From mithomps at redhat.com  Tue Dec 12 20:43:48 2017
From: mithomps at redhat.com (Mike Thompson)
Date: Tue, 12 Dec 2017 17:43:48 -0800
Subject: [Hawkular-dev] Hawkular Openshift Templates broken v3.7?
Message-ID: <CACms5tvYhzQPGPVEQB8v9Ut-moTja=YBY=EX6pJzcfTqO69DzQ@mail.gmail.com>

Do the Hawkular Openshift templates work for others(for running Hawkular
inside openshift)? I'm referring to:


   -
   https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_examples/files/examples/v3.7/cfme-templates/jboss-middleware-manager-pv-example.yaml
   -
   https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_examples/files/examples/v3.7/cfme-templates/jboss-middleware-manager-template.yaml

After copying to a local directory...

oc create -f ./jboss-middleware-manager-pv-example.yaml

*gives:*

The Template "" is invalid: metadata.name: Required value: name or
generateName is required
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20171212/ada529a6/attachment.html 

From mwringe at redhat.com  Wed Dec 13 14:06:12 2017
From: mwringe at redhat.com (Matthew Wringe)
Date: Wed, 13 Dec 2017 14:06:12 -0500
Subject: [Hawkular-dev] Pushing 'hawkular-1275' branches to master
Message-ID: <CAPq8dBJqYMQoSDaR1q-PMmyAmsg=W+iDT1EK+xkQcpmB86QTgg@mail.gmail.com>

Unless there is any objection, we would like to merge the hawkular-1275
branches to master and continue to work in the master branch.

Once we are in the master branch we will also be moving towards a one PR
per jira issue. The workflow in jira will require this and will
automatically close the jira when a PR is merged.

Thanks,

Matt Wringe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20171213/3b88117f/attachment-0001.html 

From mazz at redhat.com  Wed Dec 13 16:44:01 2017
From: mazz at redhat.com (John Mazzitelli)
Date: Wed, 13 Dec 2017 16:44:01 -0500 (EST)
Subject: [Hawkular-dev] problem with jmx exporter reading an attribute we
 need for availability
In-Reply-To: <1569993646.54365522.1513200995335.JavaMail.zimbra@redhat.com>
Message-ID: <1416207091.54374072.1513201441922.JavaMail.zimbra@redhat.com>

Ok, well, i hate to leave it like this but - this is so friggin' odd that I can only say "it is not my fault" :)

First - note someone changed jmx exporter code because they saw the same thing as we are - that's what this comment is about:

https://github.com/prometheus/jmx_exporter/blob/master/collector/src/main/java/io/prometheus/jmx/JmxScraper.java#L110

See: https://github.com/prometheus/jmx_exporter/issues/89

But the fix is not 100%.

If you grab these three files - you can see it yourself: https://github.com/jmazzitelli/test/tree/master/javaagent
(get Makefile and the two .java files and then run "make download-wildfly unzip-wildfly compile run")

You see i use the same API as the jmx exporter here: https://github.com/jmazzitelli/test/blob/master/javaagent/TestJavaAgent.java#L67

and that is what you will see:

16:31:05,290 INFO  [stdout] (Test Java Agent Thread) =============================================================
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) FIND INFORMATION ABOUT MBEAN: jboss.as:management-root=server
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) =============================================================
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) isRegistered:
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) true
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) getMBeanInfo:
16:31:05,291 INFO  [stdout] (Test Java Agent Thread)   description: The root node of the server-level management model.
16:31:05,291 INFO  [stdout] (Test Java Agent Thread)   #attributes: 19
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) getAttribute:
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) serverState=reload-required
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) queryNames:
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) []
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) queryMBeans:
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) []
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) queryNames(null, null):
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) FOUND IT: jboss.as:management-root=server
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) =============================================================


You will see SOME JMX APIs can see the MBean, queryMBeans and queryNames canNOT (note the empty arrays []).

But notice getMBeanInfo CAN see it - I can even get the attribute value from that! (you can see it is in reload-required state)

But again, queryMBeans returns nothing.

Oddly, queryNames(null, null) DOES return it in the list (see the "FOUND IT" line). It is only if I specifically ask for it does it fail in the query API.

The end result - JMX Exporter (at least sometimes) is not able to get "Server Availability" because it can't get this MBean.

For some odd reason it can get it sometimes - but it seems when it can't, it will never get it.

From mazz at redhat.com  Wed Dec 13 16:49:34 2017
From: mazz at redhat.com (John Mazzitelli)
Date: Wed, 13 Dec 2017 16:49:34 -0500 (EST)
Subject: [Hawkular-dev] problem with jmx exporter reading an attribute
 we need for availability
In-Reply-To: <1416207091.54374072.1513201441922.JavaMail.zimbra@redhat.com>
References: <1416207091.54374072.1513201441922.JavaMail.zimbra@redhat.com>
Message-ID: <1136951756.54379456.1513201774029.JavaMail.zimbra@redhat.com>

a temporary workaround: in the whitelist of the jmx exporter file, change:

"jboss.as:management-root=server",

to:

"jboss.as:*",

Once i do that, I see the jmx exporter emitting the server avail metric. of course, that sucks because it now tells jmx exporter to get every jboss.as MBean and that causes performance issues.

----- Original Message -----
> Ok, well, i hate to leave it like this but - this is so friggin' odd that I
> can only say "it is not my fault" :)
> 
> First - note someone changed jmx exporter code because they saw the same
> thing as we are - that's what this comment is about:
> 
> https://github.com/prometheus/jmx_exporter/blob/master/collector/src/main/java/io/prometheus/jmx/JmxScraper.java#L110
> 
> See: https://github.com/prometheus/jmx_exporter/issues/89
> 
> But the fix is not 100%.
> 
> If you grab these three files - you can see it yourself:
> https://github.com/jmazzitelli/test/tree/master/javaagent
> (get Makefile and the two .java files and then run "make download-wildfly
> unzip-wildfly compile run")
> 
> You see i use the same API as the jmx exporter here:
> https://github.com/jmazzitelli/test/blob/master/javaagent/TestJavaAgent.java#L67
> 
> and that is what you will see:
> 
> 16:31:05,290 INFO  [stdout] (Test Java Agent Thread)
> =============================================================
> 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) FIND INFORMATION ABOUT
> MBEAN: jboss.as:management-root=server
> 16:31:05,291 INFO  [stdout] (Test Java Agent Thread)
> =============================================================
> 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) isRegistered:
> 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) true
> 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) getMBeanInfo:
> 16:31:05,291 INFO  [stdout] (Test Java Agent Thread)   description: The root
> node of the server-level management model.
> 16:31:05,291 INFO  [stdout] (Test Java Agent Thread)   #attributes: 19
> 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) getAttribute:
> 16:31:05,291 INFO  [stdout] (Test Java Agent Thread)
> serverState=reload-required
> 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) queryNames:
> 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) []
> 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) queryMBeans:
> 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) []
> 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) queryNames(null, null):
> 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) FOUND IT:
> jboss.as:management-root=server
> 16:31:05,291 INFO  [stdout] (Test Java Agent Thread)
> =============================================================
> 
> 
> You will see SOME JMX APIs can see the MBean, queryMBeans and queryNames
> canNOT (note the empty arrays []).
> 
> But notice getMBeanInfo CAN see it - I can even get the attribute value from
> that! (you can see it is in reload-required state)
> 
> But again, queryMBeans returns nothing.
> 
> Oddly, queryNames(null, null) DOES return it in the list (see the "FOUND IT"
> line). It is only if I specifically ask for it does it fail in the query
> API.
> 
> The end result - JMX Exporter (at least sometimes) is not able to get "Server
> Availability" because it can't get this MBean.
> 
> For some odd reason it can get it sometimes - but it seems when it can't, it
> will never get it.
> 

From jstickle at redhat.com  Wed Dec 13 16:49:27 2017
From: jstickle at redhat.com (Julie Stickler)
Date: Wed, 13 Dec 2017 16:49:27 -0500
Subject: [Hawkular-dev] Hawkular Openshift Templates broken v3.7?
In-Reply-To: <CACms5tvYhzQPGPVEQB8v9Ut-moTja=YBY=EX6pJzcfTqO69DzQ@mail.gmail.com>
References: <CACms5tvYhzQPGPVEQB8v9Ut-moTja=YBY=EX6pJzcfTqO69DzQ@mail.gmail.com>
Message-ID: <CA+p6XVG2BF2K4===ezim-QDNXmnBqODJG=jaXK2YxkyyUpVapw@mail.gmail.com>

Sorry Mike, I just found your e-mail in my spam folder when I was looking
for Mazz's latest.

That's a known issue, fixed in this PR ->
https://github.com/hawkular/hawkular-services/pull/246

I believe that it's only the PV template that's missing metadata.

Ruben, we should probably either add that fix to
https://github.com/openshift/openshift-ansible/pull/5851/files  or start a
new PR.

Julie

JULIE STICKLER

TECHNICAL WRITER

Red Hat

<https://www.redhat.com/>

Westford ? 3S353

jstickle at redhat.com    T: 1(978)-399-0463-(812-0463)     IRC: jstickler

On Tue, Dec 12, 2017 at 8:43 PM, Mike Thompson <mithomps at redhat.com> wrote:

> Do the Hawkular Openshift templates work for others(for running Hawkular
> inside openshift)? I'm referring to:
>
>
>    - https://github.com/openshift/openshift-ansible/blob/master/
>    roles/openshift_examples/files/examples/v3.7/cfme-
>    templates/jboss-middleware-manager-pv-example.yaml
>    <https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_examples/files/examples/v3.7/cfme-templates/jboss-middleware-manager-pv-example.yaml>
>    - https://github.com/openshift/openshift-ansible/blob/master/
>    roles/openshift_examples/files/examples/v3.7/cfme-
>    templates/jboss-middleware-manager-template.yaml
>    <https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_examples/files/examples/v3.7/cfme-templates/jboss-middleware-manager-template.yaml>
>
> After copying to a local directory...
>
> oc create -f ./jboss-middleware-manager-pv-example.yaml
>
> *gives:*
>
> The Template "" is invalid: metadata.name: Required value: name or
> generateName is required
>
>
>
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20171213/ed4bc683/attachment.html 

From mithomps at redhat.com  Wed Dec 13 20:54:05 2017
From: mithomps at redhat.com (Mike Thompson)
Date: Wed, 13 Dec 2017 17:54:05 -0800
Subject: [Hawkular-dev] Hawkular Openshift Templates broken v3.7?
In-Reply-To: <CA+p6XVG2BF2K4===ezim-QDNXmnBqODJG=jaXK2YxkyyUpVapw@mail.gmail.com>
References: <CACms5tvYhzQPGPVEQB8v9Ut-moTja=YBY=EX6pJzcfTqO69DzQ@mail.gmail.com>
	<CA+p6XVG2BF2K4===ezim-QDNXmnBqODJG=jaXK2YxkyyUpVapw@mail.gmail.com>
Message-ID: <CACms5tturzWcwkvFTndEMOJ2FV6Shy3eC04eeX481V1qcmYuog@mail.gmail.com>

Thanks Julie for the confirmation!


On Wed, Dec 13, 2017 at 1:49 PM, Julie Stickler <jstickle at redhat.com> wrote:

> Sorry Mike, I just found your e-mail in my spam folder when I was looking
> for Mazz's latest.
>
> That's a known issue, fixed in this PR -> https://github.com/hawkular/
> hawkular-services/pull/246
>
> I believe that it's only the PV template that's missing metadata.
>
> Ruben, we should probably either add that fix to https://github.com/
> openshift/openshift-ansible/pull/5851/files  or start a new PR.
>
> Julie
>
> JULIE STICKLER
>
> TECHNICAL WRITER
>
> Red Hat
>
> <https://www.redhat.com/>
>
> Westford ? 3S353
>
> jstickle at redhat.com    T: 1(978)-399-0463-(812-0463)     IRC: jstickler
>
> On Tue, Dec 12, 2017 at 8:43 PM, Mike Thompson <mithomps at redhat.com>
> wrote:
>
>> Do the Hawkular Openshift templates work for others(for running Hawkular
>> inside openshift)? I'm referring to:
>>
>>
>>    - https://github.com/openshift/openshift-ansible/blob/master/r
>>    oles/openshift_examples/files/examples/v3.7/cfme-templates/
>>    jboss-middleware-manager-pv-example.yaml
>>    <https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_examples/files/examples/v3.7/cfme-templates/jboss-middleware-manager-pv-example.yaml>
>>    - https://github.com/openshift/openshift-ansible/blob/master/r
>>    oles/openshift_examples/files/examples/v3.7/cfme-templates/
>>    jboss-middleware-manager-template.yaml
>>    <https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_examples/files/examples/v3.7/cfme-templates/jboss-middleware-manager-template.yaml>
>>
>> After copying to a local directory...
>>
>> oc create -f ./jboss-middleware-manager-pv-example.yaml
>>
>> *gives:*
>>
>> The Template "" is invalid: metadata.name: Required value: name or
>> generateName is required
>>
>>
>>
>> _______________________________________________
>> hawkular-dev mailing list
>> hawkular-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>>
>>
>
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20171213/c0ee259c/attachment-0001.html 

From lponce at redhat.com  Thu Dec 14 09:04:05 2017
From: lponce at redhat.com (Lucas Ponce)
Date: Thu, 14 Dec 2017 15:04:05 +0100
Subject: [Hawkular-dev] problem with jmx exporter reading an attribute
 we need for availability
In-Reply-To: <1136951756.54379456.1513201774029.JavaMail.zimbra@redhat.com>
References: <1416207091.54374072.1513201441922.JavaMail.zimbra@redhat.com>
	<1136951756.54379456.1513201774029.JavaMail.zimbra@redhat.com>
Message-ID: <CAGUuq3UzDdqS2kFrWe2n==J+=_8G9RPSOcvdwZhnLWEmbvU5qQ@mail.gmail.com>

I will investigate more on this.

On Wed, Dec 13, 2017 at 10:49 PM, John Mazzitelli <mazz at redhat.com> wrote:

> a temporary workaround: in the whitelist of the jmx exporter file, change:
>
> "jboss.as:management-root=server",
>
> to:
>
> "jboss.as:*",
>
> Once i do that, I see the jmx exporter emitting the server avail metric.
> of course, that sucks because it now tells jmx exporter to get every
> jboss.as MBean and that causes performance issues.
>
> ----- Original Message -----
> > Ok, well, i hate to leave it like this but - this is so friggin' odd
> that I
> > can only say "it is not my fault" :)
> >
> > First - note someone changed jmx exporter code because they saw the same
> > thing as we are - that's what this comment is about:
> >
> > https://github.com/prometheus/jmx_exporter/blob/master/
> collector/src/main/java/io/prometheus/jmx/JmxScraper.java#L110
> >
> > See: https://github.com/prometheus/jmx_exporter/issues/89
> >
> > But the fix is not 100%.
> >
> > If you grab these three files - you can see it yourself:
> > https://github.com/jmazzitelli/test/tree/master/javaagent
> > (get Makefile and the two .java files and then run "make download-wildfly
> > unzip-wildfly compile run")
> >
> > You see i use the same API as the jmx exporter here:
> > https://github.com/jmazzitelli/test/blob/master/
> javaagent/TestJavaAgent.java#L67
> >
> > and that is what you will see:
> >
> > 16:31:05,290 INFO  [stdout] (Test Java Agent Thread)
> > =============================================================
> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) FIND INFORMATION
> ABOUT
> > MBEAN: jboss.as:management-root=server
> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread)
> > =============================================================
> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) isRegistered:
> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) true
> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) getMBeanInfo:
> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread)   description: The
> root
> > node of the server-level management model.
> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread)   #attributes: 19
> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) getAttribute:
> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread)
> > serverState=reload-required
> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) queryNames:
> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) []
> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) queryMBeans:
> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) []
> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) queryNames(null,
> null):
> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) FOUND IT:
> > jboss.as:management-root=server
> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread)
> > =============================================================
> >
> >
> > You will see SOME JMX APIs can see the MBean, queryMBeans and queryNames
> > canNOT (note the empty arrays []).
> >
> > But notice getMBeanInfo CAN see it - I can even get the attribute value
> from
> > that! (you can see it is in reload-required state)
> >
> > But again, queryMBeans returns nothing.
> >
> > Oddly, queryNames(null, null) DOES return it in the list (see the "FOUND
> IT"
> > line). It is only if I specifically ask for it does it fail in the query
> > API.
> >
> > The end result - JMX Exporter (at least sometimes) is not able to get
> "Server
> > Availability" because it can't get this MBean.
> >
> > For some odd reason it can get it sometimes - but it seems when it
> can't, it
> > will never get it.
> >
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20171214/a9885d19/attachment.html 

From jshaughn at redhat.com  Thu Dec 14 09:19:19 2017
From: jshaughn at redhat.com (Jay Shaughnessy)
Date: Thu, 14 Dec 2017 09:19:19 -0500
Subject: [Hawkular-dev] Pushing 'hawkular-1275' branches to master
In-Reply-To: <CAPq8dBJqYMQoSDaR1q-PMmyAmsg=W+iDT1EK+xkQcpmB86QTgg@mail.gmail.com>
References: <CAPq8dBJqYMQoSDaR1q-PMmyAmsg=W+iDT1EK+xkQcpmB86QTgg@mail.gmail.com>
Message-ID: <7713db5b-a361-f942-a0e4-d01dc34bfa36@redhat.com>


Note that this is for the server-side repos so that we can continue 
hawkular-1328 work from master:

 ? hawkular-commons
 ? hawkular-agent
 ? hawkular-services

The provider side repos (ruby/miq) will remain in the hawkular-1328 
branches hosted by Edgar's github account.

On 12/13/2017 2:06 PM, Matthew Wringe wrote:
> Unless there is any objection, we would like to merge the 
> hawkular-1275 branches to master and continue to work in the master 
> branch.
>
> Once we are in the master branch we will also be moving towards a one 
> PR per jira issue. The workflow in jira will require this and will 
> automatically close the jira when a PR is merged.
>
> Thanks,
>
> Matt Wringe
>
>
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20171214/54f53d10/attachment.html 

From mithomps at redhat.com  Thu Dec 14 10:35:09 2017
From: mithomps at redhat.com (Mike Thompson)
Date: Thu, 14 Dec 2017 07:35:09 -0800
Subject: [Hawkular-dev] Tip: Hawkular running in Openshift on OS/X
Message-ID: <CACms5tsvNh4DyrRk=B1r7Qi9qJik7yDBxLaPBkj+RbuKXwyhKw@mail.gmail.com>

This pertains only to Hawkular services running on Mac OS/X via Openshift.

There is a small but critical detail that *must* be adjusted before running
Hawkular in Openshift on OS/X. *Not doing so will result in an
endless(almost) looping and crashing of pods *with very little information
because of the pod logs/events crash very close to startup.

The trick here is to make sure that the OS/X docker has enough memory
because the default is only 2 Gb, which is not enough to run
Hawkular/Cassandra. To adjust this setting go into docker preferences and
find the *Advanced* tab. There you can adjust the Memory and CPU. The
memory is set initially at 2Gb which is not sufficient for
Hawkular/Cassandra -- *adjust this to 8 Gb or more*.

[image: Inline image 1]

The feel free to run Hawuklar/Cassandra:
`oc new-project mm-app;oc process -f hawkular-services-ephemeral.yaml | oc
create -f -`

Hawkular Services Ephemeral Template
<https://raw.githubusercontent.com/hawkular/hawkular-services/master/openshift/hawkular-services-ephemeral.yaml>

Special thanks to Matt Wringe and Matt Mahoney for their time and expertise
in diagnosing this issue.

[Jstickler, you may want to add this to documentation]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20171214/107d9c5f/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Advanced.jpg
Type: image/jpeg
Size: 57456 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/hawkular-dev/attachments/20171214/107d9c5f/attachment-0001.jpg 

From lponce at redhat.com  Thu Dec 14 11:18:41 2017
From: lponce at redhat.com (Lucas Ponce)
Date: Thu, 14 Dec 2017 17:18:41 +0100
Subject: [Hawkular-dev] problem with jmx exporter reading an attribute
 we need for availability
In-Reply-To: <CAGUuq3UzDdqS2kFrWe2n==J+=_8G9RPSOcvdwZhnLWEmbvU5qQ@mail.gmail.com>
References: <1416207091.54374072.1513201441922.JavaMail.zimbra@redhat.com>
	<1136951756.54379456.1513201774029.JavaMail.zimbra@redhat.com>
	<CAGUuq3UzDdqS2kFrWe2n==J+=_8G9RPSOcvdwZhnLWEmbvU5qQ@mail.gmail.com>
Message-ID: <CAGUuq3UAcSy6c1xc621smAPdwR97OcxH1fVW+DKmwm+LAjEPnA@mail.gmail.com>

Ok, got it.

It's filtered inside wildfly-core.

https://github.com/wildfly/wildfly-core/blob/3.0.8.Final/jmx/src/main/java/org/jboss/as/jmx/model/ModelControllerMBeanHelper.java#L145

So, basically, you can query any MBean but the "java.as:
management-root=server".

I will continue if we can workaround it in a way that doesn't mean to query
all.

(Perhaps to add a corner case in our custom agent).

On Thu, Dec 14, 2017 at 3:04 PM, Lucas Ponce <lponce at redhat.com> wrote:

> I will investigate more on this.
>
> On Wed, Dec 13, 2017 at 10:49 PM, John Mazzitelli <mazz at redhat.com> wrote:
>
>> a temporary workaround: in the whitelist of the jmx exporter file, change:
>>
>> "jboss.as:management-root=server",
>>
>> to:
>>
>> "jboss.as:*",
>>
>> Once i do that, I see the jmx exporter emitting the server avail metric.
>> of course, that sucks because it now tells jmx exporter to get every
>> jboss.as MBean and that causes performance issues.
>>
>> ----- Original Message -----
>> > Ok, well, i hate to leave it like this but - this is so friggin' odd
>> that I
>> > can only say "it is not my fault" :)
>> >
>> > First - note someone changed jmx exporter code because they saw the same
>> > thing as we are - that's what this comment is about:
>> >
>> > https://github.com/prometheus/jmx_exporter/blob/master/colle
>> ctor/src/main/java/io/prometheus/jmx/JmxScraper.java#L110
>> >
>> > See: https://github.com/prometheus/jmx_exporter/issues/89
>> >
>> > But the fix is not 100%.
>> >
>> > If you grab these three files - you can see it yourself:
>> > https://github.com/jmazzitelli/test/tree/master/javaagent
>> > (get Makefile and the two .java files and then run "make
>> download-wildfly
>> > unzip-wildfly compile run")
>> >
>> > You see i use the same API as the jmx exporter here:
>> > https://github.com/jmazzitelli/test/blob/master/javaagent/
>> TestJavaAgent.java#L67
>> >
>> > and that is what you will see:
>> >
>> > 16:31:05,290 INFO  [stdout] (Test Java Agent Thread)
>> > =============================================================
>> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) FIND INFORMATION
>> ABOUT
>> > MBEAN: jboss.as:management-root=server
>> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread)
>> > =============================================================
>> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) isRegistered:
>> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) true
>> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) getMBeanInfo:
>> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread)   description: The
>> root
>> > node of the server-level management model.
>> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread)   #attributes: 19
>> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) getAttribute:
>> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread)
>> > serverState=reload-required
>> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) queryNames:
>> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) []
>> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) queryMBeans:
>> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) []
>> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) queryNames(null,
>> null):
>> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread) FOUND IT:
>> > jboss.as:management-root=server
>> > 16:31:05,291 INFO  [stdout] (Test Java Agent Thread)
>> > =============================================================
>> >
>> >
>> > You will see SOME JMX APIs can see the MBean, queryMBeans and queryNames
>> > canNOT (note the empty arrays []).
>> >
>> > But notice getMBeanInfo CAN see it - I can even get the attribute value
>> from
>> > that! (you can see it is in reload-required state)
>> >
>> > But again, queryMBeans returns nothing.
>> >
>> > Oddly, queryNames(null, null) DOES return it in the list (see the
>> "FOUND IT"
>> > line). It is only if I specifically ask for it does it fail in the query
>> > API.
>> >
>> > The end result - JMX Exporter (at least sometimes) is not able to get
>> "Server
>> > Availability" because it can't get this MBean.
>> >
>> > For some odd reason it can get it sometimes - but it seems when it
>> can't, it
>> > will never get it.
>> >
>> _______________________________________________
>> hawkular-dev mailing list
>> hawkular-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20171214/469a9959/attachment.html 

From jshaughn at redhat.com  Fri Dec 15 10:55:52 2017
From: jshaughn at redhat.com (Jay Shaughnessy)
Date: Fri, 15 Dec 2017 10:55:52 -0500
Subject: [Hawkular-dev] Pushing 'hawkular-1275' branches to master
In-Reply-To: <0c0ee94d-4584-4ed3-39aa-225565a7e2b0@redhat.com>
References: <CAPq8dBJqYMQoSDaR1q-PMmyAmsg=W+iDT1EK+xkQcpmB86QTgg@mail.gmail.com>
	<0c0ee94d-4584-4ed3-39aa-225565a7e2b0@redhat.com>
Message-ID: <2600047a-a96f-c3c6-eba5-65a56039674b@redhat.com>


At this point please refrain from generating PRs against the 
hawkular-1275 branches.? We are in the process of migrating to master.? 
The Hawkular Commons PR is generated, after it is merged we can merge 
Hawkular Agent, and then Hawkular Services.? It should be completed no 
later than Monday.

On 12/14/2017 9:18 AM, Jay Shaughnessy wrote:
>
> Note that this is for the server-side repos:
> ? hawkular-commons
> ? hawkular-agent
> ? hawkular-services
>
> The provider side repos (ruby/miq) will remain in the hawkular-1328 
> branches hosted by Edgar's github account.
>
> On 12/13/2017 2:06 PM, Matthew Wringe wrote:
>> Unless there is any objection, we would like to merge the 
>> hawkular-1275 branches to master and continue to work in the master 
>> branch.
>>
>> Once we are in the master branch we will also be moving towards a one 
>> PR per jira issue. The workflow in jira will require this and will 
>> automatically close the jira when a PR is merged.
>>
>> Thanks,
>>
>> Matt Wringe
>>
>>
>> _______________________________________________
>> hawkular-dev mailing list
>> hawkular-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20171215/3c51a517/attachment.html 

From jshaughn at redhat.com  Tue Dec 19 09:40:55 2017
From: jshaughn at redhat.com (Jay Shaughnessy)
Date: Tue, 19 Dec 2017 09:40:55 -0500
Subject: [Hawkular-dev] Pushing 'hawkular-1275' branches to master (Done)
In-Reply-To: <3c4d92f9-fb9e-d22b-2964-10d19ba79276@redhat.com>
References: <CAPq8dBJqYMQoSDaR1q-PMmyAmsg=W+iDT1EK+xkQcpmB86QTgg@mail.gmail.com>
	<0c0ee94d-4584-4ed3-39aa-225565a7e2b0@redhat.com>
	<3c4d92f9-fb9e-d22b-2964-10d19ba79276@redhat.com>
Message-ID: <bb9884ef-3d4d-48c3-8ba4-31b1eff1ec41@redhat.com>


Note that the master branches now contain the Prometheus-based 
components for:

 ? hawkular-commons
 ? hawkular-agent
 ? hawkular-services

The hawkular-1275 branches are no longer in use.? The proper Services 
docker container label is now 'latest'.


On 12/15/2017 10:55 AM, Jay Shaughnessy wrote:
>
> At this point please refrain from generating PRs against the 
> hawkular-1275 branches.? We are in the process of migrating to 
> master.? The Hawkular Commons PR is generated, after it is merged we 
> can merge Hawkular Agent, and then Hawkular Services.? It should be 
> completed no later than Monday.
>
> On 12/14/2017 9:18 AM, Jay Shaughnessy wrote:
>>
>> Note that this is for the server-side repos:
>> ? hawkular-commons
>> ? hawkular-agent
>> ? hawkular-services
>>
>> The provider side repos (ruby/miq) will remain in the hawkular-1328 
>> branches hosted by Edgar's github account.
>>
>> On 12/13/2017 2:06 PM, Matthew Wringe wrote:
>>> Unless there is any objection, we would like to merge the 
>>> hawkular-1275 branches to master and continue to work in the master 
>>> branch.
>>>
>>> Once we are in the master branch we will also be moving towards a 
>>> one PR per jira issue. The workflow in jira will require this and 
>>> will automatically close the jira when a PR is merged.
>>>
>>> Thanks,
>>>
>>> Matt Wringe
>>>
>>>
>>> _______________________________________________
>>> hawkular-dev mailing list
>>> hawkular-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20171219/b399f77b/attachment.html