[Hawkular-dev] need discussion around how we are going to do domain support

[Lucas Ponce]

On Mon, Dec 4, 2017 at 12:09 PM, John Mazzitelli <mazz at redhat.com> wrote:

> > > Spent time trying to figure out how to support WildFly domain mode with
> > > the new stuff. It's not going well. Next week I will need to have some
> > > discussions on how we want to do this.
> > >
> > > The issues with domain mode (I'll try to be short):
> > >
> > > 1) host controllers do not emit JMX metrics for slave servers - just
> not
> > > implemented in WildFly. Need to go to slave servers for their own JMX
> > > metrics.
> > > 2) slave servers do not have managed interfaces available remotely
> (can't
> > > connect to slaves over remote management interface - so agent can't
> get any
> > > inventory from slaves - have to get all inventory from host controller)
> > >
> > > This means our agent in the host controller needs to collect all
> inventory
> > > for both master host controller and slave servers.
> > >
> > > But we need the slave servers to have a metrics endpoint because its
> only
> > > in the slaves where we can get JMX metrics (remember, host controller
> can't
> > > give us that, we must scrape the slave servers for the JMX metrics).
> > >
> > > If we do not have our agent in slave server (why bother if we can't
> > > connect to it over DMR API to get inventory?), we still need to
> somehow get
> > > the P8s JMX Exporter installed in the slaves. But if we just use the
> raw
> > > jmx exporter agent, how do we tell our h-services server/P8s server
> about
> > > the new endpoint that needs to be scraped? And how do we get the jmx
> > > exporter yml config to install there?
> > >
> > > So I suppose we should put our agent in a kind of "metrics only" mode
> in
> > > all slave servers so it can expose the JMX exporter and pull down the
> > > correct jmx exporter yml from h-services server and have it tell the
> server
> > > to add the scrape endpoint to p8s.
> > >
> > > But because we aren't getting inventory from slave servers, how can our
> > > agent tell the server to add the scrape endpoint? Our current
> > > implementation only adds scrape endpoints to our P8s server when new
> agents
> > > go into inventory. Workaround: either have agent store a small
> inventory
> > > (just the agent resource itself) which triggers the new scrape endpoint
> > > addition, or add a inventory REST endpoint for the agent to call to
> add the
> > > new scrape endpoint manually.
> > >
> > > OK, assume we have all this. How do we link the JMX metrics getting
> > > collected from one feed (the agent in the slave server) to the
> inventory
> > > from another feed (the agent in the host controller). Right now, it is
> > > assumed the inventory metadata from feed "A" is matched with metrics
> from
> > > the same feed. Now that we have to break that link (feed A's inventory
> > > refers to feed B's metrics) we need to figure out how to fix this.
> > >
> > > There are other ancillary issues - like how do I get the correct
> metadata
> > > defined for host controller so it can match resources/metrics from the
> > > slaves. I assume that will be "implementation details."
> > >
> > > I'm sure this sounds like gibberish, but that's how convoluted
> supporting
> > > domain mode is going to be.
> > >
> >
> > Mazz,
> >
> > What about to promote the host controller agent as the responsible for
> > everything ?
> >
> > - Host controller agent will be responsible to collect info of the domain
> > (slave servers) and write into the inventory (Domain, Server, etc).
> > - Slave server can have a "metrics-only" endpoint (perhaps same agent,
> > perhaps a *-domain.jar if we want to simplify things).
> > - Host controller agent can proxy slave metrics endpoint, so we can
> control
> > endpoints like /metrics-domain/<slave>/metrics and that is what
> inventory
> > uses to create the endpoint.
> > - Host controller will expose in a proxy way, the metrics endpoint for
> the
> > slaves.
> >
> > This approach focus the complixity in the host controller, but let to not
> > have exceptions and corner use cases in the whole system, the benefit I
> see
> > is that from MiQ and Hawkular Services we can try to maintain and uniform
> > way.
> >
> > So, as you point Host Controllers needs to know the topology of the
> domain,
> > and each slave should have a local /metrics which can be proxied from the
> > host controller main endpoint.
> > On this case, both Host Controllers and Slaves will share same feed,
> > because, they are using the same agent, and that won't break any design
> use
> > case.
> >
> > Also, I think that this proxy has a minimal technical complexity, as the
> > host controller should have all info to expose that.
>
>
> This has some promise. Just need to think about how to configure this
> proxy and implement it without having to rip apart too much of the
> internals of the agent code.
>
> Thinking out loud:
>
> We add "proxies" section to the agent config's metrics-exporters:
>
> metrics-exporter:
>   enabled: true
>   host: ${hawkular.agent.metrics.host:127.0.0.1}
>   port: ${hawkular.agent.metrics.port:9779}
>   config-dir: ${jboss.server.config.dir}
>   config-file: WF10
>   # HERE IS THE NEW SECTION
>   proxies:
>   - path: slave1/metrics
>     host: 127.0.0.1
>     port: 9780
>   - path: slave2/metrics
>     host: 127.0.0.1
>     port: 9781
>   - path: slave3/metrics
>     host: 127.0.0.1
>     port: 9782
>
> We would have to enhance our jmx exporter wrapper code so it supports
> those extra endpoints (it would simply just pass-through request/responses
> to those different endpoints).
>
> The slave agent configs would have their jmx exporter sections the same as
> always and bind to localhost:
>
> metrics-exporter:
>   enabled: true
>   host: 127.0.0.1
>   port: ${hawkular.agent.metrics.port:9779}
>   config-dir: ${jboss.server.config.dir}
>   config-file: WF10
>
> The nice thing here is they only need to expose the metric data on the
> loopback address, 127.0.0.1, since the host controller agent will be on the
> same box - no need to open this up to the wider network.
>
> The bad thing is the proxies section requires you to know how many slaves
> there are and what their ports are going to be. There is no way to know
> that ahead of time (this is all encoded in the host controller's config,
> and who knows what the user wants to use - you can start host controllers
> with --host-config option to specify a custom .xml). And besides, new
> slaves can be added to the host controller at any time.
>

Wait a minute. Metrics aside, isn't this a similar problem for the pure
inventory ?

If user starts a new slave sever dynamically, HostController agent will
detect that and add it to the inventory. Or is this process works
differently ?

If that is the case, is there any metadata we can use to pass this info ?

I guess next questions are also related to this.




> So it seems there needs to be some kind of auto-discovery of proxies, not
> a fixed configuration of them. Maybe we have a section that tells the agent
> to check for metric endpoints to be proxied?
>
>   proxies:
>   - host: 127.0.0.1
>     port-range: 9780-9880
>
> But if we have auto-discover like that, what is the proxy /path that we
> need to tell Prometheus to scrape? We need to distinguish it somehow so P
> knows "to get slave #1, I scrape host:9779/slave1/metrics and to scrape
> slave #2 its host:9779/slave2/metrics the host controller its just
> host:9779/metrics ??
>
> Maybe we tell the host controller agent to talk to the slave agent
> somehow? Over remote JMX is an interesting idea (the agent already has its
> own JMX MBean, as long as we can talk to it over remote JMX, we can have
> the host agent ask the slave agent for information about its metrics
> endpoint ("what is your host/port of /metrics endpoint?". We could possibly
> then associate it with the name of the slave and use that as the proxy path?
>
> We could then add some kind of "flag" or metadata on the "Domain WildFly
> Server" resource to say, "this resource has an agent whose metrics endpoint
> we want to proxy - go ask this agent for its details". So when the host
> agent runs its normal inventory scan, we can add special code to say, "when
> you hit a resource that has this special proxy metadata, you need to proxy
> that resource's agent's metrics endpoint." Because the Domain WildFly
> Server is a DMR resource - there is no associated JMX Server information
> with it. We would need to configure that somehow.
>
>   - name: Domain WildFly Server
>     resource-name-template: "%-"
>     path: "/server=*"
>     parents:
>     - Domain Host
>     proxy-metrics: true
>
> Oh, BTW, because the agent's remote JMX client is the jolokia client, we
> need to install a jolokia -javaagent in the slave server along with our
> agent. We'd need to add some additional information there so the host agent
> knows all the details so it can connect to the JMX server where the
> resource is hosted. How does it know the jolokia URL of the slave server?
>
> See how this snowballs? :-/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20171204/8bb17432/attachment.html