[Hawkular-dev] need discussion around how we are going to do domain support

[John Mazzitelli]

> > Spent time trying to figure out how to support WildFly domain mode with
> > the new stuff. It's not going well. Next week I will need to have some
> > discussions on how we want to do this.
> >
> > The issues with domain mode (I'll try to be short):
> >
> > 1) host controllers do not emit JMX metrics for slave servers - just not
> > implemented in WildFly. Need to go to slave servers for their own JMX
> > metrics.
> > 2) slave servers do not have managed interfaces available remotely (can't
> > connect to slaves over remote management interface - so agent can't get any
> > inventory from slaves - have to get all inventory from host controller)
> >
> > This means our agent in the host controller needs to collect all inventory
> > for both master host controller and slave servers.
> >
> > But we need the slave servers to have a metrics endpoint because its only
> > in the slaves where we can get JMX metrics (remember, host controller can't
> > give us that, we must scrape the slave servers for the JMX metrics).
> >
> > If we do not have our agent in slave server (why bother if we can't
> > connect to it over DMR API to get inventory?), we still need to somehow get
> > the P8s JMX Exporter installed in the slaves. But if we just use the raw
> > jmx exporter agent, how do we tell our h-services server/P8s server about
> > the new endpoint that needs to be scraped? And how do we get the jmx
> > exporter yml config to install there?
> >
> > So I suppose we should put our agent in a kind of "metrics only" mode in
> > all slave servers so it can expose the JMX exporter and pull down the
> > correct jmx exporter yml from h-services server and have it tell the server
> > to add the scrape endpoint to p8s.
> >
> > But because we aren't getting inventory from slave servers, how can our
> > agent tell the server to add the scrape endpoint? Our current
> > implementation only adds scrape endpoints to our P8s server when new agents
> > go into inventory. Workaround: either have agent store a small inventory
> > (just the agent resource itself) which triggers the new scrape endpoint
> > addition, or add a inventory REST endpoint for the agent to call to add the
> > new scrape endpoint manually.
> >
> > OK, assume we have all this. How do we link the JMX metrics getting
> > collected from one feed (the agent in the slave server) to the inventory
> > from another feed (the agent in the host controller). Right now, it is
> > assumed the inventory metadata from feed "A" is matched with metrics from
> > the same feed. Now that we have to break that link (feed A's inventory
> > refers to feed B's metrics) we need to figure out how to fix this.
> >
> > There are other ancillary issues - like how do I get the correct metadata
> > defined for host controller so it can match resources/metrics from the
> > slaves. I assume that will be "implementation details."
> >
> > I'm sure this sounds like gibberish, but that's how convoluted supporting
> > domain mode is going to be.
> >
> 
> Mazz,
> 
> What about to promote the host controller agent as the responsible for
> everything ?
> 
> - Host controller agent will be responsible to collect info of the domain
> (slave servers) and write into the inventory (Domain, Server, etc).
> - Slave server can have a "metrics-only" endpoint (perhaps same agent,
> perhaps a *-domain.jar if we want to simplify things).
> - Host controller agent can proxy slave metrics endpoint, so we can control
> endpoints like /metrics-domain/<slave>/metrics and that is what inventory
> uses to create the endpoint.
> - Host controller will expose in a proxy way, the metrics endpoint for the
> slaves.
> 
> This approach focus the complixity in the host controller, but let to not
> have exceptions and corner use cases in the whole system, the benefit I see
> is that from MiQ and Hawkular Services we can try to maintain and uniform
> way.
> 
> So, as you point Host Controllers needs to know the topology of the domain,
> and each slave should have a local /metrics which can be proxied from the
> host controller main endpoint.
> On this case, both Host Controllers and Slaves will share same feed,
> because, they are using the same agent, and that won't break any design use
> case.
> 
> Also, I think that this proxy has a minimal technical complexity, as the
> host controller should have all info to expose that.


This has some promise. Just need to think about how to configure this proxy and implement it without having to rip apart too much of the internals of the agent code.

Thinking out loud:

We add "proxies" section to the agent config's metrics-exporters:

metrics-exporter:
  enabled: true
  host: ${hawkular.agent.metrics.host:127.0.0.1}
  port: ${hawkular.agent.metrics.port:9779}
  config-dir: ${jboss.server.config.dir}
  config-file: WF10
  # HERE IS THE NEW SECTION
  proxies:
  - path: slave1/metrics
    host: 127.0.0.1
    port: 9780
  - path: slave2/metrics
    host: 127.0.0.1
    port: 9781
  - path: slave3/metrics
    host: 127.0.0.1
    port: 9782

We would have to enhance our jmx exporter wrapper code so it supports those extra endpoints (it would simply just pass-through request/responses to those different endpoints).

The slave agent configs would have their jmx exporter sections the same as always and bind to localhost:

metrics-exporter:
  enabled: true
  host: 127.0.0.1
  port: ${hawkular.agent.metrics.port:9779}
  config-dir: ${jboss.server.config.dir}
  config-file: WF10

The nice thing here is they only need to expose the metric data on the loopback address, 127.0.0.1, since the host controller agent will be on the same box - no need to open this up to the wider network.

The bad thing is the proxies section requires you to know how many slaves there are and what their ports are going to be. There is no way to know that ahead of time (this is all encoded in the host controller's config, and who knows what the user wants to use - you can start host controllers with --host-config option to specify a custom .xml). And besides, new slaves can be added to the host controller at any time.

So it seems there needs to be some kind of auto-discovery of proxies, not a fixed configuration of them. Maybe we have a section that tells the agent to check for metric endpoints to be proxied?

  proxies:
  - host: 127.0.0.1
    port-range: 9780-9880

But if we have auto-discover like that, what is the proxy /path that we need to tell Prometheus to scrape? We need to distinguish it somehow so P knows "to get slave #1, I scrape host:9779/slave1/metrics and to scrape slave #2 its host:9779/slave2/metrics the host controller its just host:9779/metrics ??

Maybe we tell the host controller agent to talk to the slave agent somehow? Over remote JMX is an interesting idea (the agent already has its own JMX MBean, as long as we can talk to it over remote JMX, we can have the host agent ask the slave agent for information about its metrics endpoint ("what is your host/port of /metrics endpoint?". We could possibly then associate it with the name of the slave and use that as the proxy path?

We could then add some kind of "flag" or metadata on the "Domain WildFly Server" resource to say, "this resource has an agent whose metrics endpoint we want to proxy - go ask this agent for its details". So when the host agent runs its normal inventory scan, we can add special code to say, "when you hit a resource that has this special proxy metadata, you need to proxy that resource's agent's metrics endpoint." Because the Domain WildFly Server is a DMR resource - there is no associated JMX Server information with it. We would need to configure that somehow.

  - name: Domain WildFly Server
    resource-name-template: "%-"
    path: "/server=*"
    parents:
    - Domain Host
    proxy-metrics: true

Oh, BTW, because the agent's remote JMX client is the jolokia client, we need to install a jolokia -javaagent in the slave server along with our agent. We'd need to add some additional information there so the host agent knows all the details so it can connect to the JMX server where the resource is hosted. How does it know the jolokia URL of the slave server?

See how this snowballs? :-/