[Hawkular-dev] major change in the way we will have to monitor domain mode

[John Mazzitelli]

I'm looking at how domain mode monitoring is going to work now that we are moving toward the Prometheus / jmx-exporter way of doing things.

Turns out, due to the way WildFly domain mode exposes things via JMX (rather, I should say, *doesn't* expose things) compared to DMR, I think we are going to have to require the hawkular agent to be deployed to all slave servers, in addition to the master host controller.

In domain mode, you can access all the child slave servers (the individual servers in all server groups) via the host controller's DMR tree - so we only needed one agent on the host controller to monitor the domain. For example, to see a metric for a deployment on "server-one" that is managed by host controller "master", I can simply ask the host controller for it:

/host=master/server=server-one/deployment=test-simple-1.0.0.war/subsystem=undertow/:read-attribute(name=active-sessions)
{
    "outcome" => "success",
    "result" => 0
}

The nice thing about WildFly's management DMR interface is that the domain mode tree is identical to the standalone tree with the exception that on the host controller, you simply prepend the host/server identifiers to the DMR path. So, for a standalone WildFly, a DMR path for the active-sessions metric for my test-simple application would be:

/deployment=test-simple-1.0.0.war/subsystem=undertow/:read-attribute(name=active-sessions)

If I want to ask the host controller to give me the exact same metric from that server, I simply prepend the host/server name in the DMR path:

/host=master/server=server-one/deployment=test-simple-1.0.0.war/subsystem=undertow/:read-attribute(name=active-sessions)

The agent has knowledge of this pattern; knowing this pattern we can cleverly configure the metadata so we can share types across domain and standalone for inventory discovery.

The problem: WildFly does not expose its JMX MBeans in an equally clever way. I do not see MBeans that provide metrics for the slave servers.

For example, in JMX, I see WildFly host controller has this MBean:

jboss.as:host=master,server=server-one

Looks very analogous to the DMR resource named /host=master/server=server-one resource ... right??

Well, there are no deployments associated with this MBean. You would think (following the DMR pattern) that there would be an MBean named:

jboss.as:host=master,server=server-one,deployment=test-simple-1.0.0.war

But there is not. Nor is there:

jboss.as:host=master,server=server-one,deployment=test-simple-1.0.0.war,subsystem=undertow

which is where you would expect that server's web subsystem metrics to be (if it were to follow the same DMR pattern). But, again, this doesn't exist.

I can't find JMX MBeans for anything related to the individual slave servers (not just deployments).

In short, I do not believe the Prometheus JMX Exporter can be used to collect metrics for all slave servers in a domain if that JMX Exporter is simply installed on a host controller. This is because the host controller's JMX MBeans do not expose metric data for individual slave servers.

We would have to have our agent in each slave server (which are just standalone servers - so the agent would be as if it is monitoring a standalone server). We could have an agent in the host controller, too, but it would only be responsible for monitoring/managing the host controller itself.

[this message was sent on Tuesday, November 21, 2017 at 10:11 PM EST]