I'm looking at how domain mode monitoring is going to work now that we are moving
toward the Prometheus / jmx-exporter way of doing things.
Turns out, due to the way WildFly domain mode exposes things via JMX (rather, I should
say, *doesn't* expose things) compared to DMR, I think we are going to have to require
the hawkular agent to be deployed to all slave servers, in addition to the master host
controller.
In domain mode, you can access all the child slave servers (the individual servers in all
server groups) via the host controller's DMR tree - so we only needed one agent on the
host controller to monitor the domain. For example, to see a metric for a deployment on
"server-one" that is managed by host controller "master", I can simply
ask the host controller for it:
/host=master/server=server-one/deployment=test-simple-1.0.0.war/subsystem=undertow/:read-attribute(name=active-sessions)
{
"outcome" => "success",
"result" => 0
}
The nice thing about WildFly's management DMR interface is that the domain mode tree
is identical to the standalone tree with the exception that on the host controller, you
simply prepend the host/server identifiers to the DMR path. So, for a standalone WildFly,
a DMR path for the active-sessions metric for my test-simple application would be:
/deployment=test-simple-1.0.0.war/subsystem=undertow/:read-attribute(name=active-sessions)
If I want to ask the host controller to give me the exact same metric from that server, I
simply prepend the host/server name in the DMR path:
/host=master/server=server-one/deployment=test-simple-1.0.0.war/subsystem=undertow/:read-attribute(name=active-sessions)
The agent has knowledge of this pattern; knowing this pattern we can cleverly configure
the metadata so we can share types across domain and standalone for inventory discovery.
The problem: WildFly does not expose its JMX MBeans in an equally clever way. I do not see
MBeans that provide metrics for the slave servers.
For example, in JMX, I see WildFly host controller has this MBean:
jboss.as:host=master,server=server-one
Looks very analogous to the DMR resource named /host=master/server=server-one resource ...
right??
Well, there are no deployments associated with this MBean. You would think (following the
DMR pattern) that there would be an MBean named:
jboss.as:host=master,server=server-one,deployment=test-simple-1.0.0.war
But there is not. Nor is there:
jboss.as:host=master,server=server-one,deployment=test-simple-1.0.0.war,subsystem=undertow
which is where you would expect that server's web subsystem metrics to be (if it were
to follow the same DMR pattern). But, again, this doesn't exist.
I can't find JMX MBeans for anything related to the individual slave servers (not just
deployments).
In short, I do not believe the Prometheus JMX Exporter can be used to collect metrics for
all slave servers in a domain if that JMX Exporter is simply installed on a host
controller. This is because the host controller's JMX MBeans do not expose metric data
for individual slave servers.
We would have to have our agent in each slave server (which are just standalone servers -
so the agent would be as if it is monitoring a standalone server). We could have an agent
in the host controller, too, but it would only be responsible for monitoring/managing the
host controller itself.
[this message was sent on Tuesday, November 21, 2017 at 10:11 PM EST]