[Hawkular-dev] major change in the way we will have to monitor domain mode

[Matthew Wringe]

Do we have a Jira where this work is being tracked?

>From the discussions we had I believe we were going to keep what we have
been doing with inventory (which uses DMR), but now we will require that
all the servers will require the agent to be run so that we can expose
metrics for each server (which may also be something we would want to do
anyways to prevent everything going to the host controller).

I think the plan for now is to investigate if we need to make changes to
miq to handle how things are being done now, or if we can handle it more at
the inventory level.

On Tue, Nov 21, 2017 at 10:11 PM, John Mazzitelli <mazz at redhat.com> wrote:

> I'm looking at how domain mode monitoring is going to work now that we are
> moving toward the Prometheus / jmx-exporter way of doing things.
>
> Turns out, due to the way WildFly domain mode exposes things via JMX
> (rather, I should say, *doesn't* expose things) compared to DMR, I think we
> are going to have to require the hawkular agent to be deployed to all slave
> servers, in addition to the master host controller.
>
> In domain mode, you can access all the child slave servers (the individual
> servers in all server groups) via the host controller's DMR tree - so we
> only needed one agent on the host controller to monitor the domain. For
> example, to see a metric for a deployment on "server-one" that is managed
> by host controller "master", I can simply ask the host controller for it:
>
> /host=master/server=server-one/deployment=test-simple-1.
> 0.0.war/subsystem=undertow/:read-attribute(name=active-sessions)
> {
>     "outcome" => "success",
>     "result" => 0
> }
>
> The nice thing about WildFly's management DMR interface is that the domain
> mode tree is identical to the standalone tree with the exception that on
> the host controller, you simply prepend the host/server identifiers to the
> DMR path. So, for a standalone WildFly, a DMR path for the active-sessions
> metric for my test-simple application would be:
>
> /deployment=test-simple-1.0.0.war/subsystem=undertow/:read-
> attribute(name=active-sessions)
>
> If I want to ask the host controller to give me the exact same metric from
> that server, I simply prepend the host/server name in the DMR path:
>
> /host=master/server=server-one/deployment=test-simple-1.
> 0.0.war/subsystem=undertow/:read-attribute(name=active-sessions)
>
> The agent has knowledge of this pattern; knowing this pattern we can
> cleverly configure the metadata so we can share types across domain and
> standalone for inventory discovery.
>
> The problem: WildFly does not expose its JMX MBeans in an equally clever
> way. I do not see MBeans that provide metrics for the slave servers.
>
> For example, in JMX, I see WildFly host controller has this MBean:
>
> jboss.as:host=master,server=server-one
>
> Looks very analogous to the DMR resource named
> /host=master/server=server-one resource ... right??
>
> Well, there are no deployments associated with this MBean. You would think
> (following the DMR pattern) that there would be an MBean named:
>
> jboss.as:host=master,server=server-one,deployment=test-simple-1.0.0.war
>
> But there is not. Nor is there:
>
> jboss.as:host=master,server=server-one,deployment=test-
> simple-1.0.0.war,subsystem=undertow
>
> which is where you would expect that server's web subsystem metrics to be
> (if it were to follow the same DMR pattern). But, again, this doesn't exist.
>
> I can't find JMX MBeans for anything related to the individual slave
> servers (not just deployments).
>
> In short, I do not believe the Prometheus JMX Exporter can be used to
> collect metrics for all slave servers in a domain if that JMX Exporter is
> simply installed on a host controller. This is because the host
> controller's JMX MBeans do not expose metric data for individual slave
> servers.
>
> We would have to have our agent in each slave server (which are just
> standalone servers - so the agent would be as if it is monitoring a
> standalone server). We could have an agent in the host controller, too, but
> it would only be responsible for monitoring/managing the host controller
> itself.
>
> [this message was sent on Tuesday, November 21, 2017 at 10:11 PM EST]
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20171123/f0978063/attachment-0001.html