[wildfly-dev] embedded RHQ agent subsystem into WildFly

[Brian Stansberry]

Thanks, John.

I think we need to thoroughly work through the requirements and all the 
expected behavior, and then start worrying about the implementation issues.

More in-line.

On 2/7/14, 11:04 AM, John Mazzitelli wrote:
> [sending this to the open source project community mailing lists for RHQ and Wildfly]
>
> This is to kick off a discussion regarding embedding a RHQ agent [3] into an WildFly instance (which will also be applicable to downstream projects JON and EAP respectively - but I'll continue to use RHQ and WildFly since the code will almost certainly go in those upstream projects).
>
> The thought is to embed a RHQ Agent in a running WildFly instance, specifically a Host Controller. In the standalone RHQ use-case, a single RHQ Agent process runs on a box and manages all the things on that box. Since there is one Host Controller on a box, it seems to make sense to embed a RHQ Agent there. This is something we can discuss (do we embed in Process Controller instead? The initial thought was no, probably not a good idea).
>

Right. The PC is only meant to control the lifecycle of the other 
processes, with as little complexity as possible. Reliability is 
critical so we don't want complexity.

We separated the PC from the HC to reduce the risk that a problem in the 
complex HC would cause it to fail and leave nothing consuming the 
stdout/err streams of the server processes. We didn't want "agent" 
failures affecting end-user request handling.

The java.lang.ProcessBuilder.Redirect stuff in JDK 7 *may* eliminate the 
need for a separate PC in the future. But if we want this in EAP 6.x we 
can't assume JDK 7.

A general notion way back at the start of AS7 was the PC might also make 
doing patching easier. As we tackle domain-controller-coordinated 
patching this year we'll see if that proves true. I have some doubts now.


> This would allow a person to "flip a switch" to enable RHQ management of their WildFly infrastructure running on that box without having to separately install a RHQ agent on their own.
>
> Now, by following the "Extending WildFly 8" example subsystem [1] I already have an initial implementation of an embedded agent extension subsystem [2] and it is able to run in a standalone WildFly 6 instance. It will be turned off by default, so its footprint is minimal by default (just a few extension classes need to be loaded - like the definitions, the add/remove step handlers, etc). But when you do have it enabled, a fullblown RHQ Agent will be started in the WildFly instance. Today, I have this working - it loads all the necessary plugin jars and native libraries to manage platform services and WildFly resources. The RHQ team might even consider this as its own embedded agent for use within the RHQ Server itself (since a RHQ Server is itself running in a standalone WildFly 6 instance, this thus allows a RHQ Server to manage itself with its own embedded RHQ Agent).
>
> It was my mistaken impression (which Brian mentioned to me recently) that Host Controllers have extensible subsystems like standalone instances do. I looked in domain.xml, saw <extension> and <subsystem> tags and just assumed it worked like standalone, but alas, it does not. So, one question would be - how easy would it be to have extension subsystems installed and run in a Host controller like we do in standalone instances? Otherwise, what has to be done to implement a new subsystem in a Host controller, if it is not done like the "Extending WildFly 8" example.
>

There is presently no extensibility mechanism for the HC process. We are 
going to have to create one for other reasons anyway, so we should just 
assume that the HC will be extensible.

I'm sure it will work similarly to server process extensions, with a 
very very high % of code reuse for stuff that can be both a server 
extension and an HC extension.

> Another question (probably the first question) to answer is - do we even need to embed an agent like this? Or, is simply packaging a RHQ Agent with WildFly Host Controllers all that is needed for a better user-experience than what people have today (which is to separately install RHQ agents after they install WildFly and domain/host controllers?) Can an embedded agent provide better functionality than a separate process/instance of the RHQ agent (even if it is distributed within an WildFly distro?).
>

This is one we should keep in mind throughout our discussions. There's a 
benefit (one less process, etc) but it will become clear there's a 
significant cost too. So the cost/benefit ratio should be rechecked from 
time to time as we go along.

> There are other issues, obviously, to consider:
> * If a Host Controller gets shutdown, that means the RHQ Agent goes down with it (and thus manageability of that box goes away).

Conceptually, the HC shouldn't get shut down. The PC will automatically 
restart it if it shuts down. The PC can be be shut down though. So

1) There's a UX and user education issue. Users certainly may forget 
that stopping all the EAP processes now stops the JON agent as well. 
People who want to shut down all servers will need to learn to do just 
that rather than simply shutting down the PC.

2) Adding more functionality increases the potential for problems that 
would cause the HC to fail. But it doesn't necessarily follow that 
having both in one process will be less stable overall than having two 
processes.

> * Do we allow the RHQ Agent running in a Host Controller to manage EVERYTHING on this box (like a standalone agent would) or do we just want that agent to manage only the WildFly on that box?

Any relevant categories? OS stuff seems different from Apache servers or 
things like that, which seem different from other EAP 6+ installs.

I vote we rule out controlling other EAP 6+ installs that also have a 
JON agent running embedded!

Users may want this to be configurable in some fashion. Some may want 
one agent to rule everything; others may prefer to have EAP processes 
focused on EAP.

> ** If its the latter (manage only WildFly), how do we want to be able to manage everything else on that box (or do we even want to do that?).

Would RHQ want to ship agents separately from WF? With the introduction 
of a separate WildFly Core that we're doing for WF 9 it becomes more 
feasilbe to say no, since RHQ users uninterested in managing WF won't 
have to accept a full EE appserver distribution just to get an agent.

> * What about testing requirements (what kind of tests are required to get an embedded agent into the WildFly buld?)
> * How do we do packaging of an embedded agent?
> * etc. etc. etc. yadda. yadda. yadda.
>
> OK, so that's the kickoff email. I'll let others take it from here and move this conversation in the directions that people want it to go.
>
> --John Mazz
>
> [1] https://docs.jboss.org/author/display/WFLY8/Example+subsystem
> [2] source can be found here: https://git.fedorahosted.org/cgit/rhq/rhq.git/tree/modules/enterprise/server/embeddedagent/src/main/java/org/rhq/embeddedagent/extension
> [3] see http://www.jboss.org/rhq
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>


-- 
Brian Stansberry
Senior Principal Software Engineer
JBoss by Red Hat