[Hawkular-dev] agent running as a VM javaagent - non-EAP based solution

[John Mazzitelli]

Right now, the Hawkular WildFly Agent runs as a WildFly subsystem extension - which means it must run in EAP.

To avoid this requirement, I refactored the agent code [1] so it can be run using a standard VM javaagent (using the VM argument -javaagent).

This means the agent need not run as a subsystem extension, in fact, it can run in any VM (non-EAP/WildFly based). Just pass in the -javaagent command line argument to your VM and you got an agent. You have a Karaf container in a JVM exposing metrics via JMX? You can use this. You got a vert.x server in a JVM exposing metrics via JMX? You can use this.

This new agent's config file is a single YAML file that mimics virtually the same settings as you see today in the agent's <subsystem> XML in standalone.xml.

Like the EAP-based agent, this new agent can still:

* collect DMR metrics and inventory from any EAP 6.4, EAP 7.x, or WildFly 10+ servers - local or remote.
* collect JMX metrics and inventory from any remote Jolokia endpoint or local JMX MBeanServer
* receive websocket messages from a Hawkular Server (if running in full Hawkular mode)
* run in metrics-only mode where it only stores data to Hawkular-Metrics and does not connect via websocket to the server nor stores inventory to H-Inventory
* Talk to the Hawkular Server and remote DMR/JMX endpoints via SSL/https using defined security-realms/keystores

Unlike the EAP-based agent, this new agent:

* does NOT have a CLI (jboss-cli.sh is the EAP-based agent's CLI)
* does NOT have a built-in configuration persistent mechanicm (EAP-based agents can persist their config changes to standalone.xml when those changes are made via the WildFly management interface - because the javaagent is completely independent, it has nothing like this for its YAML config).
* does NOT integrate with the lifecycle of any EAP/WildFly running in the same VM (in other words, if you ask to "reload" the EAP server, the javaagent knows nothing about that and won't also itself reload. Remember, this javaagent is now completely independent of any co-located EAP/WildFly server).

In addition, if your JVM is installed in OpenShift, you can get the nice option of putting the javaagent's YAML config in a config map and mount it so your javaagent can access it. This means you can edit your javaagent YAML directly in the OpenShift UI :)

I still need to see how well this supports the disabling/enabling of metrics on the fly from the websocket commands (e.g. I will need to add the ability to change the YAML config to persist the settings, that is not done yet - the EAP-based agent got this support for free - its config changes go into standalone.xml).

Other than persisting changes to the YAML file, I believe this javaagent will work essentially the same as the current agent (indeed, the core agent engine is identical - its the same code). Just have to make sure the websocket commands can work considering there is a YAML config backing the settings and not standalone.xml.

I need people's feedback on this. Thoughts? If you want to try it, you can build it from my PR right now using the README [2] and the instructions [3].

--John Mazz

[1] https://github.com/hawkular/hawkular-agent/pull/302

[2] https://github.com/jmazzitelli/hawkular-agent/blob/refactor-core/hawkular-javaagent/README.adoc

[3] Instructions to build using the PR and test it by collecting EAP metrics:

1) Build it:

   $ git clone git at github.com:hawkular/hawkular-agent.git
   $ cd hawkular-agent
   $ git checkout -b jmazzitelli-refactor-core master
   $ git pull https://github.com/jmazzitelli/hawkular-agent.git refactor-core
   $ mvn clean install

2) Install it - assuming you have a base EAP 6.4 or 7.x or WildFly 10+ installed at <eap.dir>

2a) Change this line in <eap.dir>/bin/standalone.conf:

   JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman"

to this:

   JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman,org.jboss.logmanager"

2b) Inside that same <eap.dir>/bin/standalone.conf you add this line in the
    appropriate place (you know where, just under where it sets JAVA_OPTS):

   JAVA_OPTS="$JAVA_OPTS -Djava.util.logging.manager=org.jboss.logmanager.LogManager -javaagent:$JBOSS_HOME/bin/hawkular-javaagent-*.jar=config=$JBOSS_HOME/standalone/configuration/real-config*.yaml,delay=10"

2c) Copy the binary/config files you built to that EAP / WildFly install:

   cp hawkular-agent/hawkular-javaagent/target/hawkular-javaagent-*-jar-with-dependencies.jar <eap.dir>/bin

   if EAP 7.x/WildFly 10+:

   cp hawkular-agent/hawkular-javaagent/src/test/resources/real-config.yaml <eap.dir>/standalone/configuration

   if EAP 6.4:

   cp hawkular-agent/hawkular-javaagent/src/test/resources/real-config-eap6.yaml <eap.dir>/standalone/configuration