[Hawkular-dev] logging in java agents is not trivial in WildFly / EAP

[John Mazzitelli]

<tl;dr>
No matter if a java agent uses JBoss Logging or simply relies on the JRE's own Java logging, it still won't work in WildFly/EAP unless you set additional things to environment variables in standalone.conf. So no matter what logging is used, this means problems still need to be overcome when putting java agents in host controllers for EAP 7.0 domain mode.
</tl;dr>

I am looking at how Java Agents should log messages when attached to WildFly or EAP. I want to see if there is a way to implement a Java Agent and NOT have to set any special JAVA_OPTS values to pass to the VM to get it to work.

Turns out, it is not a trivial issue.

I wrote a silly little test agent [1] just to see what would happen if it simply logs a message using *java* logging and spins a thread logging a message every second again using *java* logging. So, no extra third party logging libraries, just using java.util.logging classes that are in the JRE already.

Adding just -javaagent to JAVA_OPTS (in standalone.conf, add something like "-javaagent:/where/it/is/ja.jar=foo=bar") is no good, the server bombs with this error, presumably because my agent already started logging using java logger:

   Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: WFLYLOG0078: The logging subsystem requires the log manager to be org.jboss.logmanager.LogManager. The subsystem has not be initialized and cannot be used. To use JBoss Log Manager you must add the system property "java.util.logging.manager" and set it to "org.jboss.logmanager.LogManager"

Well, catch-22, because if I set java.util.logging.manager=org.jboss.logmanager.LogManager in JAVA_OPTS, this happens when the test agent tries to log:

   Could not load Logmanager "org.jboss.logmanager.LogManager"
   java.lang.ClassNotFoundException: org.jboss.logmanager.LogManager

which then falls back to java logging, which then causes the server to bomb again because it wants JBoss Logging.

If I set this in standalone.conf:

   JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman,org.jboss.logmanager"

the java agent still gets this error:

   Could not load Logmanager "org.jboss.logmanager.LogManager"
   java.lang.ClassNotFoundException: org.jboss.logmanager.LogManager

which is then followed by jboss modules error which causes the server to again to fail to start:

   WARNING: Failed to load the specified log manager class org.jboss.logmanager.LogManager
   Exception in thread "main" java.lang.NoClassDefFoundError: org/jboss/logmanager/Level
      ...
	at org.jboss.modules.Module.run(Module.java:320)

The ONLY way to get the server to start properly with this java agent installed (EVEN WITH the agent using ONLY java logging), you have to do all of the above plus add this to JAVA_OPTS:

-Xbootclasspath/p:$JBOSS_HOME/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.0.4.Final.jar 

So, in short, you need to add "-Djava.util.logging.manager=org.jboss.logmanager.LogManager" to JAVA_OPTS, you need to add ",org.jboss.logmanager" to JBOSS_MODULES_SYSTEM_PKGS, and you need to add that -Xbootclasspath to JAVA_OPTS just to get a javaagent to work even if the javaagent simply uses nothing more than standard java logging.

---

[1] $ cat JavaAgent.java
import java.util.logging.*;
public class JavaAgent {
    private static final Logger log = Logger.getLogger(JavaAgent.class.getName());
    public static void premain(String args) {
        log.severe("log in premain: " + args);
        new Thread(new Runnable() {
            public void run() {
                while (true) {
                    try {
                        Thread.sleep(1000);
                        log.severe("log in premain loop: " + args);
                    } catch (Exception e) {
                        return;
                    }
                }
            }
        }).start();
    }
}

To build it:

$ echo "Premain-Class: JavaAgent" > manifest.mf
$ javac JavaAgent.java
$ jar cvfm ja.jar manifest.mf JavaAgent*