logging in java agents is not trivial in WildFly / EAP
by Mark Ambrose
>>> ... this means problems still need to be overcome when
>>> putting java agents in host controllers for EAP 7.0 domain mode.
>>> </tl;dr>
Is a solution to these problems for when putting java agents in host
controllers for EAP 7.0 domain mode known?
...
[Host Controller] 16:16:33,937 INFO [org.jboss.as.host.controller]
(Controller Boot Thread) WFLYHC0023: Starting server TestServer1a
16:16:33,968 INFO [org.jboss.as.process.Server:TestServer1a.status]
(ProcessController-threads - 3) WFLYPC0018: Starting process
'Server:TestServer1a'
[Server:TestServer1a] Could not load Logmanager
"org.jboss.logmanager.LogManager"
[Server:TestServer1a] java.lang.ClassNotFoundException:
org.jboss.logmanager.LogManager
[Server:TestServer1a] at
java.net.URLClassLoader.findClass(URLClassLoader.java:382)
...
Thanks,
Mark Ambrose
1 hour, 2 minutes
need a briefing on what the hawkular agent is now doing with respect to inventory
by John Mazzitelli
Joel,
Now that the new inventory-into-metrics is in master and released, I need to know what you did :-D I suspect others will want to know what you did too.
Is it possible for you to write something up or have a 15-minute Blue Jeans session to discuss how inventory is stored in H-Metrics?
I am going to need to know this because I have to implement it in GoLang for HOSA, unless you want to do it :)
--John Mazz
1 day, 6 hours
BTM agent relationship with Hawkular agent
by Gary Brown
Hi
On the team call just now, Heiko raised a question about whether the BTM agent could work with the Hawkular agent. So thought I would start this discussion thread to see what the potential options are.
I currently see two issues:
1) The BTM agent must be configured on the jvm command line as a "-javaagent" to install ByteMan for instrumentation purposes. This is instantiated before the JBoss modules (and therefore subsystems etc) are initialised.
2) The hawkular agent won't necessarily be installed in all monitored servers, and instead remotely monitor some. The BTM agent would need to be installed in all servers where business transactions are executing.
One type of integration that may be possible is in terms of delivering the captured business transaction information to the backend? i.e. the BTM agent locally reports it to the Hawkular agent as a relay?
Thoughts?
Regards
Gary
1 day, 9 hours
logging in java agents is not trivial in WildFly / EAP
by John Mazzitelli
<tl;dr>
No matter if a java agent uses JBoss Logging or simply relies on the JRE's own Java logging, it still won't work in WildFly/EAP unless you set additional things to environment variables in standalone.conf. So no matter what logging is used, this means problems still need to be overcome when putting java agents in host controllers for EAP 7.0 domain mode.
</tl;dr>
I am looking at how Java Agents should log messages when attached to WildFly or EAP. I want to see if there is a way to implement a Java Agent and NOT have to set any special JAVA_OPTS values to pass to the VM to get it to work.
Turns out, it is not a trivial issue.
I wrote a silly little test agent [1] just to see what would happen if it simply logs a message using *java* logging and spins a thread logging a message every second again using *java* logging. So, no extra third party logging libraries, just using java.util.logging classes that are in the JRE already.
Adding just -javaagent to JAVA_OPTS (in standalone.conf, add something like "-javaagent:/where/it/is/ja.jar=foo=bar") is no good, the server bombs with this error, presumably because my agent already started logging using java logger:
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: WFLYLOG0078: The logging subsystem requires the log manager to be org.jboss.logmanager.LogManager. The subsystem has not be initialized and cannot be used. To use JBoss Log Manager you must add the system property "java.util.logging.manager" and set it to "org.jboss.logmanager.LogManager"
Well, catch-22, because if I set java.util.logging.manager=org.jboss.logmanager.LogManager in JAVA_OPTS, this happens when the test agent tries to log:
Could not load Logmanager "org.jboss.logmanager.LogManager"
java.lang.ClassNotFoundException: org.jboss.logmanager.LogManager
which then falls back to java logging, which then causes the server to bomb again because it wants JBoss Logging.
If I set this in standalone.conf:
JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman,org.jboss.logmanager"
the java agent still gets this error:
Could not load Logmanager "org.jboss.logmanager.LogManager"
java.lang.ClassNotFoundException: org.jboss.logmanager.LogManager
which is then followed by jboss modules error which causes the server to again to fail to start:
WARNING: Failed to load the specified log manager class org.jboss.logmanager.LogManager
Exception in thread "main" java.lang.NoClassDefFoundError: org/jboss/logmanager/Level
...
at org.jboss.modules.Module.run(Module.java:320)
The ONLY way to get the server to start properly with this java agent installed (EVEN WITH the agent using ONLY java logging), you have to do all of the above plus add this to JAVA_OPTS:
-Xbootclasspath/p:$JBOSS_HOME/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.0.4.Final.jar
So, in short, you need to add "-Djava.util.logging.manager=org.jboss.logmanager.LogManager" to JAVA_OPTS, you need to add ",org.jboss.logmanager" to JBOSS_MODULES_SYSTEM_PKGS, and you need to add that -Xbootclasspath to JAVA_OPTS just to get a javaagent to work even if the javaagent simply uses nothing more than standard java logging.
---
[1] $ cat JavaAgent.java
import java.util.logging.*;
public class JavaAgent {
private static final Logger log = Logger.getLogger(JavaAgent.class.getName());
public static void premain(String args) {
log.severe("log in premain: " + args);
new Thread(new Runnable() {
public void run() {
while (true) {
try {
Thread.sleep(1000);
log.severe("log in premain loop: " + args);
} catch (Exception e) {
return;
}
}
}
}).start();
}
}
To build it:
$ echo "Premain-Class: JavaAgent" > manifest.mf
$ javac JavaAgent.java
$ jar cvfm ja.jar manifest.mf JavaAgent*
1 day, 12 hours
playing with HOSA outside OS
by John Mazzitelli
I had a couple peeps ask me if they can run HOSA without needing to run it inside an OpenShift cluster (presumably to collect metrics from Prometheus and Jolokia-JMX endpoints that are also running outside of OpenShift). The answer is "yes" and if you are interested, here is a quick how-to.
First get a config.yaml used to configure HOSA (that's the wget command below - it just grabs the example config from github) and then run "docker run" to launch HOSA:
$ wget -O /tmp/config.yaml https://raw.githubusercontent.com/hawkular/hawkular-openshift-agent/maste...
$ docker run --net=host -v /tmp/config.yaml:/config.yaml hawkular/hawkular-openshift-agent --config=/config.yaml
This assumes you have Hawkular-Metrics server (or a full Hawkular-Services server) running on 127.0.0.1 listening to port 8080. If not, just edit config.yaml to point to your server. You can edit that config.yaml however you want.
By default, HOSA itself is a Prometheus endpoint and will collect its own metrics and store them (see config.yaml for its endpoints definitions). So by running HOSA you will automatically start getting "prometheus" data stored into your H-Metrics. You can add more endpoint definitions to the config to tell HOSA to collect from your own Prometheus and Jolokia-JMX endpoints.
You don't have to use docker - if you build the go executable locally (git clone the HOSA repo and "make build") you can run the executable directly. But it's easier to just docker run - no need to git clone, no need to install Go, no need to build anything.
2 weeks
Error connecting to Cassandra - com.datastax.driver.core.exceptions.InvalidQueryException
by Dhillon, Manjot
Hello, I am trying to start hawkular services, version hawkular-1.0.0.Alpha13-SNAPSHOT.
As per the documentation, I am using Cassandra version 3.0.12. On starting the hawkular service, I see the following error -
33m01:58:08,209 WARN [org.hawkular.alerts.engine.impl.CassDefinitionsServiceImpl] (Thread-239 (ActiveMQ-client-global-threads-15127043)) Could not connect to Cassandra cluster - assuming is not up yet. Cause: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: apsrd6612.uhc.com/10.204.85.28:9042 (com.datastax.driver.core.exceptions.InvalidQueryException: unconfigured table schema_keyspaces))
The cassandra driver included with the hawkular service - cassandra-driver-core-2.2.0-rc3.jar, which may be causing the issue.
Thanks,
Manjot
This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the intended
recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.
3 weeks, 6 days
major change in the way we will have to monitor domain mode
by John Mazzitelli
I'm looking at how domain mode monitoring is going to work now that we are moving toward the Prometheus / jmx-exporter way of doing things.
Turns out, due to the way WildFly domain mode exposes things via JMX (rather, I should say, *doesn't* expose things) compared to DMR, I think we are going to have to require the hawkular agent to be deployed to all slave servers, in addition to the master host controller.
In domain mode, you can access all the child slave servers (the individual servers in all server groups) via the host controller's DMR tree - so we only needed one agent on the host controller to monitor the domain. For example, to see a metric for a deployment on "server-one" that is managed by host controller "master", I can simply ask the host controller for it:
/host=master/server=server-one/deployment=test-simple-1.0.0.war/subsystem=undertow/:read-attribute(name=active-sessions)
{
"outcome" => "success",
"result" => 0
}
The nice thing about WildFly's management DMR interface is that the domain mode tree is identical to the standalone tree with the exception that on the host controller, you simply prepend the host/server identifiers to the DMR path. So, for a standalone WildFly, a DMR path for the active-sessions metric for my test-simple application would be:
/deployment=test-simple-1.0.0.war/subsystem=undertow/:read-attribute(name=active-sessions)
If I want to ask the host controller to give me the exact same metric from that server, I simply prepend the host/server name in the DMR path:
/host=master/server=server-one/deployment=test-simple-1.0.0.war/subsystem=undertow/:read-attribute(name=active-sessions)
The agent has knowledge of this pattern; knowing this pattern we can cleverly configure the metadata so we can share types across domain and standalone for inventory discovery.
The problem: WildFly does not expose its JMX MBeans in an equally clever way. I do not see MBeans that provide metrics for the slave servers.
For example, in JMX, I see WildFly host controller has this MBean:
jboss.as:host=master,server=server-one
Looks very analogous to the DMR resource named /host=master/server=server-one resource ... right??
Well, there are no deployments associated with this MBean. You would think (following the DMR pattern) that there would be an MBean named:
jboss.as:host=master,server=server-one,deployment=test-simple-1.0.0.war
But there is not. Nor is there:
jboss.as:host=master,server=server-one,deployment=test-simple-1.0.0.war,subsystem=undertow
which is where you would expect that server's web subsystem metrics to be (if it were to follow the same DMR pattern). But, again, this doesn't exist.
I can't find JMX MBeans for anything related to the individual slave servers (not just deployments).
In short, I do not believe the Prometheus JMX Exporter can be used to collect metrics for all slave servers in a domain if that JMX Exporter is simply installed on a host controller. This is because the host controller's JMX MBeans do not expose metric data for individual slave servers.
We would have to have our agent in each slave server (which are just standalone servers - so the agent would be as if it is monitoring a standalone server). We could have an agent in the host controller, too, but it would only be responsible for monitoring/managing the host controller itself.
[this message was sent on Tuesday, November 21, 2017 at 10:11 PM EST]
2 months, 1 week
