some new agent features
by John Mazzitelli
Just wanted to blast out a message about some new wildfly agent features.
1) Agent now supports secure communications between itself and any of its managed WildFly servers. I documented how to enable security in the management interface of WildFly here:
https://github.com/hawkular/hawkular.github.io/blob/pages/src/main/jbake/...
2) Agent now supports monitoring JMX resources via Jolokia interface.
The agent now can collect JMX metrics and availability data and can inventory JMX resources in the same manner as it can with DMR resources using the Jolokia interface (Jolokia is a very nice REST interface to JMX). Just as we have <metric-dmr>, <avail-dmr>, <resource-type-dmr>, <remote-dmr>, and related DMR settings in the agent subsystem, we now have <metric-jmx>, <avail-jmx>, <resource-type-jmx>, <remote-jmx> settings as well. So the agent can be given metadata that tells it what to do. As with remote WF servers, the agent also supports secure comm to the remote jolokia JMX server.
There are a few things currently on the TODO for the agent:
1) utilize the new inventory "name" attribute - they now have "name" as a first-class setting - I need to see what the agent has to do to use it
2) utilize the inventory's "operation" metadata - I believe operations are now first class concepts in inventory - I need to figure out how to populate it
3) provide a mechanism for the agent to detect runtime changes to inventory. Right now, after the agent does its initial inventory scan, that's it. The inventory is fixed, all resource configuration is fixed, and all we do is collect metrics. The caveat is that if someone deploys a new app or adds a new JDBC driver, we force a new scan. We need a way to detect new resources in inventory (and old resources removed from inventory) and we need a way to detect changes to resource config. This should work for both DMR and JMX inventory items.
4) Need a way to consume wildfly notifications that other subsystems emit and figure out how to send that to a hawkualr events subsystem.
5) Peter is implementing/has implemented the JDBC driver CRUD and datasource CRUD, gonna see how easy it is to implement reload.
I'm sure I'm forgetting something, but .. that's what I remember.
4 days, 12 hours
Corrections and suggestions on Explorer for android client
by Anuj Garg
Hello everyone,
A tree structure of the explorer is approx ready for the android client and
I was curious if i am wrong somewhere.
First of all, All feeds are collected by call to
/inventory/feeds
feed call for resources
/inventory/feeds/{feed}/resources
call for rec resources and metrics
/inventory/feeds/{feed}/resources/{resid}/recursiveChildren
/inventory/{feed}/resources/{resid}/metrics
Am I missing something?
Thanks
Anuj Garg
1 week, 3 days
some new hawkular openshift agent stuff
by John Mazzitelli
FYI: some new things went into HOSA (that's the Hawkular OpenShift Agent for the uninitiated).
1. The agent now emits its own metrics and can monitor itself. Right now it just emits some basic "go" metrics like memory usage, CPU usage, etc along with one agent-specific one - a counter that counts the number of data points it has collected in its lifetime. We'll add more metrics as we figure out the things people want to see, but we have the infrastructure in place now.
2. The agent is deployed as a daemonset. This means as new nodes are brought online, an agent will go along with it (or so I'm told :)
3. The agent has changed the way it discovers what to monitor - it no longer looks at annotations on pods to determine where the configmaps are for those pods. Instead, it looks up volume declarations to see if there is an agent configmap defined. This was done to be ready for the future when new security constraints will be introduced in OpenShift which would have broken our annotation approach. This approach using volumes should not hit that issue.
NOTE: If you are building the latest agent from master, we added some dependencies so you have to update your dependencies via Glide by using the "make update-deps" target prior to building from source.
2 weeks, 5 days
logging in java agents is not trivial in WildFly / EAP
by John Mazzitelli
<tl;dr>
No matter if a java agent uses JBoss Logging or simply relies on the JRE's own Java logging, it still won't work in WildFly/EAP unless you set additional things to environment variables in standalone.conf. So no matter what logging is used, this means problems still need to be overcome when putting java agents in host controllers for EAP 7.0 domain mode.
</tl;dr>
I am looking at how Java Agents should log messages when attached to WildFly or EAP. I want to see if there is a way to implement a Java Agent and NOT have to set any special JAVA_OPTS values to pass to the VM to get it to work.
Turns out, it is not a trivial issue.
I wrote a silly little test agent [1] just to see what would happen if it simply logs a message using *java* logging and spins a thread logging a message every second again using *java* logging. So, no extra third party logging libraries, just using java.util.logging classes that are in the JRE already.
Adding just -javaagent to JAVA_OPTS (in standalone.conf, add something like "-javaagent:/where/it/is/ja.jar=foo=bar") is no good, the server bombs with this error, presumably because my agent already started logging using java logger:
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: WFLYLOG0078: The logging subsystem requires the log manager to be org.jboss.logmanager.LogManager. The subsystem has not be initialized and cannot be used. To use JBoss Log Manager you must add the system property "java.util.logging.manager" and set it to "org.jboss.logmanager.LogManager"
Well, catch-22, because if I set java.util.logging.manager=org.jboss.logmanager.LogManager in JAVA_OPTS, this happens when the test agent tries to log:
Could not load Logmanager "org.jboss.logmanager.LogManager"
java.lang.ClassNotFoundException: org.jboss.logmanager.LogManager
which then falls back to java logging, which then causes the server to bomb again because it wants JBoss Logging.
If I set this in standalone.conf:
JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman,org.jboss.logmanager"
the java agent still gets this error:
Could not load Logmanager "org.jboss.logmanager.LogManager"
java.lang.ClassNotFoundException: org.jboss.logmanager.LogManager
which is then followed by jboss modules error which causes the server to again to fail to start:
WARNING: Failed to load the specified log manager class org.jboss.logmanager.LogManager
Exception in thread "main" java.lang.NoClassDefFoundError: org/jboss/logmanager/Level
...
at org.jboss.modules.Module.run(Module.java:320)
The ONLY way to get the server to start properly with this java agent installed (EVEN WITH the agent using ONLY java logging), you have to do all of the above plus add this to JAVA_OPTS:
-Xbootclasspath/p:$JBOSS_HOME/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.0.4.Final.jar
So, in short, you need to add "-Djava.util.logging.manager=org.jboss.logmanager.LogManager" to JAVA_OPTS, you need to add ",org.jboss.logmanager" to JBOSS_MODULES_SYSTEM_PKGS, and you need to add that -Xbootclasspath to JAVA_OPTS just to get a javaagent to work even if the javaagent simply uses nothing more than standard java logging.
---
[1] $ cat JavaAgent.java
import java.util.logging.*;
public class JavaAgent {
private static final Logger log = Logger.getLogger(JavaAgent.class.getName());
public static void premain(String args) {
log.severe("log in premain: " + args);
new Thread(new Runnable() {
public void run() {
while (true) {
try {
Thread.sleep(1000);
log.severe("log in premain loop: " + args);
} catch (Exception e) {
return;
}
}
}
}).start();
}
}
To build it:
$ echo "Premain-Class: JavaAgent" > manifest.mf
$ javac JavaAgent.java
$ jar cvfm ja.jar manifest.mf JavaAgent*
4 weeks, 1 day
Inventory API question
by Austin Kuo
I was creating 2 different resource types with the same http client at the
same time.
But one succeed, the other failed with the response 400 and body:
{
"errorMsg" : "The transaction has already been closed"
}
Is it not allowed to do so?
Austin.
1 month, 1 week
need a briefing on what the hawkular agent is now doing with respect to inventory
by John Mazzitelli
Joel,
Now that the new inventory-into-metrics is in master and released, I need to know what you did :-D I suspect others will want to know what you did too.
Is it possible for you to write something up or have a 15-minute Blue Jeans session to discuss how inventory is stored in H-Metrics?
I am going to need to know this because I have to implement it in GoLang for HOSA, unless you want to do it :)
--John Mazz
1 month, 2 weeks
BTM agent relationship with Hawkular agent
by Gary Brown
Hi
On the team call just now, Heiko raised a question about whether the BTM agent could work with the Hawkular agent. So thought I would start this discussion thread to see what the potential options are.
I currently see two issues:
1) The BTM agent must be configured on the jvm command line as a "-javaagent" to install ByteMan for instrumentation purposes. This is instantiated before the JBoss modules (and therefore subsystems etc) are initialised.
2) The hawkular agent won't necessarily be installed in all monitored servers, and instead remotely monitor some. The BTM agent would need to be installed in all servers where business transactions are executing.
One type of integration that may be possible is in terms of delivering the captured business transaction information to the backend? i.e. the BTM agent locally reports it to the Hawkular agent as a relay?
Thoughts?
Regards
Gary
1 month, 2 weeks
playing with HOSA outside OS
by John Mazzitelli
I had a couple peeps ask me if they can run HOSA without needing to run it inside an OpenShift cluster (presumably to collect metrics from Prometheus and Jolokia-JMX endpoints that are also running outside of OpenShift). The answer is "yes" and if you are interested, here is a quick how-to.
First get a config.yaml used to configure HOSA (that's the wget command below - it just grabs the example config from github) and then run "docker run" to launch HOSA:
$ wget -O /tmp/config.yaml https://raw.githubusercontent.com/hawkular/hawkular-openshift-agent/maste...
$ docker run --net=host -v /tmp/config.yaml:/config.yaml hawkular/hawkular-openshift-agent --config=/config.yaml
This assumes you have Hawkular-Metrics server (or a full Hawkular-Services server) running on 127.0.0.1 listening to port 8080. If not, just edit config.yaml to point to your server. You can edit that config.yaml however you want.
By default, HOSA itself is a Prometheus endpoint and will collect its own metrics and store them (see config.yaml for its endpoints definitions). So by running HOSA you will automatically start getting "prometheus" data stored into your H-Metrics. You can add more endpoint definitions to the config to tell HOSA to collect from your own Prometheus and Jolokia-JMX endpoints.
You don't have to use docker - if you build the go executable locally (git clone the HOSA repo and "make build") you can run the executable directly. But it's easier to just docker run - no need to git clone, no need to install Go, no need to build anything.
1 month, 4 weeks
