BTM agent relationship with Hawkular agent
by Gary Brown
Hi
On the team call just now, Heiko raised a question about whether the BTM agent could work with the Hawkular agent. So thought I would start this discussion thread to see what the potential options are.
I currently see two issues:
1) The BTM agent must be configured on the jvm command line as a "-javaagent" to install ByteMan for instrumentation purposes. This is instantiated before the JBoss modules (and therefore subsystems etc) are initialised.
2) The hawkular agent won't necessarily be installed in all monitored servers, and instead remotely monitor some. The BTM agent would need to be installed in all servers where business transactions are executing.
One type of integration that may be possible is in terms of delivering the captured business transaction information to the backend? i.e. the BTM agent locally reports it to the Hawkular agent as a relay?
Thoughts?
Regards
Gary
2 months
some new agent features
by John Mazzitelli
Just wanted to blast out a message about some new wildfly agent features.
1) Agent now supports secure communications between itself and any of its managed WildFly servers. I documented how to enable security in the management interface of WildFly here:
https://github.com/hawkular/hawkular.github.io/blob/pages/src/main/jbake/...
2) Agent now supports monitoring JMX resources via Jolokia interface.
The agent now can collect JMX metrics and availability data and can inventory JMX resources in the same manner as it can with DMR resources using the Jolokia interface (Jolokia is a very nice REST interface to JMX). Just as we have <metric-dmr>, <avail-dmr>, <resource-type-dmr>, <remote-dmr>, and related DMR settings in the agent subsystem, we now have <metric-jmx>, <avail-jmx>, <resource-type-jmx>, <remote-jmx> settings as well. So the agent can be given metadata that tells it what to do. As with remote WF servers, the agent also supports secure comm to the remote jolokia JMX server.
There are a few things currently on the TODO for the agent:
1) utilize the new inventory "name" attribute - they now have "name" as a first-class setting - I need to see what the agent has to do to use it
2) utilize the inventory's "operation" metadata - I believe operations are now first class concepts in inventory - I need to figure out how to populate it
3) provide a mechanism for the agent to detect runtime changes to inventory. Right now, after the agent does its initial inventory scan, that's it. The inventory is fixed, all resource configuration is fixed, and all we do is collect metrics. The caveat is that if someone deploys a new app or adds a new JDBC driver, we force a new scan. We need a way to detect new resources in inventory (and old resources removed from inventory) and we need a way to detect changes to resource config. This should work for both DMR and JMX inventory items.
4) Need a way to consume wildfly notifications that other subsystems emit and figure out how to send that to a hawkualr events subsystem.
5) Peter is implementing/has implemented the JDBC driver CRUD and datasource CRUD, gonna see how easy it is to implement reload.
I'm sure I'm forgetting something, but .. that's what I remember.
2 months, 3 weeks
Corrections and suggestions on Explorer for android client
by Anuj Garg
Hello everyone,
A tree structure of the explorer is approx ready for the android client and
I was curious if i am wrong somewhere.
First of all, All feeds are collected by call to
/inventory/feeds
feed call for resources
/inventory/feeds/{feed}/resources
call for rec resources and metrics
/inventory/feeds/{feed}/resources/{resid}/recursiveChildren
/inventory/{feed}/resources/{resid}/metrics
Am I missing something?
Thanks
Anuj Garg
3 months
some new hawkular openshift agent stuff
by John Mazzitelli
FYI: some new things went into HOSA (that's the Hawkular OpenShift Agent for the uninitiated).
1. The agent now emits its own metrics and can monitor itself. Right now it just emits some basic "go" metrics like memory usage, CPU usage, etc along with one agent-specific one - a counter that counts the number of data points it has collected in its lifetime. We'll add more metrics as we figure out the things people want to see, but we have the infrastructure in place now.
2. The agent is deployed as a daemonset. This means as new nodes are brought online, an agent will go along with it (or so I'm told :)
3. The agent has changed the way it discovers what to monitor - it no longer looks at annotations on pods to determine where the configmaps are for those pods. Instead, it looks up volume declarations to see if there is an agent configmap defined. This was done to be ready for the future when new security constraints will be introduced in OpenShift which would have broken our annotation approach. This approach using volumes should not hit that issue.
NOTE: If you are building the latest agent from master, we added some dependencies so you have to update your dependencies via Glide by using the "make update-deps" target prior to building from source.
3 months, 1 week
Inventory API question
by Austin Kuo
I was creating 2 different resource types with the same http client at the
same time.
But one succeed, the other failed with the response 400 and body:
{
"errorMsg" : "The transaction has already been closed"
}
Is it not allowed to do so?
Austin.
4 months
need a briefing on what the hawkular agent is now doing with respect to inventory
by John Mazzitelli
Joel,
Now that the new inventory-into-metrics is in master and released, I need to know what you did :-D I suspect others will want to know what you did too.
Is it possible for you to write something up or have a 15-minute Blue Jeans session to discuss how inventory is stored in H-Metrics?
I am going to need to know this because I have to implement it in GoLang for HOSA, unless you want to do it :)
--John Mazz
4 months, 1 week
playing with HOSA outside OS
by John Mazzitelli
I had a couple peeps ask me if they can run HOSA without needing to run it inside an OpenShift cluster (presumably to collect metrics from Prometheus and Jolokia-JMX endpoints that are also running outside of OpenShift). The answer is "yes" and if you are interested, here is a quick how-to.
First get a config.yaml used to configure HOSA (that's the wget command below - it just grabs the example config from github) and then run "docker run" to launch HOSA:
$ wget -O /tmp/config.yaml https://raw.githubusercontent.com/hawkular/hawkular-openshift-agent/maste...
$ docker run --net=host -v /tmp/config.yaml:/config.yaml hawkular/hawkular-openshift-agent --config=/config.yaml
This assumes you have Hawkular-Metrics server (or a full Hawkular-Services server) running on 127.0.0.1 listening to port 8080. If not, just edit config.yaml to point to your server. You can edit that config.yaml however you want.
By default, HOSA itself is a Prometheus endpoint and will collect its own metrics and store them (see config.yaml for its endpoints definitions). So by running HOSA you will automatically start getting "prometheus" data stored into your H-Metrics. You can add more endpoint definitions to the config to tell HOSA to collect from your own Prometheus and Jolokia-JMX endpoints.
You don't have to use docker - if you build the go executable locally (git clone the HOSA repo and "make build") you can run the executable directly. But it's easier to just docker run - no need to git clone, no need to install Go, no need to build anything.
4 months, 2 weeks
