hawkular java agent now supports ${x} in config
by John Mazzitelli
Just released Hawkular Java Agent 0.29.1.Final that contains a new feature.
To support deploying Java Agent within containers (though this feature isn't restricted to that use-case), you can configure many settings in the Java Agent YAML config with ${x} expressions (similar to how HWFA can be configured in standalone.xml).
It supports getting values from system properties, environment variables, with default values optionally defined. The following expressions are supported - hopefully this is self-explanatory:
${some.system.property}
${some.system.property:a-default-value}
${ENV~SOME_ENV_VAR}
${ENV~SOME_ENV_VAR:a-default-value}
Not all properties support expressions (most don't need them) but properties that define things like hosts, ports, urls, passwords, filepaths, do support expressions.
If I missed any, or if you'd like a property to support expressions that does not currently, write a HWKAGENT JIRA: https://issues.jboss.org/projects/HWKAGENT
9 years
hawkular java agent to be merged / released
by John Mazzitelli
FYI: I am going to merge the PR that introduces the javaagent and then I will release a new agent from that new master branch.
Just to be clear, here is what will then be supported:
1) The Hawkular WildFly Agent (HWFA) remains as-is. Nothing changes. The agent you've come to love and enjoy doesn't change.
2) There is a new Java Agent (HJA) that you can run in any JVM (including but not limited to EAP-based projects). You do so by passing in a "-javaagent" command line option to your JVM (e.g. java -javaagent=hawkular-javaagent.jar=config=config.yaml -jar my-app.jar ...yadda...). There are two additional properties you must set in standalone.conf if you want to run it inside an EAP JVM. The README will have the details.
This new HJA is configured with a yaml file that largely mimics all the standalone.xml data that HWFA has. There is no ${x} support in the YAML file right now.
This new HJA can talk to any EAP or WildFly server over the DMR management API. It can do deployments to your EAP/WildFly servers and monitoring of EAP/WildFly subsystems just like HWFA can.
This new HJA can talk to any JMX server just like HWFA can (it will talk to the local MBeanServer or, if remotely monitoring a JMX server, it will talk to it over Jolokia/REST API).
This new HJA can NOT run directly inside of a Host Controller due to https://issues.jboss.org/browse/WFCORE-2526 - however, you can run HJA externally in its own JVM (e.g. java -jar hawkular-javaagent.jar config=config.yaml) and have its config.yaml point to a remote Host Controller and you'll get the same functionality.
--John Mazz
9 years, 1 month
Garbage collection of outdated Servers/Metrics - especially with (orchestrated) containers
by Heiko W.Rupp
Hi,
I know I brought that up in the past ... (and I am sure Juca will easily
find where :-)
In the good old datacenter, one deployed an application server with
application, added it to monitoring and it hummed there happily. The
server sometimes got rebooted for OS updates or other things, but the
app-server always stayed the same and the monitoring system knew all its
pets by their name and the admins happily did this:
http://www.starwars-union.de/bilder/news/20110401_sunset.jpg
Nowadays in orchestration system the situation more looks like this
http://www.animationsfilme.ch/wp-content/uploads/2013/07/DespicableMe_01.jpg
where containers come and go and an application in container once it has
died is not re-started but a new container with its own ID is started.
Of course we can identify applications with labels so that I don't need
to know the container id. So I can gather and display metrics for those
and all is fine.
But: all those containers will create new
- metric ids
- inventory entries
- ???
The question is now: how long do we want/need to keep them?
Hawkular-metrics has a TTL for the datapoints, but I think metric
definitions are not evicted.
Similar a container being killed can't easily tell inventory that its
entry can go away.
For inventory-new we could use the expiration feature of
Hawkular-metrics for datapoints, where e.g. the agent would regularly
sync data and thus refresh the last-seen time to keep an entry "alive".
Also for the pure metrics - how much of historic data do we want/need.
And if we would e.g. aggregate those for long(er) term storage I think
we could perhaps actually aggregate over all individual time series over
many parallel pods and aggregate them into one for the entire
application.
Heiko
9 years, 1 month
hawkular java agent ready - itests passing
by John Mazzitelli
OK, I have been feverishly finishing this up. I believe this PR is ready. We now have a java agent that can be used in the place of the agent subsystem.
https://github.com/hawkular/hawkular-agent/pull/302
All the unit tests and itests pass - I even have itests running against the javaagent so I can say its working at least as well as the agent as a subsystem. The itests are essentially a copy of the ones we have for the wildfly subsystem agent so I'm exercising the javaagent in the same ways.
Two caveats that I can think of:
1) you can't run javaagent inside host controller directly. You have to run it as a separate process (the javaagent can also run on in its own JVM via "java -jar hawkular-javaagent.jar - it does not HAVE to be attached to some other JVM) and point to the host controller remotely. If you want to run it inside Host Controller, use the Hawkular WildFly Agent as a subsystem (i.e. use it like you do now with the original agent).
2) The "local" DMR mode isn't really local - under the covers it is going through the remote management endpoint so long as the WildFly EAP is configured on the default 127.0.0.1:9990. If it is not, just configure the agent to talk to its local WildFly as if it were a remote-dmr managed server.
To solve 1 we need the host controller to incorporate this enhancement: https://issues.jboss.org/browse/WFCORE-2526
To solve 2 I need to play games with classloading and JBoss Modules - I was given some ideas from some folks. I plan to try something to get that to work.
This javaagent has JMX support. It talks local JMX API if local, and Jolokia REST if remote. It supports cmdgw websocket command ExecuteOperationRequest to invoke JMX operations (itests show this working). So we have full DMR and JMX support.
So this is ready for people to try out for non-EAP based products with JMX as their management interface. Tom Cunningham is testing now. He already spotted a packaging issue with sprintboot which he is figuring out - I'll let him explain if he has the energy :)
I think I can merge this soon, unless someone has reservations.
FWIW: the wildfly agent subsystem support remains intact and as it was. So that didn't change and works like it always has (its itests are still there and passing so I know I didn't break those :).
9 years, 1 month
Inventory storage on metrics
by Joel Takvorian
Quick reminder for those who may not have followed the discussions: this is
about removing the inventory project as it currently stands, and replace it
with Metrics-based (so C*-based) storage, using String metrics and json
blobs.
So, I've been analyzing the usages of inventory in the wildfly agent and in
manageiq / ruby client.
As posted on another topic, I started with the following pattern for
putting inventories in string metrics:
- metrics are named:
*inventory.[feedid].[r|rt|mt].[id]*
which means that we would store JSON blobs by resource type, by metric type
and by root resource (all per feed id). Which also means we don't allow
anything outside of a feed.
This naming pattern seems to be quite easily transposable to the wildfly
agent feeding.
Things become more complicated when we look at the ruby client / manageIQ.
To help in querying, we can add those tags on every metric:
module: inventory
feed: [feedid]
type: [r|rt|mt]
With this, we can easily query, for instance, all resource types of a given
feed, or all feeds, or child resources at a given path, etc.
Now, after analysis of the ruby code (and I hope I didn't miss important
stuff), I have two remarks:
- There is a lot a methods in the ruby client that are not used in manage
IQ (listed below [1]). Of course those methods may have an interest for a
general purpose client, but since we're removing a lot of functionalities
in inventory, I would suggest not to focus on them, and just remove them
for the time being. This is possible, of course, if no other product than
manageIQ consumes then... is it the case?
- I've found just 1 method in the ruby client that would be badly impacted
by the usage of string metrics as currently defined. And unfortunately it's
also the one that is most called (in number of occurrence in code) by
manage IQ, it's "list_resources_for_type"; With the implementation I
described above, to list all resources of a given type we would have to get
all resource blobs of a feed and search in all of them to check all root &
child resource types. That would probably be terrible for performances.
A suggestion to this problem, is to tag each root resource with one tag per
resource type, something like: "
*rt.MyResourceType=resource1,resource2,resource3*".
Then, we would still have to load the whole blobs and search resources in
them, but we would exclude from the search all root resources that doesn't
have those types, and we wouldn't have to search extensively in the blobs
as we would know the paths.
*Question to metrics devs* about that: can we use any character for
tagging, or is there some limitation?
Or if you can suggest a better way, I'm all ears :)
[1] list of methods that seem to be unused from the ruby client:
- list_resources_for_feed
- list_relationships
- list_relationships_for_feed
- get_entity
- list_metrics_for_metric_type
- list_metrics_for_resource_type
- create_feed
- delete_feed
- create_resource_type
- create_resource
- create_resource_under_resource
- create_metric_type
- list_operation_definitions
- list_operation_definitions_for_resource
- create_metric_for_resource
- no_more_events
Joel
9 years, 1 month
Re: [Hawkular-dev] agent running as a VM javaagent - non-EAP based solution
by John Mazzitelli
As for the question "Did you try deployments over this?" - I have been writing a bunch of itests. I basically copied the itests we already have and tweeked them to run against the javaagent. So far, so good. All of the tests pass in this PR with virtually no changes to the core engine (they pass on my box, who knows what travis is doing - sometimes they pass, sometimes they don't).
https://github.com/hawkular/hawkular-agent/pull/302
I need to create two more sets of tests before I can say it all works the same as the wildfly agent without breakage:
1) domain tests (I've got the standalone tests passing - including deployments - just have to test domain mode)
2) immutability tests (show that the agent won't try to change anything if its immutable)
Tests I'm building are here:
https://github.com/jmazzitelli/hawkular-agent/tree/refactor-core/hawkular...
NOTE: the agents (both javaagent and wildfly agent) now support invoking JMX operations in both local and remote JMX servers in my branch - the old agent did not have this support.
9 years, 1 month
agent running as a VM javaagent - non-EAP based solution
by John Mazzitelli
Right now, the Hawkular WildFly Agent runs as a WildFly subsystem extension - which means it must run in EAP.
To avoid this requirement, I refactored the agent code [1] so it can be run using a standard VM javaagent (using the VM argument -javaagent).
This means the agent need not run as a subsystem extension, in fact, it can run in any VM (non-EAP/WildFly based). Just pass in the -javaagent command line argument to your VM and you got an agent. You have a Karaf container in a JVM exposing metrics via JMX? You can use this. You got a vert.x server in a JVM exposing metrics via JMX? You can use this.
This new agent's config file is a single YAML file that mimics virtually the same settings as you see today in the agent's <subsystem> XML in standalone.xml.
Like the EAP-based agent, this new agent can still:
* collect DMR metrics and inventory from any EAP 6.4, EAP 7.x, or WildFly 10+ servers - local or remote.
* collect JMX metrics and inventory from any remote Jolokia endpoint or local JMX MBeanServer
* receive websocket messages from a Hawkular Server (if running in full Hawkular mode)
* run in metrics-only mode where it only stores data to Hawkular-Metrics and does not connect via websocket to the server nor stores inventory to H-Inventory
* Talk to the Hawkular Server and remote DMR/JMX endpoints via SSL/https using defined security-realms/keystores
Unlike the EAP-based agent, this new agent:
* does NOT have a CLI (jboss-cli.sh is the EAP-based agent's CLI)
* does NOT have a built-in configuration persistent mechanicm (EAP-based agents can persist their config changes to standalone.xml when those changes are made via the WildFly management interface - because the javaagent is completely independent, it has nothing like this for its YAML config).
* does NOT integrate with the lifecycle of any EAP/WildFly running in the same VM (in other words, if you ask to "reload" the EAP server, the javaagent knows nothing about that and won't also itself reload. Remember, this javaagent is now completely independent of any co-located EAP/WildFly server).
In addition, if your JVM is installed in OpenShift, you can get the nice option of putting the javaagent's YAML config in a config map and mount it so your javaagent can access it. This means you can edit your javaagent YAML directly in the OpenShift UI :)
I still need to see how well this supports the disabling/enabling of metrics on the fly from the websocket commands (e.g. I will need to add the ability to change the YAML config to persist the settings, that is not done yet - the EAP-based agent got this support for free - its config changes go into standalone.xml).
Other than persisting changes to the YAML file, I believe this javaagent will work essentially the same as the current agent (indeed, the core agent engine is identical - its the same code). Just have to make sure the websocket commands can work considering there is a YAML config backing the settings and not standalone.xml.
I need people's feedback on this. Thoughts? If you want to try it, you can build it from my PR right now using the README [2] and the instructions [3].
--John Mazz
[1] https://github.com/hawkular/hawkular-agent/pull/302
[2] https://github.com/jmazzitelli/hawkular-agent/blob/refactor-core/hawkular...
[3] Instructions to build using the PR and test it by collecting EAP metrics:
1) Build it:
$ git clone git@github.com:hawkular/hawkular-agent.git
$ cd hawkular-agent
$ git checkout -b jmazzitelli-refactor-core master
$ git pull https://github.com/jmazzitelli/hawkular-agent.git refactor-core
$ mvn clean install
2) Install it - assuming you have a base EAP 6.4 or 7.x or WildFly 10+ installed at <eap.dir>
2a) Change this line in <eap.dir>/bin/standalone.conf:
JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman"
to this:
JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman,org.jboss.logmanager"
2b) Inside that same <eap.dir>/bin/standalone.conf you add this line in the
appropriate place (you know where, just under where it sets JAVA_OPTS):
JAVA_OPTS="$JAVA_OPTS -Djava.util.logging.manager=org.jboss.logmanager.LogManager -javaagent:$JBOSS_HOME/bin/hawkular-javaagent-*.jar=config=$JBOSS_HOME/standalone/configuration/real-config*.yaml,delay=10"
2c) Copy the binary/config files you built to that EAP / WildFly install:
cp hawkular-agent/hawkular-javaagent/target/hawkular-javaagent-*-jar-with-dependencies.jar <eap.dir>/bin
if EAP 7.x/WildFly 10+:
cp hawkular-agent/hawkular-javaagent/src/test/resources/real-config.yaml <eap.dir>/standalone/configuration
if EAP 6.4:
cp hawkular-agent/hawkular-javaagent/src/test/resources/real-config-eap6.yaml <eap.dir>/standalone/configuration
9 years, 1 month
Hawkular Metrics 0.25.0 - Release
by Stefan Negrea
Hello,
I am happy to announce release 0.25.0 of Hawkular Metrics. This release is
anchored by general stability improvements and enhanced query capabilities
for the external metrics alerter.
Here is a list of major changes:
- *External Metrics Alerter - Enhancements*
- Both syntax and query capabilities have been revamped to allow
defining conditions using flexible expressions with embedded stats queries
- The ExternalCondition expression is now JSON and has support for
the new tag query language
- The full documentation about this feature can be found in the Alerting
section
<http://www.hawkular.org/hawkular-metrics/docs/user-guide/#_alerting>
of the user guide
<http://www.hawkular.org/hawkular-metrics/docs/user-guide/>
- For more details please see: HWKMETRICS-566
<https://issues.jboss.org/browse/HWKMETRICS-566> and Pull Request 727
<https://github.com/hawkular/hawkular-metrics/pull/727>
- *Dropwizard-Metrics - Merged*
- The Dropwizard-Metrics reporter
<https://github.com/hawkular/hawkular-dropwizard-reporter> has been
merged into the Metrics repository in clients/dropwizard
<https://github.com/hawkular/hawkular-metrics/tree/44677cc5d6267e738ea51bd...>
module
- The old repository
<https://github.com/hawkular/hawkular-dropwizard-reporter> is
decommissioned, all new development will happen in the Metrics project
- This was done to simplify the compatibility matrix between the
reporter and Hawkular Metrics REST API; going forward they will have
identical versions. Furthermore, the compatibility is now tested via
continuous integration tests.
- Please use the new maven artifact hawkular-dropwizard-reporter
<http://origin-repository.jboss.org/nexus/content/repositories/public/org/...>
- For more details please see: HWKMETRICS-585
<https://issues.jboss.org/browse/HWKMETRICS-585>
- *Tag Query Language - Enhancements*
- The tag query language now supports the dot character in the tag
name. The list of allowed characters is *a-zA-Z_0-9.*
- The query language allows regex matching for tag values but not tag
names
- This allows translating JSON-like tag structures into Hawkular
Metrics tags and query using the new tag query language
- For example, a tag structure like [tag.subtag1: value1,
tag.subtag2.subsubtag1: value2] is now queriable via the tag query
language with queries like tag.subtag1 = value1 or
tag.subtag2.subsubtag1
- For more details please see: HWKMETRICS-611
<https://issues.jboss.org/browse/HWKMETRICS-611>
- *REST API - Request Logging*
- It is now possible to enable detailed logging for all REST API
requests
- Two properties have been added to enable this features:
hawkular.metrics.request.logging.level to enable logging for all read
requests and hawkular.metrics.request.logging.level.writes to enable
logging for write requests
- By default this feature is disabled, to enable just set the log
level via each property
- For more details please see: HWKMETRICS-589
<https://issues.jboss.org/browse/HWKMETRICS-589>
- Here is a sample log:
INFO [org.hawkular.metrics.api.jaxrs.util.RequestLoggingFilter]
(default task-49)
REST API request:
--------------------------------------
path: /metrics
segments: [metrics]
method: GET
query parameters: {type=[availability]}
Tenant: T9a116f18-28cf-41b3-8ff8-c9752ac60e26232
- *Other Updates*
- Automatically fix schema issues that occur when the server is
restarted during initial schema installation (HWKMETRICS-594
<https://issues.jboss.org/browse/HWKMETRICS-594>)
- Metric data points inserts have been optimized to use token ranges
for Cassandra writes (HWKMETRICS-599
<https://issues.jboss.org/browse/HWKMETRICS-599>)
*Hawkular Alerting - Included*
- Version 1.5.3
<https://issues.jboss.org/projects/HWKALERTS/versions/12333651>
- Project details and repository: Github
<https://github.com/hawkular/hawkular-alerts>
- Documentation: REST API
<http://www.hawkular.org/docs/rest/rest-alerts.html>, Examples
<https://github.com/hawkular/hawkular-alerts/tree/master/examples>,
Developer
Guide
<http://www.hawkular.org/community/docs/developer-guide/alerts.html>
*Hawkular Metrics Clients*
- Python: https://github.com/hawkular/hawkular-client-python
- Go: https://github.com/hawkular/hawkular-client-go
- Ruby: https://github.com/hawkular/hawkular-client-ruby
- Java: https://github.com/hawkular/hawkular-client-java
*Release Links*
Github Release:
https://github.com/hawkular/hawkular-metrics/releases/tag/0.25.0
JBoss Nexus Maven artifacts:
http://origin-repository.jboss.org/nexus/content/repositorie
s/public/org/hawkular/metrics/
Jira release tracker:
https://issues.jboss.org/projects/HWKMETRICS/versions/12333676
A big "Thank you" goes to John Sanda, Matt Wringe, Michael Burman, Joel
Takvorian, Jay Shaughnessy, Lucas Ponce, and Heiko Rupp for their project
contributions.
Thank you,
Stefan Negrea
9 years, 1 month
Re: [Hawkular-dev] Command gateway and WF agent
by John Mazzitelli
> Can you explain to me what's the big picture of the command gateway? What's
> the chain of calls when, for instance, there's a new deployment in Wildfly?
Joel asked this question - I figured it is good to post to h-dev, too, since I'm sure others don't know this.
The "big picture" is:
First, remember the MiQ UI is connected to Hawkular-Services server via websocket. The agents all connect to some Hawkular-Services server via a websocket. There can be more than one Hawkular-Services server running and the UI may be talking to a server that is different than the server the agent is talking to:
UI <---websocket---> Server A
^
|
(message bus)
|
V
Server B <---websocket---> agent
1) MiQ sends a JSON message over the websocket to a Hawkular-Services server (Server A in the diagram above).
2) Hawkular-Services server A looks at the ResourcePath in the message to determine what agent is responsible for managing that resource.
3) Hawkular-Services server A addresses the message to the proper agent and puts the JSON message on the agent's queue on the message bus.
3b) At this point the Hawkular-Services server A sends back a JSON message to the UI over the websocket that says "message forwarded to the bus".
4) The Hawkular-Services server that has a websocket connection to the targeted agent picks off that message from the bus (Server B in the diagram)
5) That Hawkular-Services server B forwards the message to the agent via its websocket connection.
6) Agent looks the the message's ResourcePath and the rest of the JSON to determine what it needs to do. It does the action.
// now the process goes in reverse
7) Agent sends back a response to the server B over websocket (either a success or fail message)
8) Server B takes response, figures out which UI the response should go to, and puts it on message bus addressed to the correct UI
9) The Server A takes the response message off the bus
10) Server A sends the response message to the UI over its websocket connection.
> Tell me if I'm correct, for what I've seen it's MIQ which, though the ruby
> client, sends a websocket event, caught by the WF agent through the command
> gateway. In the end, the WF agent will perform a full sync of affected root
> resources in inventory. Is that correct?
In the case of a "Deployment" JSON message, yes, the agent will trigger a full discovery scan so it can quickly discover the new deployment you just added to WildFly.
> If that's correct, there's no direct interaction between inventory and the
> command gateway. I'm asking because I hope we can keep the command gateway
> messages unchanged, and change only places where there's direct calls to
> the inventory rest api.
Most of the time, its just using ResourcePath API (just that simple inventory POJO). HOWEVER, I believe Jay added some code that does interact with Inventory to do things like create relationships in inventory when new cluster entities are added. That is why (I think) he did this commit. Though I can't remember where this code is that builds inventory relationships - he'll have to point that out. But I'm pretty sure there is some inventory stuff going on under the covers for that stuff - more than just using the canonical ResourcePath stuff.
9 years, 1 month
