agent using custom metric IDs
by John Mazzitelli
<tl;dr> ======
Agent is introducing two changes:
1. Metric Type definitions created by the agent will have the same ID as before, but its Name is changing (probably does not affect anyone).
2. Clients (like UI, HawkFX, etc) should no longer assume the agent's h-inventory metric definition IDs match h-metric metric IDs - instead, they must look at the "metric-id" property on the h-inventory metric definition to know how to look up the actual metric data in h-metrics. Will affect all clients, but metric IDs will default to what they are today - so nothing changes and thus nothing will break today if you run with out of box configuration.
</tl;dr> ======
There is a use-case where the agent needs to support custom metric IDs (that is, rather than accepting the out-of-box metric IDs created by the agent, allow the user to define what the metric IDs should look like). See https://issues.jboss.org/browse/HWKAGENT-78
As a refresher, remember that when you create resources in inventory, those resources can be associated with one or more "metric" definitions. Those resource metrics are themselves associated with a "metric type" definition. Today, when the agent stores metric data into Hawkular Metrics, it stores the data under the ID of the "metric" that is associated with the resource (so the h-inventory metric ID is the same as the h-metric metric ID by definition - at least for the data the hawkular wildfly agent inserts).
I am proposing two changes in the PR: https://github.com/hawkular/hawkular-agent/pull/226
First, today, the "metric type" definition that the agent creates has an ID and a Name that are identical. I am changing this so the ID stays the same (which is the metric set name, followed by "~", followed by the name of the metric -- e.g. if there was a <metric-set-dmr name="this"> that contains a <metric-dmr name="that">, the metric type ID would be "this~that"), but the Name is only the name without the set name (e.g. the name would be "that" in the previous example).
The above is a minor change, and I doubt anyone is affected by it. But I point it out just in case.
Second, it should no longer be assumed that the inventory's resource metric ID is identical to the h-metric's metric ID.
This second change will potentially affect everyone (I know it affects Heiko's HawkFX :)
Now, that said, nothing really changes now, because the defaults will remain as they are (that is, the h-inventory's metric ID will still be exactly the same as the h-metrics ID - the agent keeps them identical). The change happens when the user actually configures the agent with a custom metric ID template (e.g. <remote-jmx metric-id-template="${pod.id}-%MetricTypeName" ...>). This means h-metric IDs will be DIFFERENT than h-inventory metric IDs.
How then does a client know what h-metric IDs to look for if they only have h-inventory metric definitions? Well, recall that inventory allows for properties to be associated with any entity. I use this feature here. Rather than rely on an implicit rule ("h-inventory metric ID is the same as h-metric metric ID") I explicitly define this linkage in a property called "metric-id" on the h-inventory metric definition. Out of box, that property's value will be identical to the h-inventory metric ID (and hence why nothing really changes - since the explicit rule in this case provides the same behavior as if following the old implicit rule). In fact, I'm considering if I should set that property at all if its the same as the h-inventory ID - I think it might be better to only set a "metric-id" property if it is different. But this would require clients to know about the implicit rule if there is no metric-id property set ("is there a metric-id property set? No? Then use the h-inventory metric ID for the h-metric metric ID").
For example, see here (this is a live example I copied from the "raw" inventory JSON that HawkFX gave me for a metric) - this is the h-inventory entity definition for the metric "Heap Used" on my WildFly Server resource - notice the "properties" map has a "metric-id" value that is DIFFERENT than the "id" - that "metric-id" is something I customized in my agent config in standalone.xml (well, I used the swarm agent, so I put it in the swarm config, but its basically the same thing):
{
"path": "/t;hawkular/f;mazz/m;MI~R~%5Bmazz%2FWildFly~~%5D~MT~WildFly%20Memory%20Metrics~Heap%20Used",
"properties": {
"__identityHash": "70e59a5d427632223da36c225ba6ef8572985",
"metric-id": "feed=mazz__msn=WildFly__typeName=Heap Used__resName=WildFly Server [WildFly]__resId=WildFly~~__typeId=WildFly Memory Metrics~Heap Used"
},
"name": "Heap Used",
"identityHash": "70e59a5d427632223da36c225ba6ef8572985",
"type": {
"path": "/t;hawkular/f;mazz/mt;WildFly%20Memory%20Metrics~Heap%20Used",
"name": "Heap Used",
"identityHash": "3be5b5fdabed925ac46fdc6d8295e34bbd3147a",
"unit": "BYTES",
"type": "GAUGE",
"collectionInterval": 30,
"id": "WildFly Memory Metrics~Heap Used"
},
"id": "MI~R~[mazz/WildFly~~]~MT~WildFly Memory Metrics~Heap Used"
}
Notice this:
feed=mazz__msn=WildFly__typeName=Heap Used__resName=WildFly Server [WildFly]__resId=WildFly~~__typeId=WildFly Memory Metrics~Heap Used
is different from this:
MI~R~[mazz/WildFly~~]~MT~WildFly Memory Metrics~Heap Used
And that's the issue. Clients have to know to look for the "metric-id" property and use it when looking up metric data in h-metrics (so if you want to graph the data, you have to ask h-metrics for the data associated with the value found in the "metric-id" property).
8 years, 5 months
Status of integration of inventory and vertx metrics
by Austin Kuo
Hi,
The current status is that I have created
- a feed
- a root resource with a resource type
- a eventbus resource as a child of the root resource above with a
resource type
- a eventbus.handlers metrics with gauge metric type. This has a
property 'metric-id' corresponding to the id of the metric data.
Fortunately, now I can view the metric data graph from the client (hawkfx).
Austin,
Thanks.
8 years, 5 months
integrating ansible and hawkular to start remote wildfly servers
by John Mazzitelli
I have a very simple PoC working that shows Ansible integrated with Hawkular such that it can start a WildFly server remotely.
I'll begin with a brief summary.
SUMMARY
A client connects to the Hawkular Server via its websocket and passes in a JSON request (like it does the other kinds of requests). The JSON request looks something like:
AnsibleRequest={"playbook":"start-wildfly.yml", "extraVars": { "wildfly_home_dir":"/opt/wildfly10" } }
The command-gateway server takes the request, runs the Ansible playbook, and returns the response back over the websocket to the client with an AnsibleResponse (which includes the Ansible output to show what it did).
DETAILS
The code is in two branches in my repos. See [1] and [2]. You need both for this to work.
Hawkular-Services packages up Ansible playbooks and its associated files in standalone/configuration/ansible. See [3].
Hawkular-Commons adds to command gateway a new JSON request/response for Ansible requests. See [4] and [5].
The Ansible command is invoked in the server when the AnsibleRequest JSON is received over the websocket. This command implementation runs the Ansible playbook and returns the results back in a AnsibleResponse. See [6].
If you build commons and then services to pull the new commons in, you can run a test to see it work. I use the test command CLI utility in the agent to do this [7]. First, build the new dist and run it so you have a Hawkular Service running (for example, "mvn clean install -Pdev,embeddedc"). Next, create a test JSON request file (say, "/tmp/hawkular.json") with this content:
{ "playbook":"start-wildfly.yml",
"extraVars": {
"wildfly_home_dir":"/directory/where/you/installed/wildfly"
}
}
Now build the hawkular-agent from source (just so you get the test command CLI utility) and run the test command CLI, telling it to use the JSON in your file and send it to your Hawkular Service server:
$ cd <hawkular-agent-source-dir>/hawkular-wildfly-agent-itest-parent/hawkular-wildfly-agent-command-cli/target
$ java -jar hawkular-wildfly-agent-command-cli-*.jar --username jdoe --password password \
--command AnsibleRequest --request-file=/tmp/hawkular.json
This sends the request to your local Hawkular Service server over the websocket, the Ansible playbook "start-wildfly.yml" is run, which starts the WildFly in that home dir you specified in your JSON request, and the response is sent back. The Command CLI tool will write the responses it receives to disk - look at the cli output for the names of the files it writes - you can look in there to see what the Ansible command response was (its just the full Ansible output).
There is still LOTS to do here (for one, you'll notice it assumes your wildfly install is on "localhost" :)). This is merely a PoC that we can use as a starting point - it shows that this can be done and how we can do it.
-- John Mazz
[1] https://github.com/jmazzitelli/hawkular-commons/tree/HAWKULAR-1096-ansibl...
[2] https://github.com/jmazzitelli/hawkular-services/tree/HAWKULAR-1096-ansib...
[3] https://github.com/jmazzitelli/hawkular-services/tree/HAWKULAR-1096-ansib...
[4] https://github.com/jmazzitelli/hawkular-commons/blob/HAWKULAR-1096-ansibl...
[5] https://github.com/jmazzitelli/hawkular-commons/blob/HAWKULAR-1096-ansibl...
[6] https://github.com/jmazzitelli/hawkular-commons/blob/HAWKULAR-1096-ansibl...
[7] https://github.com/hawkular/hawkular-agent/tree/master/hawkular-wildfly-a...
8 years, 5 months
Hawkular APM / Zipkin Adapter
by Gary Brown
Hi
Here is a short demo showing the initial work capturing information reported by Zipkin clients and translating it for use by the Hawkular APM server.
https://youtu.be/x6H2YJi2v1o
We will be adding more capabilities in the coming weeks. If you have any specific requirements please let us know (or just create a jira).
If anyone has a polyglot example instrumented with zipkin clients that we could test please get in contact.
Regards
Gary
8 years, 5 months
Hawkular Services 0.0.6.Final
by Juraci Paixão Kröhling
Team,
Hawkular Services 0.0.6.Final has just been released.
The only functional change between this and the previous version is that
Hawkular Services now reports itself as a "Hawkular" server, instead of
"Wildfly" (HAWKULAR-1098)
As the previous distributions, the Agent has to be configured with an
user. This can be accomplished by:
- Adding an user via bin/add-user.sh like:
./bin/add-user.sh \
-a \
-u <theusername> \
-p <thepassword> \
-g read-write,read-only
- Changing the Agent's credential on standalone.xml to the credentials
from the previous step or by passing hawkular.rest.user /
hawkular.rest.password as system properties (-Dhawkular.rest.user=jdoe)
You can find the release packages, sources and checksums at GitHub, in
addition to Maven:
https://github.com/hawkular/hawkular-services/releases/tag/0.0.6.Final
Shortcuts for the downloads:
Zip - https://git.io/vKwxm
tar.gz - https://git.io/vKwxI
- Juca.
8 years, 5 months
Integrate metric data with inventory
by Austin Kuo
Hi,
I’m currently working on vertx metric agent, my job is to enable the
inventory such that users can view the inventory and corresponding metrics.
My question is that how “real metric data” and “inventory metrics” are
matched?
Is it simply bound via id?
Thanks,
Austin
8 years, 5 months
Hawkular Services vs Hawkular Community
by Thomas Heute
(If you don't know about Hawkular Community you can skip this email
completely)
After some discussions about Hawkular Community, here is our conclusion:
- We will not have a separate Hawkular Community repository
- We will ship 2 distributions for Hawkular Services,
- one for dev/demo/quick test that will have an embedded
Cassandra, a default user and the embedded agent enabled (This is what you
get when building Hawkular Services today with "-Pembeddedc -Pdev").
- the other one that requires an external Cassandra server,
no default user and the agent being disabled.
- We'd like a simple UI to explore data (metrics, inventory, alerts),
but this is not a priority, if someone in the community is looking for an
Angular project it would be great, please contact us on this mailing list.
We would advertise that work in the "Client" section of the revamped
hawkular.org [1] as a separate download. (Note that this would likely be
purely static content and could even be made available on a CDN)
- For other side projects that are not mature to be included in Hawkular
Community or side projects, they will be listed either in the "client" or
"labs" section of the revamped website [1] depending on their maturity and
scope.
Thanks,
Thomas
[1] WIP: http://209.132.178.114:10188/
8 years, 5 months
inventory and resource config props
by John Mazzitelli
I ran into an Inventory issue today. I don't know what is the expected behavior - I hope what I saw isn't expected :)
I ran the agent and it added everything in inventory successfully.
I then shutdown the agent, added a new resource configuration property definition to a resource in the agent configuration, then re-ran the agent.
The agent successfully found and attempted to add the new resource configuration property via the inventory bulk/ API, but inventory never stored it. When I look in inventory, I do not see my new resource configuration property attached to the already existing resource. The resource config props that were on the resource originally are still there, but the new one was never added.
The agent builds the bulk payload in its AsyncInventoryStorage class - specifically, you can see adding the resource config props here: https://github.com/hawkular/hawkular-agent/blob/master/hawkular-wildfly-a...
What is wrong?
8 years, 5 months
