Hawkular Metrics - REST API Remake
by Stefan Negrea
The following is a new meeting request:
Subject: Hawkular Metrics - REST API Remake
Organizer: "Stefan Negrea" <snegrea(a)redhat.com>
Location: Conf - 398-055-2127 & BlueJeans
Time: Monday, April 13, 2015, 9:00:00 AM - 10:00:00 AM GMT -06:00 US/Canada Central
Invitees: jsanda(a)redhat.com; hawkular-dev(a)lists.jboss.org
*~*~*~*~*~*~*~*~*~*
Hello Everybody,
I would like to schedule a meeting to discuss a proposal for changes to Hawkular Metrics REST API. The agenda also includes recent proposals from the channel and mailing list around tags and raw vs aggregated metrics from an end-point perspective.
Here is a preliminary simple spreadsheet for this remake:
https://docs.google.com/a/redhat.com/spreadsheets/d/1CQBfAenNlWbQENoK_Rms...
Thank you,
Stefan
9 years
Metrics, tags and search capabilities
by Michael Burman
Hi,
Earlier today I created HWKMETRICS-54, but I later thought about it a bit more and to me it looks like we're not sure what the tags should really do and how the system should be used. Lets assume the following scenario:
3 machines, each one running an agent that provides data for memory.free, memory.cached, cpu.idle, cpu.user, cpu.system, disk.free, disk.used
Now what user might want to do in these cases is:
a) I want to get all the statistics affecting the host 1
b) I want to get all the memory.free statistics from each host
Think about the data modeling in our current hawkular-metrics for a moment. The user starts:
I. Model everything with machine1.memory.free, machine2.memory.free etc
a) How to query machine1.* ? Can't be done. c) How to get *.memory.free? Can't be done.
How would the query succeed with our current format? By creating tags for every occasion on metric creation:
create machine1.memory.free (tags: machine='machine1', category='memory')
create machine2.memory.free (tags: machine='machine2', category='memory')
What the user finds out is that "why on earth do I have the metricId at all" ? It's a good question. In our current structure we should remove metricId and just invent something random for better Cassandra partitioning.
What it probably should look like (and this is how I assumed it was to be done until I checked the unit tests and find out that there's nothing pointing to this OpenTSDB familiar method):
II. memory.free, cpu.idle are the metricIds and I'll define it has a parameter 'machine'. When pushing a metric I set a tag with a value, such as machine='machine2'.
Now when I fetch the metric "memory.free", I can get all the memory.free valuess with 'machine'-tag indicating which machine it was gathered from. If I need to search for all machine-statistics, then I could use the tag-searching. If I wanted only machine1 memory.free, I would add a filter: tag machine='machine1'.
Or how are we supposed to model real-world-use-cases? The current model is quite cumbersome and not even necessarily doable in many cases (am I supposed to query for a metric definition before pushing any metric - because a new container could give me new set of parameters or a new machine new set of machine parameters and I need to remember to register them instead of pre-defined types which I might know).
- Micke
9 years
${project.version} in dependency versions considered harmful
by John Mazzitelli
FYI:
----- Forwarded Message -----
Subject: ${project.version} in dependency versions considered harmful
Hey everyone,
I just wanted to give a little PSA about this insidious little
expression (see $subject).
Using that expression in dependency declarations seems like a shortcut,
but it can go wrong in SO MANY WAYS. By far the most common problem is
the use of ${project.version} in a BOM or parent POM.
If anyone inherits from that parent POM or imports that BOM in an
external project, that external project's version will be used in place
of the one that the parent POM / BOM intended, and all of your carefully
managed dependencies will be wrong.
Example: jboss-as-console-bom-2.5.5.Final-redhat-1.pom
This declares org.jboss.as:console-spi:sources:${project.version}:jar.
Then, the Teiid build imports that BOM and uses it when it builds
against the console-core library. The above sources jar listed as a
second-level dependency (coming in via console-core) uses the Teiid
project version, and everything grinds to a halt.
Please, if you find dependency declarations using ${project.version},
fix it! If there are many, many references, simply switch to using a
property (eg. ${consoleVersion} in place of ${project.version})...and
DON'T use ${project.version} as the value for that property.
9 years
Hawkular 0.3.2 & Beyond - Release Planning
by Stefan Negrea
Hello Everybody,
Here is a list of major features and enhancements planned for 0.3.2 and beyond:
1) REST API - the end-points will be separated based on the metric type. Current availability and numeric, will be joined by counter and gauges metrics.
2) Counters and gauges - add two new specialized metric types to allow more flexibility for clients.
3) Numeric Aggregates - Will allow long-term storage of numeric metrics at the expense of losing some fidelity. The design is already in progress. Because this is a really complex feature, the expectation is to start the work in 0.3.2 and publish a mini-roadmap.
4) Enhanced Availability - while the current model works, the goal would be expand this to cover most of the use cases brought up in the community threads. We will start the design in 0.3.2 with the implementation expected in future releases.
4) Go client - will help integrating with third party metrics collection framework. This work is the foundation for the Heapster sink.
5) Public Java API - following the work done in 0.3.1 in the core of the project, the goal is to separate the implementation from a public API and make that available to clients
6) Update REST testing - the current set of tests is a good gauge for regressions, but the overall coverage is still low. The plan for 0.3.2 is to increase coverage.
Any discussion, suggestions or contributions are more than welcomed; so feel free to reply to this email or join #hawkular on freenode.
Thank you,
Stefan Negrea
Software Engineer
9 years
start of message bus REST client
by John Mazzitelli
I created a very simple REST client that you can use to send any json string as a bus message.
It requires no dependencies other than apache httpclient and jboss logging.
See: https://github.com/hawkular/hawkular-bus/tree/master/hawkular-bus-rest-cl...
new org.hawkular.bus.restclient.RestClient(url).postTopicMessage(topicName, jsonPayload, headers);
My agent uses it to send metrics to the hawkular bus's metric topic. I use REST rather than the bus-common API because (AFAIK) the thinking is this agent can be deployed in something like WildFly Core where JMS API isn't even available.
So right now, this agent should be sending data just like the Pinger - its sending metrics directly to Metrics REST interface for storage and sending those same metrics to the message bus's metric topic so other things like alerts can process it. Unfortunately, the kettle UI can't show any of it since it is tightly related to the Pinger's metrics and nothing else. But I think its working :)
9 years
Public Hawkular Instances
by Matthew Mahoney
A couple notes on the Public Hawkular instance:
1) Improvements have been made to the Public Hawkular instance ( http://209.132.178.218:18080 ) smoke test, as to mitigate the number of false failures that were happening.
We invite & encourage community use of this instance.
2) A new Public Hawkular instance ( http://209.132.178.218:18090 ) has been created which is intended to be used by Development/community for those use-cases such as Demos where you need to ensure that the instance will not be randomly restarted (for reasons such as when new Hawkular images are created, test cases git pushes, infrastructure improvements, etc).
This instance is an on-demand build only. Please ping/email me ( mmahoney(a)redhat.com ) if you want access to the Jenkins Hawkular Dev build job.
Thanks,
Matt
9 years
Kettle build profiles
by Viliam Rockai
Hello team,
TL;DR: Feel free to use -Pdev profile during the kettle build again. It
should not break the console stuff anymore.
There was a lot of confusion about the -Pdev profile in the
hawkular/hawkular project. Originally this was created just to provide
caching of bower packages and node.js stuff during the console build.
But then, there was some new functionality added to the profile, more
people started to use it and caching caused issues. To fix that, I've
renamed the "caching" profile (from "dev") to "cache". So now we have
three profiles available during the kettle build:
1. -Pdev: Useful for everyone and safe. It unzips the assembly, so it
can be started directly from /target directory and creates a default
user.
2. -Pcache: Useful for everyone but potentionaly dangerous. Using this
profile only makes sense with the "clean" lifecycle. It basically just
excludes the node.js stuff from the clean task, so those files are not
deleted => they are not downloaded again during the following build.
However, if there are changes in JS packages/JS build files used by the
console, you will still see the old/cached ones or even won't be able to
build the console. If you're using the -Pcache profile and the console
build fails, try to rebuild without using this profile.
3. -Plink: Useful for UI developers. This will use the local version of
the hawkular-ui-components. More info:
https://github.com/hawkular/hawkular/tree/master/ui#hawkular-ui-component...
Viliam
9 years
availability and metric endpoints
by John Sanda
There has been some good discussion around availability lately. I want to add one more to the mix, but hopefully this one is not as in-depth as some of the other topics. Right in metrics we have endpoints like,
POST /metrics/numeric/data
GET /metrics/numeric/{id}/data
POST /metrics/availability/data
GET /metrics/availability/{id}/data
I would like to change these to,
POST /metrics/data
GET /metrics/{id}/data
POST /availability/data
GET /availability/{id}/data
I think the “metrics” prefix is awkward and unnecessary. I think that it is intuitive enough that metrics on its own refers to numeric data. Thoughts?
- John
9 years
