Metrics storage usage and compression
by Michael Burman
Hi,
Lately there has been some discussion on the AOS scalability lists for our storage usage when used in Openshift. While we can scale, the issue is that some customers do not wish to allocate large amounts of storage for storing metrics, as I assume they view metrics and monitoring as secondary functions (now that's whole another discussion..)
To the numbers, they're predicting that at maximum scale, Hawkular-Metrics would use close to ~4TB of disk for one week of data. This is clearly too much, and we don't deploy any other compression methods currently than LZ4, which according to my tests is quite bad for our data model. So I created a small prototype that reads our current data model, compresses it and stores it to a new data model (and verifies that the returned data equals to sent data). For testing I used a ~55MB extract from the MiQ instance that QE was running. One caveat of course here, the QE instance is not in heavy usage. For following results, I decided to remove COUNTER type of data, as they looked to be "0" in most cases and compression would basically get rid of all of them, giving too rosy picture.
When storing to our current data model, the disk space taken by "data" table was 74MB. My prototype uses the method of Facebook's Gorilla paper (same as what for example Prometheus uses), and in this test I used a one day block size (storing one metric's one day data to one row inside Cassandra). The end result was 3,1MB of storage space used. Code can be found from bitbucket.org/burmanm/compress_proto (Golang). I know Prometheus advertises estimated 1.3 bytes per timestamped value, but those numbers require certain sort of test data that does not represent anything I have (the compression scheme's efficiency depends on the timestamp delta and value deltas and delta-deltas). The prototype lacks certain features, for example I want it to encode compression type to the first 1 byte of the header for each row - so we could add more compression types in the future for different workloads - and availabilities would probably have better compression if we changed the disk presentation to something bit based.
** Read performance
John brought up the first question - now that we store large amount of datapoints in a single row, what happens to our performance when we want to read only some parts of the data?
- We need to read rows we don't need and then discard those
+ We reduce the amount of rows read from the Cassandra (less overhead for driver & server)
+ Reduced disk usage means we'll store more of the data in memory caches
How does this affect the end result? I'll skip the last part of the advantage in my testing now and make sure all the reads for both scenarios are happening from the in-memory SSTables or at least disk cache (the testing machine has enough memory to keep everything in memory). For this scenario I stored 1024 datapoints for a single metric, storing them inside one block of data, thus trying to maximize the impact of unnecessary reads. I'm only interested in the first 360 datapoints.
In the scenario, our current method requests 360 rows from Cassandra and then processes them. In the compressed mode, we request 1 row (which has 1024 stored metrics) and then filter out those we don't need in the client. Results:
BenchmarkCompressedPartialReadSpeed-4 275371 ns/op
BenchmarkUncompressedPartialReadSpeed-4 1303088 ns/op
As we can see, filtering on the HWKMETRICS side yields quite a large speedup instead of letting Cassandra to read so many rows (all of the rows were from the same partition in this test).
** Storing data
Next, lets address some issues we're going to face because of the distributed nature of our solution. We have two issues compared to Prometheus for example (I use it as an example as it was used by one Openshift PM) - we let data to arrive out-of-order and we must deal with distributed nature of our data storage. We are also stricter when it comes to syncing to the storage, while Prometheus allows some data to be lost in between the writes. I can get back to optimization targets later.
For storing the data, to be able to apply this sort of compression to it, we would need to always know the previous stored value. To be able to do this, we would need to do read-write path to the Cassandra and this is exactly one of the weaknesses of Cassandra's design (in performance and consistency). Clearly we need to overcome this issue somehow, while still keeping those properties that let us have our advantages.
** First phase of integration
For the first phase, I would propose that we keep our current data model for short term storage. We would store the data here as it arrives and then later rewrite it to the compressed scheme in different table. For reads we would request data from the both tables and merge the results. This should not be visible to the users at all and it's a simple approach to the issue. A job framework such as the one John develops currently is required.
There are some open questions to this, and I hope some of you have some great ideas I didn't think. Please read the optimization part also if I happened to mention your idea as some future path.
- How often do we process the data and do we restrict the out-of-order capabilities to certain timeslice? If we would use something like 4 hour blocks as default, should we start compressing rows after one hour of block closing? While we can technically reopen the row and reindex the whole block, it does not make sense to do this too often. If we decide to go with the reindexing scenario, in that case we could start writing the next block before it closes (like every 15 minutes we would re-encode the currently open blocks if they have new incoming data). We have to be careful here as to not overwhelm our processing power and Cassandra's. This is a tradeoff between minimum disk space usage or minimum CPU/memory usage.
- Compression block size changes. User could configure this - increasing it on the fly is no problem for reads, but reducing is slightly more complex scenario). If user increases the size of the block, our query would just pick some extra rows that are instantly discarded, but nothing would break. However, decreasing the size would confuse our Cassandra reads unless we know the time of the block size change and adjust queries accordingly for times before this event and after.
** Optimizing the solution
The following optimizations would increase the performance of Hawkular-Metrics ingestion rate a lot and as such are probably worth investigation at some point. But they're also complex and I would want to refrain from implementing them in the first phase so that we could get compression quicker to the product - so thta we would not miss certain deadlines.
- Stop writing to the Cassandra in the first phase. Instead we write to something more ephemeral, such as mmap backed memory cache that is distributed among the Hawkular nodes. It would also need some sort of processing locality (direct the write to the node that controls the hash of the metricId for example - sort of like HBase does), unless we want to employ locks to prevent ordering issues if we encode already in the memory. From memory we would then store blocks to the permanent Cassandra store. The clients need to be token/hash-method aware to send data to the correct node.
Benefits for that solution is increased write speed as we such backend easily reaches a million writes per second and the only bottleneck would be our JSON parsing performance. Reads could be served from both storages without much overhead. This optimization would be worth it even without the compression layer, but I would say this is not our most urgent issue (but if the write ingestion speed becomes an issue, this is the best solution to increasing and it's used in many Cassandra solutions, for time series I think SignalFX uses somewhat same approach, although they first write to Kafka).
7 years, 9 months
Inventory API question
by Austin Kuo
I was creating 2 different resource types with the same http client at the
same time.
But one succeed, the other failed with the response 400 and body:
{
"errorMsg" : "The transaction has already been closed"
}
Is it not allowed to do so?
Austin.
7 years, 9 months
Hawkular.org
by Thomas Heute
With the recent repackaging effort + the ManageIQ UI, we need to rethink
hawkular.org.
It's tricky to keep it simple, what to download, for what usecase... And at
the moment it's getting outdated.
IMO, it should be trivial for end users to figure out:
- What to download
- where to download and how to install the server
- find/browse/download the various options to feed the server with data
(WF agent, ptrans supported protocoles, ...)
- find/browse/download options to use/read the data: (ManageIQ,
Grafana...)
and for the developers
- find/browse/download the client libraries (Ruby, Python...)
- Find client libraries documentation
- Find examples...
We should probably not separate/fork Hawkular Services from Hawkular
"community" too much, beside the installation procedure and few things that
we would mention as only available in the "larger" package. I use the term
"Hawkular Server" in the mind map for the 2 "flavors".
Please propose improvements to the draft attached that represents the
website structure. (A square is not necessarily a page, it may be a section
of a page)
Thomas
7 years, 9 months
agent using custom metric IDs
by John Mazzitelli
<tl;dr> ======
Agent is introducing two changes:
1. Metric Type definitions created by the agent will have the same ID as before, but its Name is changing (probably does not affect anyone).
2. Clients (like UI, HawkFX, etc) should no longer assume the agent's h-inventory metric definition IDs match h-metric metric IDs - instead, they must look at the "metric-id" property on the h-inventory metric definition to know how to look up the actual metric data in h-metrics. Will affect all clients, but metric IDs will default to what they are today - so nothing changes and thus nothing will break today if you run with out of box configuration.
</tl;dr> ======
There is a use-case where the agent needs to support custom metric IDs (that is, rather than accepting the out-of-box metric IDs created by the agent, allow the user to define what the metric IDs should look like). See https://issues.jboss.org/browse/HWKAGENT-78
As a refresher, remember that when you create resources in inventory, those resources can be associated with one or more "metric" definitions. Those resource metrics are themselves associated with a "metric type" definition. Today, when the agent stores metric data into Hawkular Metrics, it stores the data under the ID of the "metric" that is associated with the resource (so the h-inventory metric ID is the same as the h-metric metric ID by definition - at least for the data the hawkular wildfly agent inserts).
I am proposing two changes in the PR: https://github.com/hawkular/hawkular-agent/pull/226
First, today, the "metric type" definition that the agent creates has an ID and a Name that are identical. I am changing this so the ID stays the same (which is the metric set name, followed by "~", followed by the name of the metric -- e.g. if there was a <metric-set-dmr name="this"> that contains a <metric-dmr name="that">, the metric type ID would be "this~that"), but the Name is only the name without the set name (e.g. the name would be "that" in the previous example).
The above is a minor change, and I doubt anyone is affected by it. But I point it out just in case.
Second, it should no longer be assumed that the inventory's resource metric ID is identical to the h-metric's metric ID.
This second change will potentially affect everyone (I know it affects Heiko's HawkFX :)
Now, that said, nothing really changes now, because the defaults will remain as they are (that is, the h-inventory's metric ID will still be exactly the same as the h-metrics ID - the agent keeps them identical). The change happens when the user actually configures the agent with a custom metric ID template (e.g. <remote-jmx metric-id-template="${pod.id}-%MetricTypeName" ...>). This means h-metric IDs will be DIFFERENT than h-inventory metric IDs.
How then does a client know what h-metric IDs to look for if they only have h-inventory metric definitions? Well, recall that inventory allows for properties to be associated with any entity. I use this feature here. Rather than rely on an implicit rule ("h-inventory metric ID is the same as h-metric metric ID") I explicitly define this linkage in a property called "metric-id" on the h-inventory metric definition. Out of box, that property's value will be identical to the h-inventory metric ID (and hence why nothing really changes - since the explicit rule in this case provides the same behavior as if following the old implicit rule). In fact, I'm considering if I should set that property at all if its the same as the h-inventory ID - I think it might be better to only set a "metric-id" property if it is different. But this would require clients to know about the implicit rule if there is no metric-id property set ("is there a metric-id property set? No? Then use the h-inventory metric ID for the h-metric metric ID").
For example, see here (this is a live example I copied from the "raw" inventory JSON that HawkFX gave me for a metric) - this is the h-inventory entity definition for the metric "Heap Used" on my WildFly Server resource - notice the "properties" map has a "metric-id" value that is DIFFERENT than the "id" - that "metric-id" is something I customized in my agent config in standalone.xml (well, I used the swarm agent, so I put it in the swarm config, but its basically the same thing):
{
"path": "/t;hawkular/f;mazz/m;MI~R~%5Bmazz%2FWildFly~~%5D~MT~WildFly%20Memory%20Metrics~Heap%20Used",
"properties": {
"__identityHash": "70e59a5d427632223da36c225ba6ef8572985",
"metric-id": "feed=mazz__msn=WildFly__typeName=Heap Used__resName=WildFly Server [WildFly]__resId=WildFly~~__typeId=WildFly Memory Metrics~Heap Used"
},
"name": "Heap Used",
"identityHash": "70e59a5d427632223da36c225ba6ef8572985",
"type": {
"path": "/t;hawkular/f;mazz/mt;WildFly%20Memory%20Metrics~Heap%20Used",
"name": "Heap Used",
"identityHash": "3be5b5fdabed925ac46fdc6d8295e34bbd3147a",
"unit": "BYTES",
"type": "GAUGE",
"collectionInterval": 30,
"id": "WildFly Memory Metrics~Heap Used"
},
"id": "MI~R~[mazz/WildFly~~]~MT~WildFly Memory Metrics~Heap Used"
}
Notice this:
feed=mazz__msn=WildFly__typeName=Heap Used__resName=WildFly Server [WildFly]__resId=WildFly~~__typeId=WildFly Memory Metrics~Heap Used
is different from this:
MI~R~[mazz/WildFly~~]~MT~WildFly Memory Metrics~Heap Used
And that's the issue. Clients have to know to look for the "metric-id" property and use it when looking up metric data in h-metrics (so if you want to graph the data, you have to ask h-metrics for the data associated with the value found in the "metric-id" property).
7 years, 9 months
Status of integration of inventory and vertx metrics
by Austin Kuo
Hi,
The current status is that I have created
- a feed
- a root resource with a resource type
- a eventbus resource as a child of the root resource above with a
resource type
- a eventbus.handlers metrics with gauge metric type. This has a
property 'metric-id' corresponding to the id of the metric data.
Fortunately, now I can view the metric data graph from the client (hawkfx).
Austin,
Thanks.
7 years, 9 months
integrating ansible and hawkular to start remote wildfly servers
by John Mazzitelli
I have a very simple PoC working that shows Ansible integrated with Hawkular such that it can start a WildFly server remotely.
I'll begin with a brief summary.
SUMMARY
A client connects to the Hawkular Server via its websocket and passes in a JSON request (like it does the other kinds of requests). The JSON request looks something like:
AnsibleRequest={"playbook":"start-wildfly.yml", "extraVars": { "wildfly_home_dir":"/opt/wildfly10" } }
The command-gateway server takes the request, runs the Ansible playbook, and returns the response back over the websocket to the client with an AnsibleResponse (which includes the Ansible output to show what it did).
DETAILS
The code is in two branches in my repos. See [1] and [2]. You need both for this to work.
Hawkular-Services packages up Ansible playbooks and its associated files in standalone/configuration/ansible. See [3].
Hawkular-Commons adds to command gateway a new JSON request/response for Ansible requests. See [4] and [5].
The Ansible command is invoked in the server when the AnsibleRequest JSON is received over the websocket. This command implementation runs the Ansible playbook and returns the results back in a AnsibleResponse. See [6].
If you build commons and then services to pull the new commons in, you can run a test to see it work. I use the test command CLI utility in the agent to do this [7]. First, build the new dist and run it so you have a Hawkular Service running (for example, "mvn clean install -Pdev,embeddedc"). Next, create a test JSON request file (say, "/tmp/hawkular.json") with this content:
{ "playbook":"start-wildfly.yml",
"extraVars": {
"wildfly_home_dir":"/directory/where/you/installed/wildfly"
}
}
Now build the hawkular-agent from source (just so you get the test command CLI utility) and run the test command CLI, telling it to use the JSON in your file and send it to your Hawkular Service server:
$ cd <hawkular-agent-source-dir>/hawkular-wildfly-agent-itest-parent/hawkular-wildfly-agent-command-cli/target
$ java -jar hawkular-wildfly-agent-command-cli-*.jar --username jdoe --password password \
--command AnsibleRequest --request-file=/tmp/hawkular.json
This sends the request to your local Hawkular Service server over the websocket, the Ansible playbook "start-wildfly.yml" is run, which starts the WildFly in that home dir you specified in your JSON request, and the response is sent back. The Command CLI tool will write the responses it receives to disk - look at the cli output for the names of the files it writes - you can look in there to see what the Ansible command response was (its just the full Ansible output).
There is still LOTS to do here (for one, you'll notice it assumes your wildfly install is on "localhost" :)). This is merely a PoC that we can use as a starting point - it shows that this can be done and how we can do it.
-- John Mazz
[1] https://github.com/jmazzitelli/hawkular-commons/tree/HAWKULAR-1096-ansibl...
[2] https://github.com/jmazzitelli/hawkular-services/tree/HAWKULAR-1096-ansib...
[3] https://github.com/jmazzitelli/hawkular-services/tree/HAWKULAR-1096-ansib...
[4] https://github.com/jmazzitelli/hawkular-commons/blob/HAWKULAR-1096-ansibl...
[5] https://github.com/jmazzitelli/hawkular-commons/blob/HAWKULAR-1096-ansibl...
[6] https://github.com/jmazzitelli/hawkular-commons/blob/HAWKULAR-1096-ansibl...
[7] https://github.com/hawkular/hawkular-agent/tree/master/hawkular-wildfly-a...
7 years, 9 months
Hawkular APM / Zipkin Adapter
by Gary Brown
Hi
Here is a short demo showing the initial work capturing information reported by Zipkin clients and translating it for use by the Hawkular APM server.
https://youtu.be/x6H2YJi2v1o
We will be adding more capabilities in the coming weeks. If you have any specific requirements please let us know (or just create a jira).
If anyone has a polyglot example instrumented with zipkin clients that we could test please get in contact.
Regards
Gary
7 years, 9 months
Hawkular Services 0.0.6.Final
by Juraci Paixão Kröhling
Team,
Hawkular Services 0.0.6.Final has just been released.
The only functional change between this and the previous version is that
Hawkular Services now reports itself as a "Hawkular" server, instead of
"Wildfly" (HAWKULAR-1098)
As the previous distributions, the Agent has to be configured with an
user. This can be accomplished by:
- Adding an user via bin/add-user.sh like:
./bin/add-user.sh \
-a \
-u <theusername> \
-p <thepassword> \
-g read-write,read-only
- Changing the Agent's credential on standalone.xml to the credentials
from the previous step or by passing hawkular.rest.user /
hawkular.rest.password as system properties (-Dhawkular.rest.user=jdoe)
You can find the release packages, sources and checksums at GitHub, in
addition to Maven:
https://github.com/hawkular/hawkular-services/releases/tag/0.0.6.Final
Shortcuts for the downloads:
Zip - https://git.io/vKwxm
tar.gz - https://git.io/vKwxI
- Juca.
7 years, 9 months
Integrate metric data with inventory
by Austin Kuo
Hi,
I’m currently working on vertx metric agent, my job is to enable the
inventory such that users can view the inventory and corresponding metrics.
My question is that how “real metric data” and “inventory metrics” are
matched?
Is it simply bound via id?
Thanks,
Austin
7 years, 9 months
Hawkular Services vs Hawkular Community
by Thomas Heute
(If you don't know about Hawkular Community you can skip this email
completely)
After some discussions about Hawkular Community, here is our conclusion:
- We will not have a separate Hawkular Community repository
- We will ship 2 distributions for Hawkular Services,
- one for dev/demo/quick test that will have an embedded
Cassandra, a default user and the embedded agent enabled (This is what you
get when building Hawkular Services today with "-Pembeddedc -Pdev").
- the other one that requires an external Cassandra server,
no default user and the agent being disabled.
- We'd like a simple UI to explore data (metrics, inventory, alerts),
but this is not a priority, if someone in the community is looking for an
Angular project it would be great, please contact us on this mailing list.
We would advertise that work in the "Client" section of the revamped
hawkular.org [1] as a separate download. (Note that this would likely be
purely static content and could even be made available on a CDN)
- For other side projects that are not mature to be included in Hawkular
Community or side projects, they will be listed either in the "client" or
"labs" section of the revamped website [1] depending on their maturity and
scope.
Thanks,
Thomas
[1] WIP: http://209.132.178.114:10188/
7 years, 9 months
