October 2016 - hawkular-dev - Jboss List Archives

by Artur Dryomov

Hi everyone, This year I will be working on the Hawkular Android client application as part of the Google Summer of Code 2015 program. The application itself will use Hawkular API and AeroGear SDK. In coming days I’ll research these areas, especially documentation, and will try to create some sort of architecture and basic design. Thank you all for this opportunity! Artur.

1 year, 4 months

7
6
0 / 0

Hawkular Metrics Openshift Containers

by Matt Wringe

I have a new subproject in Hawkular Metrics which sets up creating components for Openshift/Fabric8 (https://github.com/hawkular/hawkular-metrics/pull/200). There are 3 main parts Cassandra: creates a custom seed provider to support ReplicationControllers in Kubernetes, creates a folder/zip archive which can be used to generate a Docker image. It may make sense to move the Cassandra parts out to a separate project. Hawkular Metrics: creates a folder/zip archive which can be used to generate a Docker image Kubernetes: pulls everything together into a single kubernetes application. Can be used to deploy an application zip into fabric8 (via drag and drop in the web console or via the maven plugin) or deploy all the components into Openshift via the kubernetes.json configuration file. The docker images are not created and deployed to a docker registry as part of the build, it will just create a folder where you can run the docker build from. None of the maven docker plugins I looked at seemed to really work properly, so its still a manual process to do the build (and push to a registry). Its something which needs to be improved. The Cassandra service currently only supports adding new nodes to a cluster and not removing them via the ReplicationController. This is due to the replication factor being set to be 1 by default (which means when a node is removed, so is the data it contained). I believe the docker subproject of hawkular metrics is obsolete and can be removed (https://github.com/hawkular/hawkular-metrics/tree/master/docker), but someone please correct me if I am wrong. It's scripts are referring to the console which no longer exists as part of the project. - Matt

1 year, 4 months

4
9
0 / 0

Tenant Id - Not Part of URL

by Stefan Negrea

Hello Everybody, I've been working on a PR for the upcoming Hawkular Metrics release that will remove the tenant id from the end-point URLs. The tenant id will be moved to either a header parameter or a query parameter. The query parameter is in place for cases (such as curl) where setting a header is not possible, difficult, or inconvenient. Here is an example of the change: Existing URL: /{tenantId}/gauge/{metricId}/data New URL: /gauge/{metricId}/data Tenant id set via: 1) header - tenantId 2) query parameter - tenantId There are two exceptions to this rule, /tenants and /db/{tenantid}/series. The /tenants end-point will be changed into something different in the upcoming releases since it is mostly a management type API that does not belong in the same place with the regular metrics endpoint. And /db/{tenantid}/series end-point is needed in this exact format for compatibility with Influxdb compatible services. Now, to the merits of this change. The tenant id is volatile, can change any time, and changes to it should be expected; but the rest of the URL is fixed. The second issue is that the tenant id is a security concern. So we were limited in design choices since a security concern was leaking as part of the URL. So removing the tenant id from the URL will give us permanent & consistent addresses for resources (metrics and metric data points). And we will gain a lot of flexibility on the security side. In the future, users could authenticate with a user/pass combo and the backend would transform that into a tenant id to be used on the request. If the same user later decides to use a tenant id to pass along the request, the URL of the resources would not change. Another expectation is that tenant id is not sufficient, it is typically a combo of id + secret; so we would have resorted to a header or query param for the second piece of information (the secret). This change will give us the flexibility to adjust the security model (the meaning of tenant ids and ways to validate them) without compromising the URL structure. This will help Hawkular Metrics as it gets integrated into more and more projects and products. Here are the links to the JIRA and the PR for this change: https://github.com/hawkular/hawkular-metrics/pull/202 https://issues.jboss.org/browse/HWKMETRICS-68 Thank you, Stefan Negrea Software Engineer

1 year, 4 months

10
27
0 / 0

New and noteworthy in hawkular-parent 25

by Peter Palaga

Hi *, hawkular-parent 25 brings the following: * srcdeps-maven-plugin 0.0.5 * meets the promisses falsely done for 0.0.4: * less console output * built without tests * wildfly-maven-plugin 1.1.0.Alpha4 I have sent PRs to all components repos. Thanks, Peter _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev

1 year, 4 months

2
1
0 / 0

"Getting Things Done" (David Allen) System: Artificer

by Brett Meyer

FWIW, I've been experimenting with Artificer as a "Getting Things Done" solution, mainly as a bit of dogfooding. However, it fits in surprisingly well. I'll be maintaining a demo that showcases a GTD ontology/classifier system, showing Artificer as not-just-another-artifact-repo, but also a powerful information and reference system. https://developer.jboss.org/en/artificer/blog/2015/03/31/getting-things-d...

1 year, 5 months

4
3
0 / 0

Labeling needs ?

by Heiko W.Rupp

Hey, we have labels in Hawkular-metrics right now, but apparently there are use cases that are not yet covered (and I know Matt has more) - Listing keys of tags. Currently one has to know the available keys to be able to list the available values - tag values are currently a comma-separated string and not an array, which may have implications on allowed characters and escaping of separators - Post-tagging of data points (tagging could be provoked by another system that e.g. parses log files and sees an anomaly) The idea behind it is that one should be able to create tags at certain points in time for a single metric or a list of metrics to uniquely identify that point in time for queries (relative performance of two versions of a deployment). What else?

9 years, 5 months

6
9
0 / 0

hawkular metrics go client - need assistance for a couple issues

by John Mazzitelli

I found two bugs in the Hawkular Go Client and I need assistance: 1. https://github.com/hawkular/hawkular-client-go/issues/10 2. https://github.com/hawkular/hawkular-client-go/issues/12 The first one I have a PR written that appears to fix it (link to PR in the issue). I would need someone to peer review it and merge it if it is OK. Frankly, I don't see how it ever worked - but I suspect that Hawkular Metrics changed the REST API recently in this area and that's why it wasn't seen before. The second one I do not have a PR, but in the issue I mention a quick and dirty fix, but I don't think it is the proper way to fix it. I need someone to look at that issue and recommend a proper fix. Just need a way or a GoLang API that encodes a string for a URL *path* (not for a URL query string, which is happening here) due to the different ways spaces are encoded. I'd do it myself, but it's really late, and I have to get up early to hopefully meet up with a turkey that is "dying" to make it on my dinner table. Bu Wa Ha... BU WA HAHAHAHAHAHA! :) That is all. --John Mazz

9 years, 5 months

2
4
0 / 0

Hawkular Inventory Storage and Queries In a Nutshell

by Lukas Krejci

Hi all, given the recent discussion about the backend we should be using for inventory, I've quickly put together a brief explanation of what inventory actually stores and how it queries things. This is expected to raise more questions than to provide answers though and is meant as a starter for the discussion that I hope will happen in the comments section of the google doc [1]. Thanks, [1] https://docs.google.com/document/d/ 1Z1pIQS4EML7_0tuSyHUPX6jzNvaeEqGm4XAiYcRGbvQ/edit?usp=sharing -- Lukas Krejci

9 years, 5 months

2
1
0 / 0

Hawkular Metrics 0.21.0 - Release

by Stefan Negrea

Hello Everybody, I am happy to announce release 0.21.0 of Hawkular Metrics. This release is anchored by performance enhancements and general fixes. Here is a list of major changes: - Cassandra - *Cassandra 3.0.9 is now the supported version of Cassandra.* - Note: this is a rollback from previously supported version of 3.7 due to Cassandra community recommendations for stability and production deployment. Cassandra 3.7 or 3.9 are still compatible but development and testing use 3.0.9 release. - Compression - Fixed an issue that allowed duplicate instances of the compression job to get scheduled on server restart (HWKMETRICS-492 <https://issues.jboss.org/browse/HWKMETRICS-492>) - Improved the fault tolerance of the compression job (HWKMETRICS-494 <https://issues.jboss.org/browse/HWKMETRICS-494>) - Improved the performance of the merge process for reading compressed data (HWKMETRICS-488 <https://issues.jboss.org/browse/HWKMETRICS-488>) - Fixed wrong ordering when fetching compressed and uncompressed data (HWKMETRICS-506 <https://issues.jboss.org/browse/HWKMETRICS-506>) - Compression job provides back pressure (HWKMETRICS-500 <https://issues.jboss.org/browse/HWKMETRICS-500>) - The job scheduler now handles failure scenarios (HWKMETRICS-505 <https://issues.jboss.org/browse/HWKMETRICS-505>) - Cassandra Schema - Fixed an issue where the server can fail to start due to Cassalog being in inconsistent state (HWKMETRICS-495 <https://issues.jboss.org/browse/HWKMETRICS-495>) - gc_grace_second is set to zero for single node clusters ( HWKMETRICS-381 <https://issues.jboss.org/browse/HWKMETRICS-381>) - API Updates - Inserting data points has server side retries to increase the fault tolerance for simple error scenarios (HWKMETRICS-510 <https://issues.jboss.org/browse/HWKMETRICS-510>) - fromEarliest parameter is now supported in all query endpoints ( HWKMETRICS-445 <https://issues.jboss.org/browse/HWKMETRICS-445>) - Configuration - The configuration options did not have a consistent naming scheme. hawkular-metrics, hawkular.metrics, and hawkular prefixes were used along no prefixes. - In this release the naming schema has been standardized to hawkular.metrics.* for metrics specific configuration and hawkular.* for general configuration. - Here is list of all configuration options currently available: ConfigurationKey <https://github.com/hawkular/hawkular-metrics/blob/release/0.21.0/api/metr...> - For more details: HWKMETRICS-508 <https://issues.jboss.org/browse/HWKMETRICS-508> *Hawkular Alerting - included* - Version 1.3.0 <https://issues.jboss.org/projects/HWKALERTS/versions/12331985> - Project details and repository: Github <https://github.com/hawkular/hawkular-alerts> - Documentation: REST API Documentation <http://www.hawkular.org/docs/rest/rest-alerts.html>, Examples <https://github.com/hawkular/hawkular-alerts/tree/master/examples>, Developer Guide <http://www.hawkular.org/community/docs/developer-guide/alerts.html> *Hawkular Metrics Clients* - Python: https://github.com/hawkular/hawkular-client-python - Go: https://github.com/hawkular/hawkular-client-go - Ruby: https://github.com/hawkular/hawkular-client-ruby - Java: https://github.com/hawkular/hawkular-client-java *Release Links* Github Release: https://github.com/hawkular/hawkular-metrics/releases/tag/0.21.0 JBoss Nexus Maven artifacts: http://origin-repository.jboss.org/nexus/content/repositorie s/public/org/hawkular/metrics/ Jira release tracker: https://issues.jboss.org/projects/HWKMETRICS/versions/12331718 A big "Thank you" goes to John Sanda, Matt Wringe, Michael Burman, Joel Takvorian, Jay Shaughnessy, Lucas Ponce, and Heiko Rupp for their project contributions. Thank you, Stefan Negrea

9 years, 5 months

4
4
0 / 0

Memory-usage of Hawkular-services

by Heiko W.Rupp

Hey, tl;dr: we need to investigate heap usage - especially in the case of compression kicking in - it looks like there could be a memory leak. Compression timing seems mostly ok. I originally wanted to see how more feeds influence the metrics compression timing. So I started the server with -Xmx512m as I did in all the weeks before and pointed a few feeds at the server - to see it crash with OOME shortly after the compression started. Now I restarted the server with -Xmx1024m and also -XX:MaxMetaspaceSize=512m (up from 256m before) and am running the server now with 1 feed for a day. To be continued below ... ( Time is in GMT, which is 2h off of my local time) hawkular_1 | 15:00:44,764 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution hawkular_1 | 15:00:45,452 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 687 ms hawkular_1 | 17:00:44,757 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution hawkular_1 | 17:00:46,796 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 2039 ms hawkular_1 | 19:00:44,751 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution hawkular_1 | 19:00:47,293 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 2541 ms hawkular_1 | 21:00:44,749 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution hawkular_1 | 21:00:46,267 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 1517 ms hawkular_1 | 23:00:44,749 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution hawkular_1 | 23:00:45,472 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 722 ms hawkular_1 | 01:00:44,749 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution hawkular_1 | 01:00:46,241 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 1492 ms hawkular_1 | 03:00:44,747 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution hawkular_1 | 03:00:45,780 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 1033 ms hawkular_1 | 05:00:44,746 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution hawkular_1 | 05:00:45,781 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 1034 ms hawkular_1 | 07:00:44,749 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution hawkular_1 | 07:00:46,386 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 1636 ms hawkular_1 | 09:00:44,748 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution hawkular_1 | 09:00:45,682 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 934 ms hawkular_1 | 11:00:44,750 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution hawkular_1 | 11:00:46,339 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 1589 ms hawkular_1 | 13:00:44,748 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution hawkular_1 | 13:00:45,880 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 1132 ms Looking at the memory usage I see the memory usage there is often a peak in heap usage around and after the compression - looking at the accumulated GC-time also shows heavy GC activity. I guess the peak in heap usage (and thus committed heap) comes from promoting objects from young generation into old during the compression and then later after compression is over, they are garbage collected, so that heap usage goes down and the system is able to free some memory. Starting at around 4:10pm (2pm in the above output) I am running with 10 extra feeds (hawkfly on an external box) While it looks like there is a slowly growing non-heap, it was growing a lot when the 10 extra feeds connected. Also it looks like it is growing a bit more each time compression kicks in ("non-heap") The first compression run with 11 feeds did not take too long, the next was hawkular_1 | 17:00:44,749 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution hawkular_1 | 17:00:50,277 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 5528 ms Which used a lot more memory than before. Non-heap was able to reclaim some memory though. hawkular_1 | 19:00:44,751 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution hawkular_1 | 19:00:50,093 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 5341 ms hawkular_1 | 21:00:44,751 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution hawkular_1 | 21:00:49,465 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 4714 ms hawkular_1 | 23:00:44,753 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution hawkular_1 | 23:00:48,925 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 4171 ms hawkular_1 | 01:00:44,750 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution hawkular_1 | 01:00:48,554 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 3803 ms hawkular_1 | 03:00:44,761 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution hawkular_1 | 03:00:48,659 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 3898 ms hawkular_1 | 05:00:44,748 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution hawkular_1 | 05:00:49,134 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 4385 ms hawkular_1 | 07:00:44,755 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution hawkular_1 | 07:00:49,831 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 5076 ms hawkular_1 | 09:00:44,751 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution hawkular_1 | 09:00:49,508 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 4757 ms Now at 11:20 (9:20 for above logs) I started 5 more feeds (with a 2min sleep between starts) hawkular_1 | 11:00:44,749 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution hawkular_1 | 11:00:49,751 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 5002 ms hawkular_1 | 13:00:44,749 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution hawkular_1 | 13:00:56,594 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 11845 ms hawkular_1 | 15:00:44,754 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution hawkular_1 | 15:00:53,985 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 9231 ms And another 5 starting at 17:12 (15:00 in above logs timezone) hawkular_1 | 17:00:44,768 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution hawkular_1 | 17:00:57,824 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 13056 ms And another 5 starting at 19:57 (17:57 in above log timezone ) hawkular_1 | 19:00:44,751 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution hawkular_1 | 19:01:24,401 WARN org.apache.activemq.artemis.ra AMQ152007: Thread Thread[TearDown/ActiveMQActivation,5,] could not be finished hawkular_1 | 19:01:40,918 WARN org.apache.activemq.artemis.ra AMQ152007: Thread Thread[TearDown/ActiveMQActivation,5,default-threads] could not be finished hawkular_1 | 19:02:04,619 WARN org.apache.activemq.artemis.ra AMQ152007: Thread Thread[TearDown/ActiveMQActivation,5,default-threads] could not be finished hawkular_1 | 19:02:22,423 WARN org.apache.activemq.artemis.ra AMQ152007: Thread Thread[TearDown/ActiveMQActivation,5,default-threads] could not be finished hawkular_1 | 19:02:30,247 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 105495 ms This took almost 2 minutes, where the server was non-responsive 21:00:44,753 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution 21:01:06,520 INFO org.hawkular.metrics.core.jobs.CompressData Finished compressing data in 21767 ms 23:00:44,751 INFO org.hawkular.metrics.core.jobs.CompressData Starting execution And here it went south and the server more or less died with OOME exceptions (it is still responding to queries, and potentially even ingesting new data, but the scheduler seems not to run anymore. I can imagine that once the OOME happened, the scheduler thread died, freeing up memory that the compression job held and which then allowed the server itself to continue. But it certainly is in an unusable state. This is the final "one day" with "max" values plotted: ![](cid:7C83CDE5-A45F-42C6-853A-2956DC3E1719@redhat.com "Bildschirmfoto 2016-10-11 um 08.21.53.png") The peaks at compression time for heap-max (green line) are clearly visible.

9 years, 5 months

2
7
0 / 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

hawkular-dev October 2016