June 2016 - hawkular-dev - Jboss List Archives

BTM agent relationship with Hawkular agent

by Gary Brown

Hi On the team call just now, Heiko raised a question about whether the BTM agent could work with the Hawkular agent. So thought I would start this discussion thread to see what the potential options are. I currently see two issues: 1) The BTM agent must be configured on the jvm command line as a "-javaagent" to install ByteMan for instrumentation purposes. This is instantiated before the JBoss modules (and therefore subsystems etc) are initialised. 2) The hawkular agent won't necessarily be installed in all monitored servers, and instead remotely monitor some. The BTM agent would need to be installed in all servers where business transactions are executing. One type of integration that may be possible is in terms of delivering the captured business transaction information to the backend? i.e. the BTM agent locally reports it to the Hawkular agent as a relay? Thoughts? Regards Gary

3 days, 10 hours

10
9
0 / 0

Alpha 9 targets

by Heiko W.Rupp

Hey, I've added that at the bottom of [1]: we should concentrate on stabilizing the code base (fix bugs). Other tasks are: * More work on labels * Allow to use tokens all over the place * Move Hawkular onto WildFly 10 (if available ) * Finally address (and fix) availability handling Heiko [1] http://www.hawkular.org/blog/2015/12/16/hawkular-1.0.0.Alpha8-released.html

3 days, 22 hours

13
13
0 / 0

some new agent features

by John Mazzitelli

Just wanted to blast out a message about some new wildfly agent features. 1) Agent now supports secure communications between itself and any of its managed WildFly servers. I documented how to enable security in the management interface of WildFly here: https://github.com/hawkular/hawkular.github.io/blob/pages/src/main/jbake/... 2) Agent now supports monitoring JMX resources via Jolokia interface. The agent now can collect JMX metrics and availability data and can inventory JMX resources in the same manner as it can with DMR resources using the Jolokia interface (Jolokia is a very nice REST interface to JMX). Just as we have <metric-dmr>, <avail-dmr>, <resource-type-dmr>, <remote-dmr>, and related DMR settings in the agent subsystem, we now have <metric-jmx>, <avail-jmx>, <resource-type-jmx>, <remote-jmx> settings as well. So the agent can be given metadata that tells it what to do. As with remote WF servers, the agent also supports secure comm to the remote jolokia JMX server. There are a few things currently on the TODO for the agent: 1) utilize the new inventory "name" attribute - they now have "name" as a first-class setting - I need to see what the agent has to do to use it 2) utilize the inventory's "operation" metadata - I believe operations are now first class concepts in inventory - I need to figure out how to populate it 3) provide a mechanism for the agent to detect runtime changes to inventory. Right now, after the agent does its initial inventory scan, that's it. The inventory is fixed, all resource configuration is fixed, and all we do is collect metrics. The caveat is that if someone deploys a new app or adds a new JDBC driver, we force a new scan. We need a way to detect new resources in inventory (and old resources removed from inventory) and we need a way to detect changes to resource config. This should work for both DMR and JMX inventory items. 4) Need a way to consume wildfly notifications that other subsystems emit and figure out how to send that to a hawkualr events subsystem. 5) Peter is implementing/has implemented the JDBC driver CRUD and datasource CRUD, gonna see how easy it is to implement reload. I'm sure I'm forgetting something, but .. that's what I remember.

2 weeks

2
1
0 / 0

agent PR - deploy application tests

by John Mazzitelli

Agent PR needs to be peer-reviewed and merged: PR: https://github.com/hawkular/hawkular-agent/pull/145 JIRA: https://issues.jboss.org/browse/HWKAGENT-54 This adds integration tests that test deploying and undeploying an application.

2 weeks, 3 days

3
2
0 / 0

Corrections and suggestions on Explorer for android client

by Anuj Garg

Hello everyone, A tree structure of the explorer is approx ready for the android client and I was curious if i am wrong somewhere. First of all, All feeds are collected by call to /inventory/feeds feed call for resources /inventory/feeds/{feed}/resources call for rec resources and metrics /inventory/feeds/{feed}/resources/{resid}/recursiveChildren /inventory/{feed}/resources/{resid}/metrics Am I missing something? Thanks Anuj Garg

2 weeks, 6 days

3
2
0 / 0

Public Hawkular Instances - Feedback Please

by Matthew Mahoney

Hawkular Community, The JON QE team is querying feedback regarding the availability & usefulness of the Public Hawkular Instances: http://209.132.178.218:18080 (Try it out - " kick the tires ") http://209.132.178.218:18090 (On-demand build for Development / Integration usage) * Are either of these instances being used, and if so for what purpose? * What can we do to make these instances more usable / useful? Thanks! Matt

3 months, 1 week

7
7
0 / 0

Hawkular Metrics 0.13.0 - Release

by Stefan Negrea

Hello Everybody, I am happy to announce release 0.13.0 of Hawkular Metrics. This release is anchored by Hawkular integration enhancements, under-the-cover refactorings and fixes, and a new bulk data generation tool. Here is a list of major changes in this release: 1) Data Generation Tool * A new tool that generates bulk metrics data to be used in performance and load testing. * It generates directly Cassandra data files (SSTables), which leads to a very fast generation process for large amounts of metrics data. * For more details: https://github.com/hawkular/hawkular-metrics/tree/master/data-generator (HWKMETRICS-355) 2) Receive Metrics via Hawkular Bus * When deployed within Hawkular distribution, the project now accepts metrics via the Hawkular Bus; until now only the REST API had support for Metrics insertion. * Hawkular Metrics currently support publishing of newly inserted metrics to the bus and receiving metrics via the bus. * For more details: HWKMETRICS-347, HWKMETRICS-352 3) Sorted Stacked Metrics Results * When requesting stacked metrics aggregation the results are now ordered (HWKMETRICS-353) 4) External Integrations * Heapster sink now divides storing to multiple calls. This is an improvement over the initial implementation that had one REST API call per metric (HWKMETRICS-290) * ptrans now works with a Hawkular Metrics instance protected by Hawkular Accounts via Hawkular distribution (HWKMETRICS-342) * Grafana integration via Influxdb compatible end-point allows connections to a Hawkular Metrics instance protected by Hawkular Accounts via Hawkular distribution (HWKMETRICS-343) Github Release: https://github.com/hawkular/hawkular-metrics/releases/tag/0.13.0 JBoss Nexus Maven artifacts: http://origin-repository.jboss.org/nexus/content/repositories/public/org/... Jira release tracker: https://issues.jboss.org/browse/HWKMETRICS/fixforversion/12329530 Hawkular Metrics Clients * Ruby: https://github.com/hawkular/hawkular-client-ruby * Python: https://github.com/hawkular/hawkular-client-python * Go: https://github.com/hawkular/hawkular-client-go A big "Thank you" goes to John Sanda, Thomas Segismont, Mike Thompson, Matt Wringe, Michael Burman, and Heiko Rupp for their project contributions. Thank you, Stefan Negrea Software Engineer

5 months

3
2
0 / 0

hawkular wildfly agent - feed ID

by John Mazzitelli

There was some concern that the feed ID autogenerated by the agent might not be what a person always wants (e.g. feed ID for WF10 agents will be the unique UUID of the wildfly server). So, if you want to define your own feed ID, you can configure it now. In the agent's standalone.xml <storage-adapter> you can specify a feedId attribute if you want. By default, it isn't specified, so the agent will autogenerate a unique feed ID for itself (this should really be the normal mode of operation, but some people like to complain, so I made it easy to shut them up :-) This is in master. Will be in the next release.

8 months, 2 weeks

10
13
0 / 0

Maven swagger plugin

by Gary Brown

Hi The current version of the maven-swagger-plugin used by hawkular doesn't include derived types in the generated docs. I've tried updating to 2.3.4 and it still does not work. From some basic tests with 3.0.0, it seems to produce the correct content in the json schema - but has slightly different attributes to the plugin configuration, which when updated does not seem to run the template correctly. The other issue is some API changes when updating the com.wordnik:swagger-* dependencies in line with this maven plugin, as there have been some GAV and API changes. Its not an urgent issue, but it looks like we will be impacted by changes when we need to upgrade, so thought I would raise the issue. Regards Gary

1 year

3
2
0 / 0

[GSoC] Hawkular Android Client

by Artur Dryomov

Hi everyone, This year I will be working on the Hawkular Android client application as part of the Google Summer of Code 2015 program. The application itself will use Hawkular API and AeroGear SDK. In coming days I’ll research these areas, especially documentation, and will try to create some sort of architecture and basic design. Thank you all for this opportunity! Artur.

1 year, 4 months

7
6
0 / 0

Hawkular Metrics Openshift Containers

by Matt Wringe

I have a new subproject in Hawkular Metrics which sets up creating components for Openshift/Fabric8 (https://github.com/hawkular/hawkular-metrics/pull/200). There are 3 main parts Cassandra: creates a custom seed provider to support ReplicationControllers in Kubernetes, creates a folder/zip archive which can be used to generate a Docker image. It may make sense to move the Cassandra parts out to a separate project. Hawkular Metrics: creates a folder/zip archive which can be used to generate a Docker image Kubernetes: pulls everything together into a single kubernetes application. Can be used to deploy an application zip into fabric8 (via drag and drop in the web console or via the maven plugin) or deploy all the components into Openshift via the kubernetes.json configuration file. The docker images are not created and deployed to a docker registry as part of the build, it will just create a folder where you can run the docker build from. None of the maven docker plugins I looked at seemed to really work properly, so its still a manual process to do the build (and push to a registry). Its something which needs to be improved. The Cassandra service currently only supports adding new nodes to a cluster and not removing them via the ReplicationController. This is due to the replication factor being set to be 1 by default (which means when a node is removed, so is the data it contained). I believe the docker subproject of hawkular metrics is obsolete and can be removed (https://github.com/hawkular/hawkular-metrics/tree/master/docker), but someone please correct me if I am wrong. It's scripts are referring to the console which no longer exists as part of the project. - Matt

1 year, 4 months

4
9
0 / 0

Tenant Id - Not Part of URL

by Stefan Negrea

Hello Everybody, I've been working on a PR for the upcoming Hawkular Metrics release that will remove the tenant id from the end-point URLs. The tenant id will be moved to either a header parameter or a query parameter. The query parameter is in place for cases (such as curl) where setting a header is not possible, difficult, or inconvenient. Here is an example of the change: Existing URL: /{tenantId}/gauge/{metricId}/data New URL: /gauge/{metricId}/data Tenant id set via: 1) header - tenantId 2) query parameter - tenantId There are two exceptions to this rule, /tenants and /db/{tenantid}/series. The /tenants end-point will be changed into something different in the upcoming releases since it is mostly a management type API that does not belong in the same place with the regular metrics endpoint. And /db/{tenantid}/series end-point is needed in this exact format for compatibility with Influxdb compatible services. Now, to the merits of this change. The tenant id is volatile, can change any time, and changes to it should be expected; but the rest of the URL is fixed. The second issue is that the tenant id is a security concern. So we were limited in design choices since a security concern was leaking as part of the URL. So removing the tenant id from the URL will give us permanent & consistent addresses for resources (metrics and metric data points). And we will gain a lot of flexibility on the security side. In the future, users could authenticate with a user/pass combo and the backend would transform that into a tenant id to be used on the request. If the same user later decides to use a tenant id to pass along the request, the URL of the resources would not change. Another expectation is that tenant id is not sufficient, it is typically a combo of id + secret; so we would have resorted to a header or query param for the second piece of information (the secret). This change will give us the flexibility to adjust the security model (the meaning of tenant ids and ways to validate them) without compromising the URL structure. This will help Hawkular Metrics as it gets integrated into more and more projects and products. Here are the links to the JIRA and the PR for this change: https://github.com/hawkular/hawkular-metrics/pull/202 https://issues.jboss.org/browse/HWKMETRICS-68 Thank you, Stefan Negrea Software Engineer

1 year, 4 months

10
27
0 / 0

New and noteworthy in hawkular-parent 25

by Peter Palaga

Hi *, hawkular-parent 25 brings the following: * srcdeps-maven-plugin 0.0.5 * meets the promisses falsely done for 0.0.4: * less console output * built without tests * wildfly-maven-plugin 1.1.0.Alpha4 I have sent PRs to all components repos. Thanks, Peter _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev

1 year, 4 months

2
1
0 / 0

"Getting Things Done" (David Allen) System: Artificer

by Brett Meyer

FWIW, I've been experimenting with Artificer as a "Getting Things Done" solution, mainly as a bit of dogfooding. However, it fits in surprisingly well. I'll be maintaining a demo that showcases a GTD ontology/classifier system, showing Artificer as not-just-another-artifact-repo, but also a powerful information and reference system. https://developer.jboss.org/en/artificer/blog/2015/03/31/getting-things-d...

1 year, 4 months

4
3
0 / 0

Who owns OpenShift 'hawkular' namespace?

by Juraci Paixão Kröhling

Team, I'm looking for the owner of the OpenShift namespace 'hawkular'. To check if you own it, open the following URL and check the "Domains" part. In my case, I see only "jpkroehling". https://openshift.redhat.com/app/console/settings Thanks! - Juca.

9 years, 7 months

2
2
0 / 0

Hawkular.org

by Thomas Heute

With the recent repackaging effort + the ManageIQ UI, we need to rethink hawkular.org. It's tricky to keep it simple, what to download, for what usecase... And at the moment it's getting outdated. IMO, it should be trivial for end users to figure out: - What to download - where to download and how to install the server - find/browse/download the various options to feed the server with data (WF agent, ptrans supported protocoles, ...) - find/browse/download options to use/read the data: (ManageIQ, Grafana...) and for the developers - find/browse/download the client libraries (Ruby, Python...) - Find client libraries documentation - Find examples... We should probably not separate/fork Hawkular Services from Hawkular "community" too much, beside the installation procedure and few things that we would mention as only available in the "larger" package. I use the term "Hawkular Server" in the mind map for the 2 "flavors". Please propose improvements to the draft attached that represents the website structure. (A square is not necessarily a page, it may be a section of a page) Thomas

9 years, 8 months

9
59
0 / 0

Master's thesis: Alert Prediction in Metric Data Based on Time Series Analysis

by Pavol Loffay

Hello, yesterday I successfully defended my thesis [1]. I would like to thank to everyone from Hawkular team who helped me with the problems I was facing. Last but not least I would like to thank to company Red Hat for the opportunity to work on this project. [1]: https://is.muni.cz/auth/th/374115?lang=en Pavol

9 years, 8 months

9
9
0 / 0

Sync with Inventory

by Austin Kuo

Hi all, I’m currently investigating how to sync with inventory server. Here’s the example scenario: Consider the following problem. A user creates version 1 of the app with two http servers, one listening on port 8080, the other on port 8181. In version 2, the http server listening on port 8181 is no longer needed. When the old version is stopped and the new version started, there will be just one http server listening. The application is not aware of the previous state. What should we do so that the second http server is removed from Inventory? Thanks in advance.

9 years, 9 months

3
11
0 / 0

Metrics and use of metrics_idx

by Michael Burman

Hi, This sparked my interest after the discussions in PR #523 (adding cache to avoid metrics_idx writes). Stefan commented that he still wants to write to this table to keep metrics available instantly, jsanda wants to write them asynchronously. Maybe we should instead just stop writing there? Why? We do the same thing in tenants also at this time, we don't write there if someone writes a metric to a new tenant. We fetch the partition keys from metrics_idx table. Now, the same ideology could be applied to the metrics_idx writing, read the partition keys from data. There's a small performance penalty, but the main thing is that we don't really need that information often - in most use cases never. If we want to search something with for example tags, we search it with tags - that metricId has been manually added to the metrics_idx table. No need to know if there's metrics which were not initialized. This should be the preferred way of doing things in any case - use tags instead of pushing metadata to the metricId. If we need to find out if id exists, fetching that from the PK (PartitionKey) index is fast. The only place where we could slow down is if there's lots of tenants with lots of metricIds each and we want to fetch all the metricIds of a single tenant. In that case the fetching of definitions could slow down. How often do users fetch all the tenant metricIds without any filtering? And how performance critical is this sort of behavior? And what use case does list of ids serve (without any information associated to them) ? If you need to fetch datapoints from a known metricId, there's no need for metrics_idx table writing or reading. So this index writing only applies to listing metrics. - Micke

9 years, 9 months

4
11
0 / 0

inventory/deprecated not so easy to use

by John Mazzitelli

From: http://www.hawkular.org/docs/rest/rest-inventory.html#_api_endpoints "The old API has been kept so that clients can gradually move to the new API, but the old API has moved from /hawkular/inventory to /hawkular/inventory/deprecated. The clients using the old API will have to be modified to use the new endpoint but are otherwise unaffected." This isn't exactly true. Not ALL the API from <0.17.0.Final is there at /deprecated - if I'm using some of the old API I have to modify the code, but not all. Jay tried using the old agent with the new inventory, and he switched the inventory context from "hawkular/inventory" to "hawkular/inventory/deprecated" (there is a inventory-context attribute on <storage-adapter> in the agent config for just such an emergency - just a change to the standalone.xml setting). However, it failed because the "/bulk" API isn't available under /deprecated. So if anyone has a client that wants to talk to the old inventory, they need to know WHICH API was deprecated (feeds/ deprecated, bulk/ was not) and ONLY change those calls to use the /deprecated URL (e.g. change the code to use /entity/feed, but leaving /bulk alone). Which kinda sucks because now you have to pinpoint what calls need to change in the code and change code - when it would be much easier to simply be able to change a configuration setting and not have to change or recompile/build any code - you can just use the old client as is with the config change. In other words, if I have an inventory client wants to use the old inventory API, it should switch to "/deprecated" URL and expect it all to work as before.

9 years, 9 months

2
1
0 / 0

Hawkular-services 0.0.4 released

by Heiko W.Rupp

Hi, hawkular-services 0.0.4 has been released Notable changes are: - bump agent to 0.19.2 - retry talking to the server until connection succeeds - 'Hostname' property of server has been brought back - Hawkular-metrics updated to master from yesterday evening - HWKMETRICS-420 Add ADMIN AvailabilityType - HWKMETRICS-393 endpoints for fetching raw data for multiple metrics You can find the release .zip at: https://github.com/hawkular/hawkular-services/releases/tag/0.0.4.Final Thanks for Stefan helping me with the bits. I have made docker images for this and a wildfly with included hawkular-agent 0.19.2 available. See http://pilhuhn.blogspot.de/2016/06/using-hawkular-services-via-docker.html

9 years, 9 months

1
0
0 / 0

PR review request: https://github.com/hawkular/hawkular-apm/pull/463

by Gary Brown

Hi In the client (instrumented) environment, Hawkular APM uses the ServiceLoader to obtain implementations for various components. This enhancement introduces a 'ServiceStatus' check, to determine whether an implementation is suitable to be returned (i.e. has all the config it needs). This will enable the client to be packaged with multiple implementations of a particular interface, and enable selection of the appropriate impl at runtime based on the configuration (e.g. environment variables). Regards Gary

9 years, 9 months

1
0
0 / 0

Update on Hawkular android client

by Anuj Garg

Hello Guys, I am back to tell what is new going on in Android client of Hawkular - Old metrics related classes modified to adjust metric from explorer - Metric data is now fetched in terms of buckets - Metric chart fragment now contain list to choose time limits - Some other beutififation related to axis lables Last but not the least Aerogear Pipe for android is modified to handle special symbols in path param of REST call. As always there is a visual video to enjoy the progress https://drive.google.com/file/d/0B8tYKC1-UERHWVluZDlkSUN5eG8/view?usp=sha... What is next :- - Most changes are added to only Gauge type at present and will be extended to others after acceptace Thanks Anuj Garg

9 years, 9 months

2
2
0 / 0

Hawkular APM (formerly BTM) version 0.9.0.Final released

by Gary Brown

Hi I'm pleased to announce that Hawkular APM 0.9.0.Final has been released. The main purpose of this release is the rename of the project from BTM (Business Transaction Management) to APM (Application Performance Management), to reflect the wider scope of the project's capabilities. This is the first release under the new project name, and consequently there are a number of naming related changes (e.g. REST endpoints, Java package/class names, model, etc) that mean this release is not backward compatible with the previous release. The details of the release can be found here: https://github.com/hawkular/hawkular-apm/releases/tag/0.9.0.Final As part of this release, we have also moved the documentation for the project to gitbook: https://hawkular.gitbooks.io/hawkular-apm-user-guide/content/ Regards Gary

9 years, 9 months

1
0
0 / 0

Avail-percentage query in metrics?

by Heiko W.Rupp

Hey, is there a way to query Hawkular-metrics like this: "give me the distribution of availability (up, down, ..) over a given time range" The result could be aggregated time per value or relative percentages - this does not matter. Heiko

9 years, 9 months

2
3
0 / 0

hawkular metrics schema upgrade bug

by John Sanda

New schema change management code was introduced in hawkular-metrics 0.15.0. There were some bugs that break upgrades. Upgrading with 0.15.0 and 0.16.0 will not work. A few people have run into this. I have already submitted a PR with the fixes to be included in the next release. The ticket for this is https://issues.jboss.org/browse/HWKMETRICS-425 <https://issues.jboss.org/browse/HWKMETRICS-425>. - John

9 years, 9 months

2
1
0 / 0

Want to try hawkular-services via Docker?

by Heiko W.Rupp

If you want to give it a try you can save the following into a file docker-compose.yml and then run docker-compose up hawkular This image has the -Pdev user jdoe/password. hawkular: image: "pilhuhn/hawkular-services:latest" ports: - "8080:8080" - "8443:8443" - "9990:9990" volumes: - /tmp/opt/hawkular/data:/opt/data links: - myCassandra environment: - HAWKULAR_BACKEND=remote - CASSANDRA_NODES=myCassandra myCassandra: image: cassandra:3.7 environment: - CASSANDRA_START_RPC=true volumes: - /tmp/opt/hawkular/cassandra:/var/lib/cassandra -- Reg. Adresse: Red Hat GmbH, Technopark II, Haus C, Werner-von-Siemens-Ring 14, D-85630 Grasbrunn Handelsregister: Amtsgericht München HRB 153243 Geschäftsführer: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander

9 years, 9 months

2
2
0 / 0

Can’t run inventory hawkular-service

by Austin Kuo

Hi all, I started a cassandra cluster with: > ccm create test -v 2.2.6 -n 1 -s && ccm start test and then I tried to start the service with: > ./dist/target/hawkular-services-dist-0.0.3.Final-SNAPSHOT/bin/standalone.sh And it showed the warning and I checked the sever status at localhost:8080. It said that inventory is unavailable. WARN [org.hawkular.inventory.cdi] (ServerService Thread Pool -- 69) HAWKINV003501: Inventory backend failed to initialize in an attempt 14 of 15 with message: Could not instantiate implementation: com.thinkaurelius.titan.diskstorage.cassandra.thrift.CassandraThriftStoreManager. Anyone who has encountered this? Thanks!

9 years, 9 months

4
4
0 / 0

PR to review: https://github.com/hawkular/hawkular-apm/pull/461

by Gary Brown

Hi Hopefully should be quick to review, just about making previously hard coded defaults for retry mechanism configurable via system properties/environment variables. Any takers? Regards Gary

9 years, 9 months

1
0
0 / 0

Request for PR review - build-tools - https://github.com/hawkular/hawkular-build-tools/pull/25

by Lukas Krejci

Hi all, I've made some adjustments to our swagger -> asciidoc script so that it is able to emit information about inheritance of the data definitions. I'd like to use this for the next Inventory release, because the REST docs are border-line useless without it. I made biiig updates to the REST docs for the new inventory REST API and would like to have it in shape before the imminent release of it. Therefore I humbly ask for review of the above mentioned PR so that we can release new build-tools and parent that I could then consume in inventory for the new release. Thanks, -- Lukas Krejci

9 years, 9 months

2
1
0 / 0

Has to provide tenant header to look up tenant

by Austin Kuo

I found something weird when I was playing with the rest api. When I GET ‘inventory/tenant’, this request has to provide tenant header. It does not make sense to me because I look up my tenants because I don’t know them. Austin.

9 years, 9 months

3
2
0 / 0

Metrics performance testing PR#520

by Thomas Segismont

Hi, Today I've been looking at Metrics insertion performance. My setup is the following: - on my laptop I run the Gatling load scenario (which is very similar to the perf test job scenario) - I also run a Metrics standalone instance - on another machine connected to my LAN, I run a single node C* cluster In order to avoid problems due to memory constraints, I set min and max heap size to 2048m. Below this value the server spends significant time in garbage collection (with high number of concurrent clients). I ran Gatling 3 times with different number of clients (think agents): 100,1000,2000 ---- mvn gatling:execute -Dclients=X -Dramp=0 -Dloops=50 ---- From 100 to 1000 virtual clients, the throughput raised accordingly (x10). From 1000 to 2000, I hit the same kind of plateau Filip was observing: I got barely 15% throughput increase. None of the machines had reached cpu/io/memory limits. So I ran the 2000 virtual clients test again and jstack (result attached). As you can see, most of the task handler threads are in state WAITING inside the com.datastax.driver.core.HostConnectionPool.awaitAvailableConnection method. Then, following the "Monitoring and tuning the pool" section of the driver doc, I added some code to print the number of open connections, active requests, and maximum capacity. It confirmed that the maximum capacity was reached. Note that by default with the V3 protocol, the driver creates just one connection per host and allows a maximum of 1024 concurrent requests. After that I added new options in the code to let the user define the max number of connections per host as well as requests per connection. In my tests, I set the maximum connection limit to 10 and maximum concurrency to 5000. These are arbitrary values, they are simply bigger than the default. Best values would depend on the environment and should be determined with testing. Then I ran Gatling 3 times again (100, 1000, 2000 virtual clients). This time throughput increased linearly with the number of clients, from 100 to 1000 and then 2000. I also tried with 3000, but this time throughput did not increase as much. This was certainly due to both machines now using more than 80% cpu. I sent a pull request [2] with the changes needed for the new configuration parameters. Hopefully it can help in Filip's tests as well. Regards, -- Thomas Segismont JBoss ON Engineering Team [1] http://datastax.github.io/java-driver/manual/pooling/#monitoring-and-tuni... [2] https://github.com/hawkular/hawkular-metrics/pull/520

9 years, 9 months

3
2
0 / 0

JMS/MDB related PR to be reviewed

by Gary Brown

https://github.com/hawkular/hawkular-apm/pull/456 Hawkular APM uses JMS as its event processing backbone, with events being passed around in batches. When some or all of a batch of events fails to be processed by a particular component, it has a retry mechanism that resubmits the failed events to be processed again (subject to a max retry count). The problem with the current approach is that it is resubmitting the events to the source topic upon which the events were received - however this topic may also be used by other processing components that had processed the events successfully. For example, processing components A and B subscribe to a particular topic T - A processes all of the events in batch B1 successfully, and B only processes half, retrying the other half. When the failed half are published back to topic T again, then A receives them a second time, even though they were processed successfully the first time. Therefore the retry mechanism needs to be more targetted, so that the resubmitted events are only processed by the component that failed to process the events before. There are two possible ways to do this: 1) Define a retry queue - this means doubling up on all of the subscribers, so that each subscriber (i.e. processing capability) would have a topic subscriber to receive the initial events, and a retry queue to handle failed attempts. Its possible a single retry queue could be used with a message selector to distinguish the target processing unit. 2) Use a message selector on the original topic - so in this case, no more destinations need to be added, the initial message has no property defined so will be received by all topic subscribers, whereas when the message is resubmitted, it will have a target subscriber named, which will be routed through to the correct subscriber based on a message selector. This PR implements this second approach. On initial tests the performance hit of using a message selector appears negligible, although this was my initial concern. If this does prove to be an issue in the future, then the single queue with message selector approach in (1) could be used. Anyone able to review/merge the PR for me? Thanks in advance. Regards Gary

9 years, 9 months

1
0
0 / 0

hawkular services with embedded c profile - need PR reviewed/merged

by John Mazzitelli

I just submitted a new PR for hawkular services: https://github.com/hawkular/hawkular-services/pull/22 This re-introduces the ability to build Hawkular Services with embedded Cassandra if you want. This will ONLY embed Cassandra IF you build with the special Maven profile -Pembeddedc. If you never build with that profile, Hawkular Services will build as it does now (that is, requires you to install and start your own Cassandra separately). So this will allow developers to continue doing what they've been doing (that is, no need to run a standalone C*) if they want. Again, the default build is NOT to embed Cassandra. You must explicitly tell Maven you want to embed C* via -Pembeddedc to get this feature.

9 years, 9 months

5
17
0 / 0

Hawkular Community - Distribution

by Stefan Negrea

Hello Everybody, I want to give an update on the Hawkular Community distribution since Hawkular Services is now on a good track. The Hawkular Community distribution will be built on top of the Hawkular Services with two additions: Embedded Cassandra and Hawkular UI. The Embedded Cassandra was removed from the Hawkular Services but will make its way into the community distribution. The Hawkular UI will be the original web interface from previous Hawkular bundle but updated to work with Hawkular Services. In the short term, the Hawkular Community distributions will be just a rebundling of Hawkular Services with the Embedded Cassandra. The repository for this effort is: https://github.com/hawkular/hawkular-community The Hawkular UI updates are now in the planning stages and I will provide an update as soon as the timeline becomes more clear. The code has been relocated this new repository: https://github.com/hawkular/hawkular-ui Lastly, I renamed old hawkular/hawkular repository hawkular/hawkular-dist-old, marked it as archived, and removed access to it. The new url: https://github.com/hawkular/hawkular-dist-old . My initial reaction was to update hawkular/hawkular for the new community distribution. But after looking at the commit and pull request history it did not make sense to continue using that repository. Also, it would have been really confusing to keep the repository with the old name knowing it will probably be never be updated again, if somebody takes a look at the org and spots that repository it may wonder if the community is still active. A note about the hawkular/hawkular rename, Github has automatic redirects for renamed repositories. If somebody navigates to the old url, they will be redirected to the new name. So as long we do not have another hawkular/hawkular, the old url will keep working (which is is a good idea given existing links and forks). But we should never ever name anything again hawkular/hawkular because it is a very misleading name. Thank you, Stefan Negrea

9 years, 9 months

2
1
0 / 0

PR to be reviewed

by Gary Brown

https://github.com/hawkular/hawkular-apm/pull/454 https://issues.jboss.org/browse/HWKAPM-484 Consolidation of how processing of bulk and individual events are handled. Regards Gary

9 years, 9 months

1
0
0 / 0

Hawkular Services 0.0.3.Final

by Juraci Paixão Kröhling

Team, Hawkular Services 0.0.3.Final has just been released. This version includes Agent 0.19.0.Final and features the removal of the embedded Cassandra. You'll need a local installation of Cassandra running before you start the server. The distribution can be downloaded here: https://repository.jboss.org/nexus/service/local/repositories/releases/co... As the previous distributions, the Agent has to be configured with an user. This can be accomplished by: - Adding an user via bin/add-user.sh like: ./bin/add-user.sh \ -a \ -u <theusername> \ -p <thepassword> \ -g read-write,read-only - Changing the Agent's credential on standalone.xml to the credentials from the previous step or by passing hawkular.rest.user / hawkular.rest.password as system properties (-Dhawkular.rest.user=jdoe) Known issue: https://issues.jboss.org/browse/HWKINVENT-186 - Juca.

9 years, 9 months

2
1
0 / 0

Performance testing (and PR #519)

by Michael Burman

Hi, So the discussion on #hawkular pointed out that our current metrics performance testing is seeing a regression in the performance, after the merging of PR #519. I found this odd as I had done a lot of testing for this feature, including multiple runs of JMH (OpenJDK's Java benchmarking tool) and using a profiler to witness reduced CPU usage as well as reduced GC activity. The issue surrounds over RxJava's concatWith vs. mergeWith. Now, concatWith creates internal Queues and other stuff to keep track of the stuff, whereas mergeWith does not. So, Filip's tools say mergeWith is slower, my intuition/profiler and benchmarking says otherwise. Odd situation, but there's a small difference in testing methologies here. I've done testing against hawkular-core-service (that is, only the internal handling), while Filip uses the whole Wildfly in his testing. Although the performance degredation is quite severe with JAX-RS involved, it shouldn't cause different end results. The test results for my runs with JMH are here (with first the end results and then individual runs and statistics). https://gist.github.com/burmanm/e3b1ab0e67bcae189072718d39b66b6c The test that is run here is: https://github.com/hawkular/hawkular-metrics/blob/master/integration-test... Which mimics the behavior in which multiple concurrent senders are sending 1 metric in each call. With our backend, there's not a huge difference if we send everything in one request or single requests (this is more JAX-RS issue). The tests are using both 1 datapoint per 1 metric and 10 datapoints per 1 metric. The performance is returned as inserted metrics per second (so multiply by 10 the second result -> 9012 metrics / second -> 90120 datapoints / second). Let the flaming begin.. or something ;) Other notes: There are other important optimization strategies involved with the 1 metric 1 datapoint scenario. First one is changing the behavior of indexUpdates. Removing unnecessary index updates doubles the performance from 13k metrics to 26k metrics per second. Second optimization strategy with single Cassandra node deployments is to force datapoint writing to use batch statements (with different partitions - which isn't normally recommended), that increases the performance from the 26k to >60k. I'm trying to find better solution for the latter case, but haven't so far found one. - Micke

9 years, 9 months

2
1
0 / 0

PR to review

by Gary Brown

Hi https://github.com/hawkular/hawkular-apm/pull/449 Any takers? Regards Gary

9 years, 9 months

1
0
0 / 0

Hawkular Commons 0.7.3.Final released

by Juraci Paixão Kröhling

Team, Hawkular Commons 0.7.3.Final was just released. This release includes the JAX-RS filter for requiring Hawkular-Tenant, as well as the annotation `@TenantRequired`, where you can mark a specific class or method as required or not required. PRs to Alerts and Inventory will follow shortly. I also expect to be able to send a PR to Metrics soon. - Juca.

9 years, 9 months

3
4
0 / 0

Prepare for Hawkular Services 0.0.3.Final

by Juraci Paixão Kröhling

Team, We'll have a release of Hawkular Services tomorrow, for version 0.0.3.Final. This release will include the removal of embedded Cassandra. There should not be anything required for individual components, but it would be good if you'd test the current Hawkular Services master to see if your component is working as expected. If there are version changes required, send a PR by EOD today. - Juca.

9 years, 9 months

1
0
0 / 0

services build has a problem

by John Mazzitelli

This recently changed and doesn't work like it did before. I can't figure out what's wrong. The -Pdev build isn't working correctly (and neither is the non-dev build). When you build without -Pdev, the agent should be disabled by default. But it is not, and you will get agent errors on startup. When you build with -Pdev, the agent should be enabled and with the hawkualr.rest.user/password properties set inside standalone.xml. But they are not, and you again get agent startup errors. This used to work a few days ago. The .xsl that used to tweak the standalone.xml doesn't seem to be doing what it was supposed to do. Look here: https://github.com/hawkular/hawkular-services/blob/master/feature-pack/sr... See where it sets the system properties to control the agent enabled flag and the credentials? This isn't doing anyting that I can see because the distro doesn't have any system proeprties in standalone.xml, but this used to work. We'd get those <system-properties> elements in standalone.xml. I can't figure out what changed. Hoping someone knows the deal and can put this back.

9 years, 9 months

2
3
0 / 0

Some progress on Dockerization of Hawkular-services

by Heiko W.Rupp

Hey, I was able to make some progress with Dockerizing Hawkular-services More technical details below As it is relatively hard (not impossible) to do stuff like adding a user inside a (immutable) docker container or to edit standalone.xml to enable the agent, we need to decide how we add the (default) user to hawkular docker container and also if/how to enable the embedded agent. Also as we more understand hawkular-services as one package, we should perhaps figure out if we really need the flexibility to modify standalone.xml on the fly with all our snippets and templates or if we create a 'hardcoded' one that can be used for the process going forward. At the moment wildfly10 is our base and will be for a longer period, so the server-supplied standalone.xml will not change anyway. Anyway my Poc is at https://github.com/pilhuhn/hawkular-services/tree/dockerize-all-the-things There are 3 routes that I went (basically refining the process) (If you want to run any you first need to $ mkdir -p /tmp/opt/hawkular as this is where the docker run commands expect the data for now To get a 'working server' you need to "mvn install -Pdev first" 1) dist/ creating a Dockerfile by hand and manually running docker build Dockerfile and then docker-compose to run a container from this image linked to a C* 3.5 one. Base image is the jboss/base-jdk:8 one and the Dockerfile basically overlays the whole hawkular-services distribution on top You can run this with $ docker-compose up 2) docker-dist/ here maven creates the Dockerfile on the fly and feeds it to Docker. This uses the fabric8.io docker plugin from Roland Huss. Again it uses the same base image + overlay as 1). The container tag is derived from the maven properties. To build and run you can go into the directory and then $ mvn docker:build $ mvn docker:start 3) docker-dist2/ Similar to 2) this uses the mvn docker plugin, but this time is uses jboss/wildfly:10.0.0.Final as the base image. Unfortunately it still overlays the whole wildfly from us, but we should be able to filter that out. The good thing is though that the image is already much smaller as in 2) To build and run the same process as 2) applies * The next step is now to figure out how to only apply our changes and not to supply the base wildfly into the conatiner. I certainly need help from someone with maven-fu :) But while the current state is not optimal it shows that this is very much doable and we should proceed. Heiko

9 years, 9 months

7
14
0 / 0

memory footprint with swarm

by John Mazzitelli

I was running the swarm agent and so while I was here, I decided to take a quick look at memory footprint. I took these numbers from the Gnome System Monitor GUI utility. swarm agent: Memory=188M, Resident Memory=208M full wildfly agent: Memory=351M, Resident Memory=378M So swarm agent was roughly half the size of full wildfly agent. This was not a scientific study :) I just started the agent, when I saw it finished initializing, I looked at the System Monitor and read the memory numbers. And Jay claims if he put his mind to it, he can shrink the swarm agent further. :) But we aren't doing any more work on it at the moment.

9 years, 9 months

2
1
0 / 0

Updates on android Client of hawkular

by Anuj Garg

Last 10 days haven't been so handsome to me or android client. Most of it upgraded including ->aerogear Library ->butterknife ->gsm ->support libraries Gradle update is not possible at present as sdkmanager plugin by jake warthon haven't been upgraded and its alternative is forcing me to build multidex application. So I am back to old gradle until I find a workaround. Some part in journey gave happiness too. With PR on hawkular android client accepted. It will work like https://drive.google.com/open?id=0B8tYKC1-UERHNWhSeUZQdmVtS2M Next I am planning to direct clicks on metrics to their chart presentation Thanks Anuj Garg

9 years, 9 months

2
1
0 / 0

Anyone available to review......

by Gary Brown

https://github.com/hawkular/hawkular-apm/pull/444 Thanks. Regards Gary

9 years, 9 months

1
0
0 / 0

Hawkular Client Ruby

by Lucas Ponce

Hello, These days we have had several discussions about the hawkular client ruby. Today there are two main uses cases (Hawkular Metrics from OpenShift and Hawkular Services for MiQ provider) and there are some complexity in the side effects of updating the clients related to the backward compatibility. I have started to collect some notes about this https://docs.google.com/document/d/17bAPbBJ6_2wdXAAg_XIQRMi0_RQfODyOEmdOs... I hope it can help to gather all requeriments to have a clear picture of what could be a good decision for future implementations, studying the pros/cons. Please, feel free to add your comments or thoughts about the topic. Thanks, Lucas

9 years, 9 months

2
2
0 / 0

Merging of own pull-requests ( please don't )

by Heiko W.Rupp

Hey, I want to remind everyone (including myself :-) that one should not merge ones own pull-requests (and also abstain from directly committing to master). There are many reasons why this is good practice - peer review finds issues - peer review allows discussions on the concrete implementation of a bugfix/feature request - other findings can be addressed in the same pull-request and do not need a subsequent one. - It gives oneself some time to reflect what was done. - It gives time to also add this one little file that was not git-add'ed I totally understand that we are in some areas rather thin on developers (Juca's work on hawkular-services and accounts come to mind) and that there are always situations where a direct commit or self-merged PR is needed. But it should not be the default. I do not want us to get to a structure where only a handful of managers have push rights to force this above mentioned peer review. Thanks Heiko

9 years, 9 months

2
1
0 / 0

Hawkular Commons 0.7.4.Final

by Juraci Paixão Kröhling

Team, Based on the IRC discussion and on the related thread, I removed the Embedded Cassandra from Nest and did a release of Commons (0.7.4.Final). The integration tests for Nest are deploying the Embedded Cassandra, while the feature pack is not shipping it. If you need help achieving this on your module, let me know. Next week's Hawkular Services will be based on this version of Commons, so, it won't be shipping Cassandra anymore. - Juca.

9 years, 9 months

2
3
0 / 0

Removal of embedded Cassandra

by Juraci Paixão Kröhling

Team, As mentioned on IRC, we should not ship an embedded Cassandra with Hawkular Services. Here are a couple of approaches we can follow to accomplish that: - Keep everything as is, but change the default value for `hawkular.backend` to `remote` instead of the current `embedded_cassandra` - Remove the embedded Cassandra entirely from Nest. This has implications on how the integration tests are executed: we'd need to ensure that each integration test suite starts an embedded Cassandra before the test is executed. Or use an approach like Metrics, which requires Cassandra to be running before the tests are started. Which approach should be the one to follow? - Juca.

9 years, 9 months

6
10
0 / 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

hawkular-dev June 2016