major change in the way we will have to monitor domain mode
by John Mazzitelli
I'm looking at how domain mode monitoring is going to work now that we are moving toward the Prometheus / jmx-exporter way of doing things.
Turns out, due to the way WildFly domain mode exposes things via JMX (rather, I should say, *doesn't* expose things) compared to DMR, I think we are going to have to require the hawkular agent to be deployed to all slave servers, in addition to the master host controller.
In domain mode, you can access all the child slave servers (the individual servers in all server groups) via the host controller's DMR tree - so we only needed one agent on the host controller to monitor the domain. For example, to see a metric for a deployment on "server-one" that is managed by host controller "master", I can simply ask the host controller for it:
/host=master/server=server-one/deployment=test-simple-1.0.0.war/subsystem=undertow/:read-attribute(name=active-sessions)
{
"outcome" => "success",
"result" => 0
}
The nice thing about WildFly's management DMR interface is that the domain mode tree is identical to the standalone tree with the exception that on the host controller, you simply prepend the host/server identifiers to the DMR path. So, for a standalone WildFly, a DMR path for the active-sessions metric for my test-simple application would be:
/deployment=test-simple-1.0.0.war/subsystem=undertow/:read-attribute(name=active-sessions)
If I want to ask the host controller to give me the exact same metric from that server, I simply prepend the host/server name in the DMR path:
/host=master/server=server-one/deployment=test-simple-1.0.0.war/subsystem=undertow/:read-attribute(name=active-sessions)
The agent has knowledge of this pattern; knowing this pattern we can cleverly configure the metadata so we can share types across domain and standalone for inventory discovery.
The problem: WildFly does not expose its JMX MBeans in an equally clever way. I do not see MBeans that provide metrics for the slave servers.
For example, in JMX, I see WildFly host controller has this MBean:
jboss.as:host=master,server=server-one
Looks very analogous to the DMR resource named /host=master/server=server-one resource ... right??
Well, there are no deployments associated with this MBean. You would think (following the DMR pattern) that there would be an MBean named:
jboss.as:host=master,server=server-one,deployment=test-simple-1.0.0.war
But there is not. Nor is there:
jboss.as:host=master,server=server-one,deployment=test-simple-1.0.0.war,subsystem=undertow
which is where you would expect that server's web subsystem metrics to be (if it were to follow the same DMR pattern). But, again, this doesn't exist.
I can't find JMX MBeans for anything related to the individual slave servers (not just deployments).
In short, I do not believe the Prometheus JMX Exporter can be used to collect metrics for all slave servers in a domain if that JMX Exporter is simply installed on a host controller. This is because the host controller's JMX MBeans do not expose metric data for individual slave servers.
We would have to have our agent in each slave server (which are just standalone servers - so the agent would be as if it is monitoring a standalone server). We could have an agent in the host controller, too, but it would only be responsible for monitoring/managing the host controller itself.
[this message was sent on Tuesday, November 21, 2017 at 10:11 PM EST]
2 months, 1 week
Hawkular Metrics 0.13.0 - Release
by Stefan Negrea
Hello Everybody,
I am happy to announce release 0.13.0 of Hawkular Metrics. This release is anchored by Hawkular integration enhancements, under-the-cover refactorings and fixes, and a new bulk data generation tool.
Here is a list of major changes in this release:
1) Data Generation Tool
* A new tool that generates bulk metrics data to be used in performance and load testing.
* It generates directly Cassandra data files (SSTables), which leads to a very fast generation process for large amounts of metrics data.
* For more details: https://github.com/hawkular/hawkular-metrics/tree/master/data-generator (HWKMETRICS-355)
2) Receive Metrics via Hawkular Bus
* When deployed within Hawkular distribution, the project now accepts metrics via the Hawkular Bus; until now only the REST API had support for Metrics insertion.
* Hawkular Metrics currently support publishing of newly inserted metrics to the bus and receiving metrics via the bus.
* For more details: HWKMETRICS-347, HWKMETRICS-352
3) Sorted Stacked Metrics Results
* When requesting stacked metrics aggregation the results are now ordered (HWKMETRICS-353)
4) External Integrations
* Heapster sink now divides storing to multiple calls. This is an improvement over the initial implementation that had one REST API call per metric (HWKMETRICS-290)
* ptrans now works with a Hawkular Metrics instance protected by Hawkular Accounts via Hawkular distribution (HWKMETRICS-342)
* Grafana integration via Influxdb compatible end-point allows connections to a Hawkular Metrics instance protected by Hawkular Accounts via Hawkular distribution (HWKMETRICS-343)
Github Release:
https://github.com/hawkular/hawkular-metrics/releases/tag/0.13.0
JBoss Nexus Maven artifacts:
http://origin-repository.jboss.org/nexus/content/repositories/public/org/...
Jira release tracker:
https://issues.jboss.org/browse/HWKMETRICS/fixforversion/12329530
Hawkular Metrics Clients
* Ruby: https://github.com/hawkular/hawkular-client-ruby
* Python: https://github.com/hawkular/hawkular-client-python
* Go: https://github.com/hawkular/hawkular-client-go
A big "Thank you" goes to John Sanda, Thomas Segismont, Mike Thompson, Matt Wringe, Michael Burman, and Heiko Rupp for their project contributions.
Thank you,
Stefan Negrea
Software Engineer
3 months, 1 week
[hawkular-alerts] Integration with Eureka
by Ashutosh Dhundhara
Hi All,
Is it possible to track UP/DOWN status of micro-services registered with
Eureka and fire email notifications if a service goes down? If yes, please
point me to right direction.
--
Regards,
Ashutosh Dhundhara
3 months, 1 week
Welcome Pallavi Gupta as our GSoC student for this summer
by Heiko W.Rupp
Hey,
please welcome Pallavi as our GSoC 2017 student.
Pallavi will work on improving the Hawkular-Android client
especially in the area of Alerting and Alert trigger setup.
Anuj, our last year GSoC student will be her main mentor.
Daniel Passos and I will also help here.
Heiko
--
Reg. Adresse: Red Hat GmbH, Technopark II, Haus C,
Werner-von-Siemens-Ring 14, D-85630 Grasbrunn
Handelsregister: Amtsgericht München HRB 153243
Geschäftsführer: Charles Cachera, Michael Cunningham, Michael O'Neill,
Eric Shander
6 months
Issues installing Hawkular Service and Alert
by Hassnan Khan
My name is Hassnan Khan and I work at Fandango in the NOC department. We
use an ELK stack to process our logs into readable data that we see on
Kibana (4.6.4) / ElasticSearch 2.4.x. We've been trying to get a standalone
alerting system and haven't had much luck. I came across Hawkular Services
+ Alerts and thought this is something we could possibly use.
Unfortunately, we haven't been able to install it successfully. We've tried
all the installation guides on github + hawkular blogs and keep getting an
error when trying to connect to Cassandra even though it is obviously
running.
Is there any tips you can give on successfully installing the suite?
6 months, 2 weeks
hawkular wildfly agent - feed ID
by John Mazzitelli
There was some concern that the feed ID autogenerated by the agent might not be what a person always wants (e.g. feed ID for WF10 agents will be the unique UUID of the wildfly server).
So, if you want to define your own feed ID, you can configure it now. In the agent's standalone.xml <storage-adapter> you can specify a feedId attribute if you want. By default, it isn't specified, so the agent will autogenerate a unique feed ID for itself (this should really be the normal mode of operation, but some people like to complain, so I made it easy to shut them up :-)
This is in master. Will be in the next release.
6 months, 3 weeks
Maven swagger plugin
by Gary Brown
Hi
The current version of the maven-swagger-plugin used by hawkular doesn't include derived types in the generated docs.
I've tried updating to 2.3.4 and it still does not work. From some basic tests with 3.0.0, it seems to produce the correct content in the json schema - but has slightly different attributes to the plugin configuration, which when updated does not seem to run the template correctly.
The other issue is some API changes when updating the com.wordnik:swagger-* dependencies in line with this maven plugin, as there have been some GAV and API changes.
Its not an urgent issue, but it looks like we will be impacted by changes when we need to upgrade, so thought I would raise the issue.
Regards
Gary
10 months, 3 weeks
[GSoC] Hawkular Android Client
by Artur Dryomov
Hi everyone,
This year I will be working on the Hawkular Android client application as
part of the Google Summer of Code 2015 program.
The application itself will use Hawkular API and AeroGear SDK. In coming
days I’ll research these areas, especially documentation, and will try to
create some sort of architecture and basic design.
Thank you all for this opportunity!
Artur.
1 year, 2 months
Hawkular Metrics Openshift Containers
by Matt Wringe
I have a new subproject in Hawkular Metrics which sets up creating
components for Openshift/Fabric8
(https://github.com/hawkular/hawkular-metrics/pull/200).
There are 3 main parts
Cassandra: creates a custom seed provider to support
ReplicationControllers in Kubernetes, creates a folder/zip archive which
can be used to generate a Docker image. It may make sense to move the
Cassandra parts out to a separate project.
Hawkular Metrics: creates a folder/zip archive which can be used to
generate a Docker image
Kubernetes: pulls everything together into a single kubernetes
application. Can be used to deploy an application zip into fabric8 (via
drag and drop in the web console or via the maven plugin) or deploy all
the components into Openshift via the kubernetes.json configuration file.
The docker images are not created and deployed to a docker registry as
part of the build, it will just create a folder where you can run the
docker build from. None of the maven docker plugins I looked at seemed
to really work properly, so its still a manual process to do the build
(and push to a registry). Its something which needs to be improved.
The Cassandra service currently only supports adding new nodes to a
cluster and not removing them via the ReplicationController. This is due
to the replication factor being set to be 1 by default (which means when
a node is removed, so is the data it contained).
I believe the docker subproject of hawkular metrics is obsolete and can
be removed
(https://github.com/hawkular/hawkular-metrics/tree/master/docker), but
someone please correct me if I am wrong. It's scripts are referring to
the console which no longer exists as part of the project.
- Matt
1 year, 2 months
