Get rid of Travis?
by Michael Burman
Hi,
I'm proposing we get rid of the Travis for now (at least for metrics)
and stick to something else, such as the old Jenkins. At the moment, it
can take several runs for metrics PRs to finish, since Cassandra stops
responding in the Travis runs every time. Yesterday I started my PR 4
times, last week one PR for 10 times.
22:28:24,629 ERROR [com.datastax.driver.core.ControlConnection]
(cluster1-reconnection-1) [Control connection] Cannot connect to any
host, scheduling retry in 600000 milliseconds
Sometimes it doesn't even get this far and fails with running the
installation script of Cassandra. These errors have made the Travis runs
completely irrelevant, whenever they fail it just makes sense to restart
them without reading logs (we should get automatic script to restart
always when it fails) until they succeed. No errors reported by Travis
are trustworthy.
We need a working CI solution and this isn't it. There's no "community
visibility" if the results have no meaning, using Jenkins at least
provides us with "failed / not failed", Travis doesn't provide even this.
- Micke
8 years, 7 months
Inventory: transient feeds - or how to tackle the pets vs cattle scenario
by Heiko W.Rupp
Hey,
Right now we identify "agents" via their feed-id.
An instrumented wildfly comes online, registers
its feed with the server, sends its resource discovery
and later metrics with the feed id.
Over its lifecycle, the server may be stopped and re-started
several times.
This is great in the classical use case with installations
on tin or VMs.
In container-land especially with systems like Kubernetes,
containers are started once and after they have died for
whatever reason they are not restarted again.
So the id of an individual container is less and less interesting.
The interesting part is the overall app, that contains of many
containers linked together with several of them representing
an individual service of the app.
So basically we would rather need to record the app and other
metadata for identifying individual parts of the app (e.g. the web
servers or the data bases) and then get pointers to individual
stuff.
The feed would not need to survive for too long, but some of
its collected data perhaps. And then e.g. the discovery of resources
in a new container of the exact same type as before should be sort
of a no-op, as we know this already. Could we short-circuit that
by storing the docker-image-hash (or similar) and once we see this
known one abort the discovery?
Another aspect is certainly that we want to keep (some) historic
records of the died container - e.g. some metrics and the point
when it died. Suppose k8s kills a container and spins a new one
up (same image) on a different node, then logically it is a continuation
of the first one, but in a different place (but they have different feed
ids)
Now a more drastic scenario: As orchestration systems like k8s or
Docker-Swarm have their own registries, that can be queried : do we need
a
hawkular-inventory for this at all?
( We still need it for the non-OpenShift/K8s/Docker-Swarm envs )
--
Reg. Adresse: Red Hat GmbH, Technopark II, Haus C,
Werner-von-Siemens-Ring 14, D-85630 Grasbrunn
Handelsregister: Amtsgericht München HRB 153243
Geschäftsführer: Charles Cachera, Michael Cunningham, Michael O'Neill,
Eric Shander
8 years, 7 months
Hawkular Inventory 0.20.0.Final released
by Lukas Krejci
Hi all,
I'm proud to announce that Hawkular Inventory 0.20.0.Final has been released.
Apart from an updated version of Sqlg and slightly reduced distribution size,
the *big* new feature this version brings is the support for keeping history
of changes to inventory. From now on, you are able to retrieve the history of
updates made to an entity, but you can also "look at" an entity, or whole
inventory, as it looked at certain point in time - questions like "what DB was
this application connected to last month?", "What was the list of feeds
reporting on our production environment in March", etc. All that possible
using a single new query parameter to the REST endpoints - "?at=<TIMESTAMP>".
0.20.0.Final doesn't offer the ability to compact or delete the "old" data
from inventory (delete now merely marks the entity in question as deleted from
the specified time onward) - that will be the focus of our very next release.
We wanted to let this release out though so that people can start enjoying the
nostalgic views at the good old days of their inventory ;)
Special thanks go out to Joel Takvorian for reviews, fixes and the work in
upstream Sqlg.
Thanks,
--
Lukas Krejci
8 years, 7 months
[hawkular-apm]How to inspect query string within a HttpRequest
by 埜陆
hi everyone I just want to make the query string as a property within a
httprequest by setting the businessTransaction
e.g. got two httprequest
GET /foo/bar?text=hello
GET /foo/bar?text=world
and want to classify them with query text by evaluateURL
then node will got a property with name text and values hello and
world
after seeing the apm-javax-servlet.json only the 'processHeaders' will
trigger a businessTranaction process when all other 'setdetails' has not
been invoked which will attach the query string to 'node details' , so the
action 'evaluateUrl' will never take effect, and all other actions like
'setContent' or 'setProperty' only findvalues in Headers and
Content, unfortunately they don't hold the query string which only exists
in details by default.
I am new to this, so am I missing something ? is there any way to achieve
what I want without modify the default config? thanks
best regards
8 years, 7 months
RfC: no longer support Ruby 2.0 in Ruby-Gem?
by Heiko W.Rupp
Hey,
ManageIQ is using Ruby 2.2.4+ and Ruby 2.0 is
pretty old now (Jruby 9.1.x is Ruby 2.3+), so
I would like to no longer "support" Ruby 2.0.
For the time being that only means that
we do not test against 2.0 anymore in Travis.
What do people think?
What about 2.1 ?
Heiko
8 years, 7 months
Cassandra Cluster with docker.
by Ruben Vargas Palma
I was testing a little bit with Hawkular-Services + Cassandra cluster
using docker containers, I created a GitHub repository with my setup:
https://github.com/rubenvp8510/hwk-cassandra-docker
It seems to work fine, at least on my environment, I did some different
tests but I couldn't test more than 3 nodes on my machine, but hopefully
it will work for N nodes.
Some issues I found:
- Sometimes hawkular throws a timeout exception "Cassandra timeout
during read query at consistency LOCAL_ONE", I only managed to reproduce
it 2 times.
- This setup is not enough robust because the Keyspace has replica
factor of 1, so if one node dies hawkular-services started complain
about it.
Regards
8 years, 7 months
Hawkular Metrics 0.20.0 - Release
by Stefan Negrea
Hello Everybody,
I am happy to announce release 0.20.0 of Hawkular Metrics. This is a major
release anchored by the inclusion of Hawkular Alerting
<https://github.com/hawkular/hawkular-alerts> in the distribution, a brand
new compression algorithm, and numerous API enhancements and fixes.
*Hawkular Metrics + Alerting*
Hawkular Metrics now includes Hawkular Alerting out of the box; it's no
longer necessary to install and manage two separate components. Hawkular
Metrics delivers a single EAR file containing components from both
projects. Incoming metric data is efficiently filtered and evaluated,
generating alerts and events for defined triggers. The combined EAR
artifact is the primary binary distribution of Hawkular Metrics going
forward. For more details: HWKMETRICS-491
<https://issues.jboss.org/browse/HWKMETRICS-491>
Hawkular Alerting Details
- Version 1.2.1
<https://issues.jboss.org/browse/HWKALERTS/fixforversion/12331699/> is
included in this release
- Project details and repository: Hawkular Alerting - Github
<https://github.com/hawkular/hawkular-alerts>
- Documentation: Hawkular Alerting REST API Documentation
<http://www.hawkular.org/docs/rest/rest-alerts.html>, Hawkular Alerting
Examples
<https://github.com/hawkular/hawkular-alerts/tree/master/examples>, Hawkular
Alerting Developer Guide
<http://www.hawkular.org/community/docs/developer-guide/alerts.html>
Deployment Notes
1. Deployment Naming
- The new EAR should be deployed as 'hawkular-metrics.ear' in
WildFly. This naming is required to allow for proper dependencies to be
established by the EAR components.
2. standalone.xml Configuration
- The following snippet must be added to standalone.xml to support
the Hawkular Alerting component. Add this additional
cache-container entry
after the existing cache-container entries:
<cache-container name="hawkular-alerts"
module="org.jboss.as.clustering.infinispan">
<local-cache name="partition"/>
<local-cache name="triggers"/>
<local-cache name="data"/>
<local-cache name="publish"/>
<local-cache name="schema"/>
</cache-container>
Usage Notes
To use Hawkular Alerting with Hawkular Metrics there is a naming convention
when defining trigger conditions. For a metric with name 'X', the alerting
DataId to reference it will be '*X',* where the depends on the metric's
type. For example, let's consider a metric with name 'HeapUsed' and type
'gauge'. This example defines a trigger condition to test if the used heap
exceeded 80%: "hm_g_HeapUsed > .8". The 'hm_g' prefix indicates that
HeapUsed is a gauge.
Because Hawkular Metrics allows the same metric name for different types,
the prefix is needed to make clear the target metric. The prefixes are:
- hm_a: availability
- hm_c: counter
- hm_cr: counter rate
- hm_g: gauge
- hm_gr: gauge rate
- hm_s: string
*Compression*
Hawkular Metrics will now compress all the data with a compression scheme
based on Facebook's Gorilla paper. A job will be run every two hours that
compresses data received during the 2 hour block since last compression
(there's one hour time for out-of-order writes to be written to the
Cassandra). These data points are then written to a new storage format
that's designed after the Gorilla compression and also compressed with the
LZ4 algorithm to give even better compression ratio. Benefits include a
faster reading time of larger block of datapoints and very large disk space
savings. For more details: HWKMETRICS-464
<https://issues.jboss.org/browse/HWKMETRICS-464>
*Other major changes*
-
REST API - Query Improvements
- Querying raw & rate data by tags is now also possible via POST
{metric_type}/query endpoints for all types of metrics (HWKMETRICS-466
<https://issues.jboss.org/browse/HWKMETRICS-466>)
- Added new POST endpoints for stats gauges/stats/query and
counters/stats/query (HWKMETRICS-465
<https://issues.jboss.org/browse/HWKMETRICS-465>)
- It is now possible to fetch metric definitions filtered by a list
of ids not just tags (HWKMETRICS-461
<https://issues.jboss.org/browse/HWKMETRICS-461>)
-
Cassandra Driver Configuration
- Driver configuration options are now exposed for connection and
request timeouts
- Connection timeout can be configured via
CASSANDRA_CONNECTION_TIMEOUT environment variable, or
hawkular-metrics.cassandra.connection.timeout system property; value
is specified in milliseconds and defaults to 5 seconds.
- Request timeout can be configured via CASSANDRA_REQUEST_TIMEOUT
environment variable, or hawkular-metrics.cassandra.request.timeout
system property; value is specified in milliseconds and defaults to 12
seconds.
- For more details: HWKMETRICS-490
<https://issues.jboss.org/browse/HWKMETRICS-490>
-
Job Scheduler
- Duplicate instances of jobs were wrongly scheduled on server restart
because the job scheduler was not checking if the job was
already scheduled
(HWKMETRICS-461 <https://issues.jboss.org/browse/HWKMETRICS-461>)
- Triggers now have a delay to properly schedule jobs in the future (
HWKMETRICS-224 <https://issues.jboss.org/browse/HWKMETRICS-224>)
- Resolved an issue where long running job can miss future executions
because the current execution takes longer than the repeat interval (
HWKMETRICS-477 <https://issues.jboss.org/browse/HWKMETRICS-477>)
- Prevent execution of a job multiple times for the same execution
time (HWKMETRICS-486 <https://issues.jboss.org/browse/HWKMETRICS-486>)
- RepeatingTrigger now allows specifying trigger start time (
HWKMETRICS-476 <https://issues.jboss.org/browse/HWKMETRICS-476>)
-
Admin Endpoints
- Admin related endpoints are going to be protected via an admin token,
to be sent for admin related REST endpoints via Hawkular-Admin-Token
request header
- The first endpoint to be protected by this mechanism is /tenants
since its functionality is cross tenant
- The admin token can be set via command line arguments or
environment variable at container startup; by default it is not set which
means no access to protected endpoints
- This is the foundation for adding more admin related functionality
in upcoming releases (such as Cassandra cluster metrics, system
telemetry,
or admin operations)
- The details:
- system property key: hawkular.metrics.admin-token
- environment variable: ADMIN_TOKEN
- default value: null (admin endpoints are not accessible until
set)
- header name: Hawkular-Admin-Token
- stored in system config using one-way hashing
- For more details: HWKMETRICS-478
<https://issues.jboss.org/browse/HWKMETRICS-478>
*Hawkular Metrics Clients*
- Python: https://github.com/hawkular/hawkular-client-python
- Go: https://github.com/hawkular/hawkular-client-go
- Ruby: https://github.com/hawkular/hawkular-client-ruby
- Java: https://github.com/hawkular/hawkular-client-java
*Release Links*
Github Release:
https://github.com/hawkular/hawkular-metrics/releases/tag/0.20.0
JBoss Nexus Maven artifacts:
http://origin-repository.jboss.org/nexus/content/repositories/public/org/
hawkular/metrics/
Jira release tracker:
https://issues.jboss.org/browse/HWKMETRICS/fixforversion/12331360
A big "Thank you" goes to John Sanda, Matt Wringe, Michael Burman, Joel
Takvorian, Jay Shaughnessy, Lucas Ponce, and Heiko Rupp for their project
contributions.
Thank you,
Stefan Negrea
8 years, 7 months
Heads up: PRs in h-services are now also tested with the ruby-client tests
by Heiko W.Rupp
Hey,
thanks to Josejulios work, we are now running an additional test job
on hawkular-services after the normal build that puts the pr-outcome
into a Docker container, starts this and runs the Ruby-client tests agains
it.
This may trigger false positives / negatives but should gives us
an additional indication if changes in components would have a
negative effect on the work with ManageIQ.
8 years, 7 months
Reminder: Hawkular-services release timing
by Heiko W.Rupp
Hey,
a short reminder: Hawkular-services are (normally) done
on Tuesday (mid-) morning Europe time.
(Updates of) components that want to be included need
to have a PR ready at this point in time.
The PR for the component must not be for a src-dep nor
a Snapshot.
8 years, 7 months
