----- Original Message -----From: "John Sanda" <jsanda@redhat.com>
To: "Discussions around Hawkular development" <hawkular-dev@lists.jboss.org>
Cc: "Matt Wringe" <mwringe@redhat.com>
Sent: Tuesday, March 1, 2016 11:02:33 PM
Subject: Upgrading Hawkular Metrics to Cassandra 3.x
Since Hawkular is still primarily in development mode upgrading dependencies,
including databases, is fairly painless. Metrics 0.8.0 however was included
in OpenShift 3.1, which means upgrading Cassandra in Metrics is going to be
more involved than upgrading the parent pom and upgrading our dev
environments, e.g., changing ccm cluster. The following is a brief
description of the steps that need to be performed,
Run nodetool drain
Shut down the node
Back up configuration and data files
Install new version of Cassandra
Apply configuration changes (from old version) to new version. This would
include files like cassandra.yaml, cassandra-env.sh, etc.
Start the node
Run nodetool upgradesstables
Repeat for each node in the cluster
These steps can be performed as a rolling upgrade; however, no application
schema changes should be made until the upgrade is complete and until
cluster reports schema agreement.
With respect to OpenShift, how much of this needs to be automated? Matt, can
you share some insights here? I would like to figure out sooner rather than
later how much work we think is involved.
If we can automate all of it or parts of it, we should really try and do that. If we can't then we need documentation going over all the steps required and testing to make sure that it all still functions properly.This could be a tricky task to get done right, you will probably want to get started on that sooner rather than later.
The more we need to automate, the more involved and complicated it is going to be. Updating cassandra.yaml isn’t too bad because we use yaml parse to make changes; however, yaml parsers do not preserve comments so we will lose all comments in cassandra.yaml which is unfortunate because they are an important source of documentation. I am not sure that we can programmatically update cassandra-env.sh since it does not have a structured format. In RHQ we essentially replaced cassandra-env.sh with a properties file so that we could use our configuration subsystem to manage it. Note that it did not have to be a properties file, just a structured format. Everything was done in Java since we had to support both Windows and Linux platforms. We should be able to use shell scripts here which will make things a lot easier.