On Jan 4, 2016, at 5:43 AM, Juraci Paixão Kröhling <jpkroehling(a)redhat.com> wrote:
Team,
What's the recommended approach for handling data migrations? Is there a
library similar to liquibase?
- Juca.
Liquibase is designed specifically for the RDBMS. When RHQ started moving to Cassandra, I
started working on a patch for Liquibase to add support for Cassandra. After some
discussion on the liquibase dev list, I eventually decided to abandon the effort because
of the amount of changes involved and because it became clearer that liquibase was not a
good fit because of it being very RDBMS-centric. We decided to implement our own solution
in RHQ to address our immediate needs. It has been a while since I have looked to see what
other solutions might be out there. I have come across something for Rails applications,
and I think someone may have tried to add support in Flyway.
There are some things that need to be taken into consideration. I will briefly discuss
some of those now.
* Should the migrations be done at installation/deployment time or at runtime?
This is probably the most important consideration because everything else in large part
stems from it. Some changes like adding/removing a column or adding/removing a table are
fast and efficient in Cassandra. I therefore think it is acceptable to do these types of
changes at deployment time. Other changes like adding data or moving/transforming data
that could be long running operations. While it increases application code complexity,
these changes should be done at runtime generally speaking.
* How should migrations be implemented?
With the RDBMS, we can easily manipulate, transform, and move data with SQL. That is not
the case with CQL. We have to resort to writing code on top of the driver to make the
changes. In some situations a better approach might be to generate new SSTables and stream
those into Cassandra. For larger data migrations is likely to be a faster as you
completely bypass the whole CQL layer. Ultimately, I think both approaches need to be an
option.
* Where should migration meta data be stored?
We need to keep track of migrations that have been applied. There might be migrations that
are specific to a particular environment, e.g., dev vs prod. Since we are trying to avoid
additional data stores, I think it makes sense to store migration meta data in Cassandra.
Maybe we a migrations keyspace that tracks the migrations for each of the hawkular
keyspaces.