[keycloak-dev] Data migration strategy

[Stian Thorgersen]

As mentioned before to allow users from keeping their existing database when upgrading to a new version we'll need a migration strategy.

There's 3 separate things to deal with: JPA database, Mongo database and finally JSON representation (exports). For 1.1.0.Beta1 as the JPA and Mongo schemas have changed we'll need to sort out for this release. JSON representations have not changed (afaik) so we can leave this for later when it's needed.

For relational databases I propose (and I have a working prototype for this) we use Liquibase. Liquibase makes it possible to create a generic "change-set" that transforms a database from one version to the next. As it supports creating these change-sets from a running database it's fairly trivial to generate this automatically and only requires a short review. If data in any columns needs updating or adding that still needs to be done manually. The change-sets are also database agnostic and Liquibase concerts these into SQL specific to the current database. I've made a JPAUpdaterSpi, with Liquibase being an optional provider. This means there's no hard dependency on Liquibase. However, if it's not included it won't be possible to upgrade the database automatically. In that case the user could use Liquibase command line tool to update the database manually.

For Mongo databases I propose something similar to Liquibase, but we'll have to write it ourselves. We'll create an interface that has one method transform(DB db), and for each release where data has changed we'll have to provide an implementation of this that can rewrite the data. We'll maintain a list of these transformations, and store which has been applied in a collection.

For representations (again we don't need this now) we'll do something similar to Mongo, a chain of transforms that can convert the JSON representations from one version to the next.

With regards to Liquibase there was recently a thread on tag that suggest we discuss inclusion of 3rd party components in projects. AeroGear also have similar needs and they want to use LiquiBase as well. Just to make sure there's no issues with using Liquibase I'll write a mail to tag. Although, other similar projects are far from as nice as Liquibase (don't have database agnostic script, don't have automatic generation of changsets, etc.). If we had to do it manually we'd have to replicate what Liquibase does with regards to recording applied changes in a table, then we'd also have to write SQL scripts for each release for each database we support.

First before I write mail to tag. Any objections to using Liquibase here? Or proposals for another solution?