[keycloak-dev] Storage and migration issues

[Stian Thorgersen]

Maintaining the storage providers are a incredible time sync and very error prone. On top of that we also have migration to deal with. It would be worth to improve this before we go to product.

Why is storage so horrible?:

* Multiple implementations - JPA + Mongo + Representations + Cache
* Multiple database providers - every time we add anything it seems to break on at least one database
* Migration - we have 4 different types of migration to deal with. Representations, JPA, Mongo and the generic one as well

Not only do we spend a lot of time even adding the simplest thing to the model, but we also spend a lot of time getting it to work on all supported databases before release.

This all made me think that we have to many options! I don't think users care how we store realm and client meta-data. They may care about users though. With that regards I think we should have a single realm store that is managed by Keycloak instead of supporting an external store. For users we'd still have the option to use JPA or Mongo, but that's a significantly simpler model, in fact I don't think we've barely touched it for a long time. We could also use the same managed store for users OOTB.

For the realm store I'd like to use either a document store or a key-value store. Relational databases are just to much of a pain to get to work, migration is painful and it makes users expect that the database of their choice will work as well.

Alternatives could be:

* Cassandra
* MongoDB
* Plain files - we'd have a revision number of a realm or client and we'd replicate the details with Infinispan and store on all nodes
* Infinispan store - Infinispan has built-in support for persisting data
* DMR - in domain mode WildFly manages atomic updates to all nodes. Not sure if standalone.xml etc can deal with the amount of data we have though.
* Continue storing in any store, but store a JSON blob for realms and clients. In relational database we'd have two tables, realms and clients, with an id, revision and blob

With regards to migration I think we DO need to support updating from any version. In community this is an incredibly nice to have, but could also be more than that if a user is stuck on an old version and can't update to the intermediate version due to a bug. However, in product it's going to be required. Think about it a major release is supported for 8 years or something. We could end up with having 4 or more major versions out there at any given time. We can't expect a customer that's been on product v1 to upgrade to v2, then v3 to be able to upgrade to v4.