[keycloak-dev] Storage and migration issues

[Bill Burke]

I'm not convinced any of this will make things any better.  It just 
might introduce different problems.  We're still going to have to 
support a relational model for User storage so we're still going to have 
to deal with the headaches of supporting different RDBMS dbs.  I wish we 
could just drop Mongo. Does Red Hat even support Mongo in product?  What 
are the advantages of storing in Mongo anyways?

I do agree that its a pain that at least one of these model backends 
seems to break all the time.  Our migration and import/export facilities 
are not well tested.

* Representations are just going to always be different than the data 
model.  This could be because of REST API backwards compatibility and 
usability.  There also seems to be this mandate that exported datamodels 
must be human creatable and readable.  This means referencing everything 
by name instead of ID and/or creating more hierarchical structures that 
look nice to humans.  All of which don't map very well to a functional 
datastore.
* The way we do caching gives us a lot of flexibility.  We can have 
specific caching logic for various parts of the data model.  All of 
which is datastore type agnostic.
* A lot of users are using UserSession RDBMS store because they can't 
use, or don't know how to use Infinispan in the cloud.
* I don't think users are going to want to manage 2 different types of 
databases, i.e. Mongo + RDBMS.
* User databases need to be queryable in all sorts of ways. This pretty 
much requires a relational model.
* Would client databases need to be queryable?


I do not agree at all that we need to be able to migrate any old version 
of community Keycloak.  I want 1st version of product to be clean and 
not have tons of baggage from deprecated and obsolete APIs, SPIs, and 
data models.  Keycloak will be close to 3 years old when we go to 
product.  It will have changed a lot.  Sure, we should support migrating 
between any version of product because thats one of the things people 
pay us for.  But community?

It just seems like our queue is so big and the benefits of this are 
questionable.

On 7/21/2015 2:50 AM, Stian Thorgersen wrote:
> Maintaining the storage providers are a incredible time sync and very error prone. On top of that we also have migration to deal with. It would be worth to improve this before we go to product.
>
> Why is storage so horrible?:
>
> * Multiple implementations - JPA + Mongo + Representations + Cache
> * Multiple database providers - every time we add anything it seems to break on at least one database
> * Migration - we have 4 different types of migration to deal with. Representations, JPA, Mongo and the generic one as well
>
> Not only do we spend a lot of time even adding the simplest thing to the model, but we also spend a lot of time getting it to work on all supported databases before release.
>
> This all made me think that we have to many options! I don't think users care how we store realm and client meta-data. They may care about users though. With that regards I think we should have a single realm store that is managed by Keycloak instead of supporting an external store. For users we'd still have the option to use JPA or Mongo, but that's a significantly simpler model, in fact I don't think we've barely touched it for a long time. We could also use the same managed store for users OOTB.
>
> For the realm store I'd like to use either a document store or a key-value store. Relational databases are just to much of a pain to get to work, migration is painful and it makes users expect that the database of their choice will work as well.
>
> Alternatives could be:
>
> * Cassandra
> * MongoDB
> * Plain files - we'd have a revision number of a realm or client and we'd replicate the details with Infinispan and store on all nodes
> * Infinispan store - Infinispan has built-in support for persisting data
> * DMR - in domain mode WildFly manages atomic updates to all nodes. Not sure if standalone.xml etc can deal with the amount of data we have though.
> * Continue storing in any store, but store a JSON blob for realms and clients. In relational database we'd have two tables, realms and clients, with an id, revision and blob
>
> With regards to migration I think we DO need to support updating from any version. In community this is an incredibly nice to have, but could also be more than that if a user is stuck on an old version and can't update to the intermediate version due to a bug. However, in product it's going to be required. Think about it a major release is supported for 8 years or something. We could end up with having 4 or more major versions out there at any given time. We can't expect a customer that's been on product v1 to upgrade to v2, then v3 to be able to upgrade to v4.
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>

-- 
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com