[keycloak-dev] Storage and migration issues

Tue Jul 21 13:53:50 EDT 2015

In the near future, Marek and I just want to drop support for migrating 
earlier keycloak community versions so we can clean up the codebase and 
datamodel and drop a bunch of migration code.  The idea would be that 
you would have to upgrade to 1.7 community before you could upgrade to 
2.0 community.  As it is, we don't do point releases of earlier 
community versions anyways.

On 7/21/2015 12:07 PM, Scott Rossillo wrote:
>
> Please do not stick community users without an upgrade path. I know upgrades can be painful but isn’t this part of the reason for using Liquibase? Also consider that some of today’s community users may wish to become paying customers. If we can’t get from 1.x to whatever becomes product, we can’t pay RedHat. :) Besides, keeping the upgrades working keeps the product honest and fosters a more open community of users and contributors.
>
> IMO, the only real advantage of Mongo over an RDBMS for Keycloak is that it can be easily replicated and sharded. So, in theory, performance and uptime can be improved over a relational DB. However, in practice, some of the RDBMS systems can be set up for serious HA.
>
> I’ve been reading the threads on compatibility and I’m not so sure I understand why deprecated APIs and SPIs are required to be kept around to support an upgrade? Is that for non RDBMS data stores?
>
>
>> On Jul 21, 2015, at 11:23 AM, Bill Burke <bburke at redhat.com> wrote:
>>
>> I'm not convinced any of this will make things any better.  It just
>> might introduce different problems.  We're still going to have to
>> support a relational model for User storage so we're still going to have
>> to deal with the headaches of supporting different RDBMS dbs.  I wish we
>> could just drop Mongo. Does Red Hat even support Mongo in product?  What
>> are the advantages of storing in Mongo anyways?
>>
>> I do agree that its a pain that at least one of these model backends
>> seems to break all the time.  Our migration and import/export facilities
>> are not well tested.
>>
>> * Representations are just going to always be different than the data
>> model.  This could be because of REST API backwards compatibility and
>> usability.  There also seems to be this mandate that exported datamodels
>> must be human creatable and readable.  This means referencing everything
>> by name instead of ID and/or creating more hierarchical structures that
>> look nice to humans.  All of which don't map very well to a functional
>> datastore.
>> * The way we do caching gives us a lot of flexibility.  We can have
>> specific caching logic for various parts of the data model.  All of
>> which is datastore type agnostic.
>> * A lot of users are using UserSession RDBMS store because they can't
>> use, or don't know how to use Infinispan in the cloud.
>> * I don't think users are going to want to manage 2 different types of
>> databases, i.e. Mongo + RDBMS.
>> * User databases need to be queryable in all sorts of ways. This pretty
>> much requires a relational model.
>> * Would client databases need to be queryable?
>>
>>
>> I do not agree at all that we need to be able to migrate any old version
>> of community Keycloak.  I want 1st version of product to be clean and
>> not have tons of baggage from deprecated and obsolete APIs, SPIs, and
>> data models.  Keycloak will be close to 3 years old when we go to
>> product.  It will have changed a lot.  Sure, we should support migrating
>> between any version of product because thats one of the things people
>> pay us for.  But community?
>>
>> It just seems like our queue is so big and the benefits of this are
>> questionable.
>>
>> On 7/21/2015 2:50 AM, Stian Thorgersen wrote:
>>> Maintaining the storage providers are a incredible time sync and very error prone. On top of that we also have migration to deal with. It would be worth to improve this before we go to product.
>>>
>>> Why is storage so horrible?:
>>>
>>> * Multiple implementations - JPA + Mongo + Representations + Cache
>>> * Multiple database providers - every time we add anything it seems to break on at least one database
>>> * Migration - we have 4 different types of migration to deal with. Representations, JPA, Mongo and the generic one as well
>>>
>>> Not only do we spend a lot of time even adding the simplest thing to the model, but we also spend a lot of time getting it to work on all supported databases before release.
>>>
>>> This all made me think that we have to many options! I don't think users care how we store realm and client meta-data. They may care about users though. With that regards I think we should have a single realm store that is managed by Keycloak instead of supporting an external store. For users we'd still have the option to use JPA or Mongo, but that's a significantly simpler model, in fact I don't think we've barely touched it for a long time. We could also use the same managed store for users OOTB.
>>>
>>> For the realm store I'd like to use either a document store or a key-value store. Relational databases are just to much of a pain to get to work, migration is painful and it makes users expect that the database of their choice will work as well.
>>>
>>> Alternatives could be:
>>>
>>> * Cassandra
>>> * MongoDB
>>> * Plain files - we'd have a revision number of a realm or client and we'd replicate the details with Infinispan and store on all nodes
>>> * Infinispan store - Infinispan has built-in support for persisting data
>>> * DMR - in domain mode WildFly manages atomic updates to all nodes. Not sure if standalone.xml etc can deal with the amount of data we have though.
>>> * Continue storing in any store, but store a JSON blob for realms and clients. In relational database we'd have two tables, realms and clients, with an id, revision and blob
>>>
>>> With regards to migration I think we DO need to support updating from any version. In community this is an incredibly nice to have, but could also be more than that if a user is stuck on an old version and can't update to the intermediate version due to a bug. However, in product it's going to be required. Think about it a major release is supported for 8 years or something. We could end up with having 4 or more major versions out there at any given time. We can't expect a customer that's been on product v1 to upgrade to v2, then v3 to be able to upgrade to v4.
>>> _______________________________________________
>>> keycloak-dev mailing list
>>> keycloak-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>
>>
>> --
>> Bill Burke
>> JBoss, a division of Red Hat
>> http://bill.burkecentral.com
>> _______________________________________________
>> keycloak-dev mailing list
>> keycloak-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>

-- 
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com