[keycloak-user] How to apply updates to keycloak instances

[Jesse Chahal]

@Stian
The approach described sounds similar to liquibase to me but with json
and specific to keycloak. I feel like a lot of possible bugs could
arise from this approach or at least quite a few feature requests.
Would each json file only contain a single change? Would order matter
in how they get applied if you put a bunch of json files in this
directory at once? Can the same file be applied multiple times? These
are the kind of issues I would expect to come up with this type of
change management system. When I mentioned write our own tool/script
to do it I was kind of thinking of a writing a liquibase like system
that calls keycloak's rest api.

@ Scott
If I would compare the solution you mentioned to one of the options I
listed in my original question  "I've also considered writing my own
updater tool using a scripting language (python/ruby) that calls
keycloak's rest api." The worrying thing to me is that there is
another piece of code that needs to maintained by our company and
requires quite a bit of knowledge of keycloak's rest api. There would
probably need to be some serious thought put into the architecture of
the tool as well. Without a doubt it does provide the most control. We
also live by a different methodology in regards to updating production
clusters. From our perspective it is more of an issue to update
manually as it becomes much easier to miss a step or in someway screw
up if steps are performed manually. I'm not sure what the security
implications would be from it occurring automatically, especially if
during each step there is thorough testing (including from a security
team). For our CI/CD pipeline our goal is to have it so every commit
can automagically end up on production without human intervention.

Currently we use a combination of an initial realm file to be included
on startup and also use jq to modify the keycloak-server.json for new
keycloak clusters. We don't need to generate realm or client keys as
it is included in the initial realm file. That doesn't work for
existing systems backed by a database that cannot be thrown away. That
kind of leave me with the original option (and hardest) of "write a
proprietary liquibase like system built ontop of keycloaks rest api".
This is a hard problem to solve

On Mon, May 23, 2016 at 1:49 PM, Anthony Fryer <anthony.fryer at gmail.com> wrote:
> Thanks, I'll check it out.
>
>
> On 05:38, Tue, 24/05/2016 Scott Rossillo <srossillo at smartling.com> wrote:
>>
>> We use Jose4J[0] to create the keys and then jq[1] to modify the realm
>> file.
>>
>>  See the first line of code here for a super simple example of how to
>> generate realm keys:
>> https://bitbucket.org/b_c/jose4j/wiki/JWT%20Examples
>>
>> PS - this may be doable with Keycloak but Jose4J is very lightweight for
>> writing a simple script on a CI server.
>>
>> [0]: https://bitbucket.org/b_c/jose4j
>> [1]: https://stedolan.github.io/jq/
>>
>>
>> Scott Rossillo
>> Smartling | Senior Software Engineer
>> srossillo at smartling.com
>>
>> On May 21, 2016, at 10:20 PM, Anthony Fryer <anthony.fryer at gmail.com>
>> wrote:
>>
>> Hi Scott,
>>
>> How do you generate the realm keys when creating the new keycloak dev
>> instances?  Do you use a keycloak api or some other way?  I'm interested in
>> having a standard realm template that is used to create new realms but would
>> need to change the realm keys when importing this template into keycloak.
>>
>> Cheers,
>>
>> Anthony
>>
>> On Sat, May 21, 2016 at 3:43 AM, Scott Rossillo <srossillo at smartling.com>
>> wrote:
>>>
>>> We’re using Keycloak on production, stage/QA, development environments
>>> and every developer’s workstation / laptop.
>>>
>>> While there will always be differing options on how to successfully do
>>> change management, we’ve found a very effective method for handling Keycloak
>>> provisioning in all environments so that developers don’t need to mess
>>> around with. We’re a continuous integration / deployment shop using micro
>>> services and everything has to “just work” … I’ll give an overview of our
>>> process here but please keep in mind a few things:
>>>
>>> 1. This approach works for us, I’m not saying it’s the best way
>>> 2. We do _not_ allow production config changes to be automated due to
>>> security implications
>>> 3. We're very opinionated in our approach to configuration management and
>>> we don’t ever modify 3rd party software databases directly. We always use
>>> APIs.
>>>
>>> We deploy Keycloak to all environments using Docker images. On developer
>>> workstations we use Docker Compose to orchestrate bringing up all services a
>>> developer may need, including Keycloak.
>>>
>>> We have 4 docker images for Keycloak:
>>> - Keycloak Base
>>>    \- Keycloak HA
>>>    \- Keycloak Dev
>>> - Keycloak config manager*
>>>
>>> The base image includes all customizations necessary to bring up a
>>> Keycloak instance configured with our modules and themes installed.
>>> The HA instance builds off base and configures Keycloak to run as a
>>> cluster node. This is used on stage and prod.
>>> The dev instance builds off base and includes our realm file. On startup,
>>> this instance loads our realm configuration if it’s not already loaded.
>>>
>>> All docker images are built and published by the CI server and Keycloak
>>> HA can be deployed to stage and prod after a clean CI build.
>>>
>>> Developers are free to add clients for testing, do whatever they want,
>>> etc. to their running dev instance. If they want to get back to our stock
>>> build, they pull the latest Docker image from our private Docker repo and
>>> restart it.
>>>
>>> Adding clients to stage and prod requires approval and is done by a hand.
>>> This is for security reasons. Once a configuration change is detected on
>>> stage - say a client is added - our CI server exports the realm from stage,
>>> changes the realm keys, and creates a new Keycloak Dev instance with the
>>> updated realm file.
>>>
>>> *A word about configuration management:
>>>
>>> Obviously, the realm file we generate knows the URLs of staging services,
>>> not local or development environment URLs. To overcome this we introduced
>>> another Docker based service called the Keycloak configuration manger. It
>>> runs on development environments and workstations. It’s responsible for
>>> discovering running services and updating Keycloak via its admin endpoints
>>> to reflect the proper configuration for the given environment.
>>>
>>> That’s it. The whole process is automated with the exception of
>>> configuration changes to stage and prod which require a security review.
>>>
>>> Hope this helps. Let me know if you’d like me to elaborate on anything.
>>>
>>> Best,
>>> Scott
>>>
>>> Scott Rossillo
>>> Smartling | Senior Software Engineer
>>> srossillo at smartling.com
>>>
>>> On May 20, 2016, at 1:46 AM, Stian Thorgersen <sthorger at redhat.com>
>>> wrote:
>>>
>>> Firstly, just wanted to highlight that core Keycloak team are devs, not
>>> sysadmins/ops guys, so we have limited experience in continuous delivery and
>>> maintenance of real production systems. Hence, we'd love input from the
>>> community on this.
>>>
>>> As it stands we don't really have a proper solution. I believe the best
>>> you can do at the moment is either using import feature, partial import or
>>> admin rest endpoints. Import is not going to work IMO as it requires
>>> re-creating the whole realm. Partial import may work, but would work best
>>> for new resources rather than modifying existing resources as it does a
>>> delete/create operation rather than attempt to modify. With the admin rest
>>> endpoints you'd get the best control of what's going on, but obviously that
>>> leaves a fair amount of the work.
>>>
>>> In the future we have an idea of introducing an "import directory" it
>>> would be possible to drop json files in here that would add, modify or
>>> delete resources (realms, clients, roles, users, whatever). This would allow
>>> dropping json files before the server starts and the server would then
>>> import on startup. It would also be possible to do this at runtime and new
>>> files would be detected at runtime. Finally, we also had an idea of an
>>> offline mode to run import of this (it would basically start the server
>>> without http listener, import files, then stop, so it could be used in a
>>> script/tool). Import is probably not the best name for it, as it would
>>> support modify and delete as well as "importing" new things.
>>>
>>> On 19 May 2016 at 19:53, Jesse Chahal <jessec at stytch.com> wrote:
>>>>
>>>> Following some of the best practices for continuous Integration and
>>>> continuous delivery there needs to be environments for build, test,
>>>> and production. This would mean that following these practices would
>>>> require you to have multiple versions of keycloak at different stages
>>>> of development cycle. Some of these environments might not have
>>>> important persistent data while others might. In order to have builds
>>>> transition from one environment to another there may be configuration
>>>> changes required for a build to be valid. This is especially true when
>>>> new services (openid clients) are being added or "default" accounts.
>>>> I'm trying to come up with a scripted way of updating keycloak
>>>> instances that are backed up by an RDMS. This may include adding new
>>>> clients, adding new users, updating realm config, etc... Originally I
>>>> was planning on simply exporting the realm config and importing it
>>>> every time keycloak starts. If I enabled the OVERWRITE option I might
>>>> overwrite things that I do not want overridden. This is especially
>>>> true if there is some config that differ's based on whether it is a
>>>> build, test, or production instance. If I don't enable it then it is
>>>> only useful for new/blank keycloak environments. I considered using
>>>> liquibase but since I do not have control of schema changes created by
>>>> the keycloak team I might run into issues with my liquibase file not
>>>> being valid after a migration/liquibase update by the keycloak team as
>>>> my liquibase file would run after keycloak's does. There might also be
>>>> some other unknown issues our liquibase changes conflicting somehow
>>>> with keycloak's liquibase changes. I've also considered writing my own
>>>> updater tool using a scripting language (python/ruby) that calls
>>>> keycloak's rest api. The issues with this mechanism is it feels like I
>>>> am recreating the wheel as well as not being able to find good
>>>> documentation on keycloak's openid endpoints/url's used for different
>>>> oauth2 flows. Even if I did find this documentation it would also
>>>> require me to find a good openid client for the scripting language.
>>>> This doesn't matter for our normal clients as they simply use the
>>>> keycloak subsystems and adapters instead. I've also looked at commonly
>>>> used server configuration software such as chef, puppet, and ansible.
>>>> I don't see a good solution using any of those tools yet either. What
>>>> have other people done for cases like this? Please don't tell me there
>>>> is someone who is doing this all manually because that doesn't work in
>>>> modern software development.
>>>>
>>>> - doesn't accidentally delete users
>>>> - doesn't accidentally delete clients
>>>> - doesn't invalidate sessions (optional)
>>>> - works to bring up new, correctly configured, keycloak instances
>>>> - handles applying updates to existing keycloak instances
>>>> - can handle minor differences between keycloak instances (build,
>>>> test, production) when updating
>>>> - preferably can work well in rolling deployment scenario's.
>>>> -- I hope the keycloak team is taking these into consideration when
>>>> doing database migration between 1-2 releases. It would be nice if
>>>> they set some specific rules for rolling updates between versions (aka
>>>> backwards breaking changes)
>>>> _______________________________________________
>>>> keycloak-user mailing list
>>>> keycloak-user at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/keycloak-user
>>>
>>>
>>> _______________________________________________
>>> keycloak-user mailing list
>>> keycloak-user at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/keycloak-user
>>>
>>>
>>>
>>> _______________________________________________
>>> keycloak-user mailing list
>>> keycloak-user at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/keycloak-user
>>
>>
>>
>