Re: [keycloak-user] How to apply updates to keycloak instances

I've been thinking about this problem for awhile and so far the solutions that I come up with all require that keycloak keeps tracks of changes in a database table (exactly how it works for liquibase). The GUI has a partial import feature. I haven't used it too extensively but I believe it probably does some sort of JSONtoPOJO serialization in order to figure out what the partial update it needs to be doing. Maybe we could add unique id identifiers to the existing/exported JSON files and have keycloaks import features determine whether the JSON file had already been applied or not. If there is a rest api for this as well then building an external cli or GUI tool would be much more feasible. Scott's solution requires either the external app to know the state of keycloak or keycloak's state to be blank. Its the best that could have been done with keycloak as it is now. Anyone have any comments regarding this possible solution? On Thu, May 26, 2016 at 11:19 PM, Stian Thorgersen <sthorger(a)redhat.com&gt; wrote: > Can you give me some examples of issues around Dockerized deployments and > services that are located at runtime (do you mean services that are > provisioned at runtime?)? > > On 26 May 2016 at 19:47, Scott Rossillo <srossillo(a)smartling.com&gt; wrote: >> >> Stian, that’s fair, it does solve the OP's CI/CD problem when moving in >> the dev -> stage -> prod direction. >> >> Scott Rossillo >> Smartling | Senior Software Engineer >> srossillo(a)smartling.com >> >> On May 26, 2016, at 1:41 PM, Stian Thorgersen <sthorger(a)redhat.com&gt; wrote: >> >> >> >> On 26 May 2016 at 19:11, Scott Rossillo <srossillo(a)smartling.com&gt; wrote: >>> >>> I guess it’s a matter of requirements, but with micro service >>> architectures there’s usually some sort of discovery mechanism required to >>> locale services at runtime. Netflix offers Eureka and then there’s etcd from >>> CoreOS that’s being used by Kubernetes. My point is that even if Keycloak >>> devs build some sort of way of picking up changes from the filesystem on >>> startup, that doesn’t solve all use cases. >> >> >> The problem is continuous integration right, and pushing changes from a >> test environment into production? So you need a reliable way to apply >> changes to both environments. >> >>> >>> >>> It doesn’t solve issues with Dockerized deployments and it doesn’t solve >>> the issue where services have to be located at runtime >> >> >> What are the issues it doesn't solve? >> >>> >>> >>> Scott Rossillo >>> Smartling | Senior Software Engineer >>> srossillo(a)smartling.com >>> >>> On May 26, 2016, at 2:27 AM, Stian Thorgersen <sthorger(a)redhat.com&gt; >>> wrote: >>> >>> >>> >>> On 26 May 2016 at 02:15, Jesse Chahal <jessec(a)stytch.com&gt; wrote: >>>> >>>> @Stian >>>> The approach described sounds similar to liquibase to me but with json >>>> and specific to keycloak. I feel like a lot of possible bugs could >>>> arise from this approach or at least quite a few feature requests. >>>> Would each json file only contain a single change? Would order matter >>>> in how they get applied if you put a bunch of json files in this >>>> directory at once? Can the same file be applied multiple times? These >>>> are the kind of issues I would expect to come up with this type of >>>> change management system. When I mentioned write our own tool/script >>>> to do it I was kind of thinking of a writing a liquibase like system >>>> that calls keycloak's rest api. >>> >>> >>> We haven't figured out all the details, but what you are proposing sounds >>> better. A single document that lists all changes, that can also import other >>> files, sorts out the ordering and we could add same type of ids as Liquibase >>> does to changesets. >>> >>> You could write it to use the rest api, then use a separate db to store >>> what changes have been applied, but would be better if Keycloak deals with >>> loading the changes directly as it can write to the db what changes have >>> been applied. >>> >>> One big issue is what happens if manual changes are done through the >>> admin console. One though (although probably very tricky to get right) is >>> that changes done through the admin console is added to the changeset. >>> >>>> >>>> >>>> @ Scott >>>> If I would compare the solution you mentioned to one of the options I >>>> listed in my original question "I've also considered writing my own >>>> updater tool using a scripting language (python/ruby) that calls >>>> keycloak's rest api." The worrying thing to me is that there is >>>> another piece of code that needs to maintained by our company and >>>> requires quite a bit of knowledge of keycloak's rest api. There would >>>> probably need to be some serious thought put into the architecture of >>>> the tool as well. Without a doubt it does provide the most control. We >>>> also live by a different methodology in regards to updating production >>>> clusters. From our perspective it is more of an issue to update >>>> manually as it becomes much easier to miss a step or in someway screw >>>> up if steps are performed manually. I'm not sure what the security >>>> implications would be from it occurring automatically, especially if >>>> during each step there is thorough testing (including from a security >>>> team). For our CI/CD pipeline our goal is to have it so every commit >>>> can automagically end up on production without human intervention. >>>> >>>> Currently we use a combination of an initial realm file to be included >>>> on startup and also use jq to modify the keycloak-server.json for new >>>> keycloak clusters. We don't need to generate realm or client keys as >>>> it is included in the initial realm file. That doesn't work for >>>> existing systems backed by a database that cannot be thrown away. That >>>> kind of leave me with the original option (and hardest) of "write a >>>> proprietary liquibase like system built ontop of keycloaks rest api". >>>> This is a hard problem to solve >>> >>> >>> Why proprietary? If we can agree on a design we'll happily accept a >>> contribution and maintain it as well. >>> >>>> >>>> >>>> On Mon, May 23, 2016 at 1:49 PM, Anthony Fryer < anthony.fryer(a)gmail.com&gt; >>>> wrote: >>>> > Thanks, I'll check it out. >>>> > >>>> > >>>> > On 05:38, Tue, 24/05/2016 Scott Rossillo <srossillo(a)smartling.com&gt; >>>> > wrote: >>>> >> >>>> >> We use Jose4J[0] to create the keys and then jq[1] to modify the >>>> >> realm >>>> >> file. >>>> >> >>>> >> See the first line of code here for a super simple example of how to >>>> >> generate realm keys: >>>> >> https://bitbucket.org/b_c/jose4j/wiki/JWT%20Examples >>>> >> >>>> >> PS - this may be doable with Keycloak but Jose4J is very lightweight >>>> >> for >>>> >> writing a simple script on a CI server. >>>> >> >>>> >> [0]: https://bitbucket.org/b_c/jose4j >>>> >> [1]: https://stedolan.github.io/jq/ >>>> >> >>>> >> >>>> >> Scott Rossillo >>>> >> Smartling | Senior Software Engineer >>>> >> srossillo(a)smartling.com >>>> >> >>>> >> On May 21, 2016, at 10:20 PM, Anthony Fryer < anthony.fryer(a)gmail.com&gt; >>>> >> wrote: >>>> >> >>>> >> Hi Scott, >>>> >> >>>> >> How do you generate the realm keys when creating the new keycloak dev >>>> >> instances? Do you use a keycloak api or some other way? I'm >>>> >> interested in >>>> >> having a standard realm template that is used to create new realms >>>> >> but would >>>> >> need to change the realm keys when importing this template into >>>> >> keycloak. >>>> >> >>>> >> Cheers, >>>> >> >>>> >> Anthony >>>> >> >>>> >> On Sat, May 21, 2016 at 3:43 AM, Scott Rossillo >>>> >> <srossillo(a)smartling.com&gt; >>>> >> wrote: >>>> >>> >>>> >>> We’re using Keycloak on production, stage/QA, development >>>> >>> environments >>>> >>> and every developer’s workstation / laptop. >>>> >>> >>>> >>> While there will always be differing options on how to successfully >>>> >>> do >>>> >>> change management, we’ve found a very effective method for handling >>>> >>> Keycloak >>>> >>> provisioning in all environments so that developers don’t need to >>>> >>> mess >>>> >>> around with. We’re a continuous integration / deployment shop using >>>> >>> micro >>>> >>> services and everything has to “just work” … I’ll give an overview >>>> >>> of our >>>> >>> process here but please keep in mind a few things: >>>> >>> >>>> >>> 1. This approach works for us, I’m not saying it’s the best way >>>> >>> 2. We do _not_ allow production config changes to be automated due >>>> >>> to >>>> >>> security implications >>>> >>> 3. We're very opinionated in our approach to configuration >>>> >>> management and >>>> >>> we don’t ever modify 3rd party software databases directly. We >>>> >>> always use >>>> >>> APIs. >>>> >>> >>>> >>> We deploy Keycloak to all environments using Docker images. On >>>> >>> developer >>>> >>> workstations we use Docker Compose to orchestrate bringing up all >>>> >>> services a >>>> >>> developer may need, including Keycloak. >>>> >>> >>>> >>> We have 4 docker images for Keycloak: >>>> >>> - Keycloak Base >>>> >>> \- Keycloak HA >>>> >>> \- Keycloak Dev >>>> >>> - Keycloak config manager* >>>> >>> >>>> >>> The base image includes all customizations necessary to bring up a >>>> >>> Keycloak instance configured with our modules and themes installed. >>>> >>> The HA instance builds off base and configures Keycloak to run as a >>>> >>> cluster node. This is used on stage and prod. >>>> >>> The dev instance builds off base and includes our realm file. On >>>> >>> startup, >>>> >>> this instance loads our realm configuration if it’s not already >>>> >>> loaded. >>>> >>> >>>> >>> All docker images are built and published by the CI server and >>>> >>> Keycloak >>>> >>> HA can be deployed to stage and prod after a clean CI build. >>>> >>> >>>> >>> Developers are free to add clients for testing, do whatever they >>>> >>> want, >>>> >>> etc. to their running dev instance. If they want to get back to our >>>> >>> stock >>>> >>> build, they pull the latest Docker image from our private Docker >>>> >>> repo and >>>> >>> restart it. >>>> >>> >>>> >>> Adding clients to stage and prod requires approval and is done by a >>>> >>> hand. >>>> >>> This is for security reasons. Once a configuration change is >>>> >>> detected on >>>> >>> stage - say a client is added - our CI server exports the realm from >>>> >>> stage, >>>> >>> changes the realm keys, and creates a new Keycloak Dev instance with >>>> >>> the >>>> >>> updated realm file. >>>> >>> >>>> >>> *A word about configuration management: >>>> >>> >>>> >>> Obviously, the realm file we generate knows the URLs of staging >>>> >>> services, >>>> >>> not local or development environment URLs. To overcome this we >>>> >>> introduced >>>> >>> another Docker based service called the Keycloak configuration >>>> >>> manger. It >>>> >>> runs on development environments and workstations. It’s responsible >>>> >>> for >>>> >>> discovering running services and updating Keycloak via its admin >>>> >>> endpoints >>>> >>> to reflect the proper configuration for the given environment. >>>> >>> >>>> >>> That’s it. The whole process is automated with the exception of >>>> >>> configuration changes to stage and prod which require a security >>>> >>> review. >>>> >>> >>>> >>> Hope this helps. Let me know if you’d like me to elaborate on >>>> >>> anything. >>>> >>> >>>> >>> Best, >>>> >>> Scott >>>> >>> >>>> >>> Scott Rossillo >>>> >>> Smartling | Senior Software Engineer >>>> >>> srossillo(a)smartling.com >>>> >>> >>>> >>> On May 20, 2016, at 1:46 AM, Stian Thorgersen < sthorger(a)redhat.com&gt; >>>> >>> wrote: >>>> >>> >>>> >>> Firstly, just wanted to highlight that core Keycloak team are devs, >>>> >>> not >>>> >>> sysadmins/ops guys, so we have limited experience in continuous >>>> >>> delivery and >>>> >>> maintenance of real production systems. Hence, we'd love input from >>>> >>> the >>>> >>> community on this. >>>> >>> >>>> >>> As it stands we don't really have a proper solution. I believe the >>>> >>> best >>>> >>> you can do at the moment is either using import feature, partial >>>> >>> import or >>>> >>> admin rest endpoints. Import is not going to work IMO as it requires >>>> >>> re-creating the whole realm. Partial import may work, but would work >>>> >>> best >>>> >>> for new resources rather than modifying existing resources as it >>>> >>> does a >>>> >>> delete/create operation rather than attempt to modify. With the >>>> >>> admin rest >>>> >>> endpoints you'd get the best control of what's going on, but >>>> >>> obviously that >>>> >>> leaves a fair amount of the work. >>>> >>> >>>> >>> In the future we have an idea of introducing an "import directory" >>>> >>> it >>>> >>> would be possible to drop json files in here that would add, modify >>>> >>> or >>>> >>> delete resources (realms, clients, roles, users, whatever). This >>>> >>> would allow >>>> >>> dropping json files before the server starts and the server would >>>> >>> then >>>> >>> import on startup. It would also be possible to do this at runtime >>>> >>> and new >>>> >>> files would be detected at runtime. Finally, we also had an idea of >>>> >>> an >>>> >>> offline mode to run import of this (it would basically start the >>>> >>> server >>>> >>> without http listener, import files, then stop, so it could be used >>>> >>> in a >>>> >>> script/tool). Import is probably not the best name for it, as it >>>> >>> would >>>> >>> support modify and delete as well as "importing" new things. >>>> >>> >>>> >>> On 19 May 2016 at 19:53, Jesse Chahal <jessec(a)stytch.com&gt; wrote: >>>> >>>> >>>> >>>> Following some of the best practices for continuous Integration and >>>> >>>> continuous delivery there needs to be environments for build, test, >>>> >>>> and production. This would mean that following these practices >>>> >>>> would >>>> >>>> require you to have multiple versions of keycloak at different >>>> >>>> stages >>>> >>>> of development cycle. Some of these environments might not have >>>> >>>> important persistent data while others might. In order to have >>>> >>>> builds >>>> >>>> transition from one environment to another there may be >>>> >>>> configuration >>>> >>>> changes required for a build to be valid. This is especially true >>>> >>>> when >>>> >>>> new services (openid clients) are being added or "default" >>>> >>>> accounts. >>>> >>>> I'm trying to come up with a scripted way of updating keycloak >>>> >>>> instances that are backed up by an RDMS. This may include adding >>>> >>>> new >>>> >>>> clients, adding new users, updating realm config, etc... Originally >>>> >>>> I >>>> >>>> was planning on simply exporting the realm config and importing it >>>> >>>> every time keycloak starts. If I enabled the OVERWRITE option I >>>> >>>> might >>>> >>>> overwrite things that I do not want overridden. This is especially >>>> >>>> true if there is some config that differ's based on whether it is a >>>> >>>> build, test, or production instance. If I don't enable it then it >>>> >>>> is >>>> >>>> only useful for new/blank keycloak environments. I considered using >>>> >>>> liquibase but since I do not have control of schema changes created >>>> >>>> by >>>> >>>> the keycloak team I might run into issues with my liquibase file >>>> >>>> not >>>> >>>> being valid after a migration/liquibase update by the keycloak team >>>> >>>> as >>>> >>>> my liquibase file would run after keycloak's does. There might also >>>> >>>> be >>>> >>>> some other unknown issues our liquibase changes conflicting somehow >>>> >>>> with keycloak's liquibase changes. I've also considered writing my >>>> >>>> own >>>> >>>> updater tool using a scripting language (python/ruby) that calls >>>> >>>> keycloak's rest api. The issues with this mechanism is it feels >>>> >>>> like I >>>> >>>> am recreating the wheel as well as not being able to find good >>>> >>>> documentation on keycloak's openid endpoints/url's used for >>>> >>>> different >>>> >>>> oauth2 flows. Even if I did find this documentation it would also >>>> >>>> require me to find a good openid client for the scripting language. >>>> >>>> This doesn't matter for our normal clients as they simply use the >>>> >>>> keycloak subsystems and adapters instead. I've also looked at >>>> >>>> commonly >>>> >>>> used server configuration software such as chef, puppet, and >>>> >>>> ansible. >>>> >>>> I don't see a good solution using any of those tools yet either. >>>> >>>> What >>>> >>>> have other people done for cases like this? Please don't tell me >>>> >>>> there >>>> >>>> is someone who is doing this all manually because that doesn't work >>>> >>>> in >>>> >>>> modern software development. >>>> >>>> >>>> >>>> - doesn't accidentally delete users >>>> >>>> - doesn't accidentally delete clients >>>> >>>> - doesn't invalidate sessions (optional) >>>> >>>> - works to bring up new, correctly configured, keycloak instances >>>> >>>> - handles applying updates to existing keycloak instances >>>> >>>> - can handle minor differences between keycloak instances (build, >>>> >>>> test, production) when updating >>>> >>>> - preferably can work well in rolling deployment scenario's. >>>> >>>> -- I hope the keycloak team is taking these into consideration when >>>> >>>> doing database migration between 1-2 releases. It would be nice if >>>> >>>> they set some specific rules for rolling updates between versions >>>> >>>> (aka >>>> >>>> backwards breaking changes) >>>> >>>> _______________________________________________ >>>> >>>> keycloak-user mailing list >>>> >>>> keycloak-user(a)lists.jboss.org >>>> >>>> https://lists.jboss.org/mailman/listinfo/keycloak-user >>>> >>> >>>> >>> >>>> >>> _______________________________________________ >>>> >>> keycloak-user mailing list >>>> >>> keycloak-user(a)lists.jboss.org >>>> >>> https://lists.jboss.org/mailman/listinfo/keycloak-user >>>> >>> >>>> >>> >>>> >>> >>>> >>> _______________________________________________ >>>> >>> keycloak-user mailing list >>>> >>> keycloak-user(a)lists.jboss.org >>>> >>> https://lists.jboss.org/mailman/listinfo/keycloak-user >>>> >> >>>> >> >>>> >> >>>> > >>> >>> >>> >> >> >