<div dir="ltr">Adding list back..<div><br></div><div>I don't see much value in a solution that doesn't also consider changes done directly through admin console and/or admin endpoints. A proper solution would use something along the lines of Liquibase/Git to have all changes versioned and applied serially. That way they can be reproduced fully.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 10 June 2016 at 21:03, Jesse Chahal <span dir="ltr"><<a href="mailto:jessec@stytch.com" target="_blank">jessec@stytch.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I've been thinking about this problem for awhile and so far the<br>
solutions that I come up with all require that keycloak keeps tracks<br>
of changes in a database table (exactly how it works for liquibase).<br>
The GUI has a partial import feature. I haven't used it too<br>
extensively but I believe it probably does some sort of JSONtoPOJO<br>
serialization in order to figure out what the partial update it needs<br>
to be doing. Maybe we could add unique id identifiers to the<br>
existing/exported JSON files and have keycloaks import features<br>
determine whether the JSON file had already been applied or not. If<br>
there is a rest api for this as well then building an external cli or<br>
GUI tool would be much more feasible. Scott's solution requires either<br>
the external app to know the state of keycloak or keycloak's state to<br>
be blank. Its the best that could have been done with keycloak as it<br>
is now. Anyone have any comments regarding this possible solution?<br>
<div class="HOEnZb"><div class="h5"><br>
On Thu, May 26, 2016 at 11:19 PM, Stian Thorgersen <<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a>> wrote:<br>
> Can you give me some examples of issues around Dockerized deployments and<br>
> services that are located at runtime (do you mean services that are<br>
> provisioned at runtime?)?<br>
><br>
> On 26 May 2016 at 19:47, Scott Rossillo <<a href="mailto:srossillo@smartling.com">srossillo@smartling.com</a>> wrote:<br>
>><br>
>> Stian, that’s fair, it does solve the OP's CI/CD problem when moving in<br>
>> the dev -> stage -> prod direction.<br>
>><br>
>> Scott Rossillo<br>
>> Smartling | Senior Software Engineer<br>
>> <a href="mailto:srossillo@smartling.com">srossillo@smartling.com</a><br>
>><br>
>> On May 26, 2016, at 1:41 PM, Stian Thorgersen <<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a>> wrote:<br>
>><br>
>><br>
>><br>
>> On 26 May 2016 at 19:11, Scott Rossillo <<a href="mailto:srossillo@smartling.com">srossillo@smartling.com</a>> wrote:<br>
>>><br>
>>> I guess it’s a matter of requirements, but with micro service<br>
>>> architectures there’s usually some sort of discovery mechanism required to<br>
>>> locale services at runtime. Netflix offers Eureka and then there’s etcd from<br>
>>> CoreOS that’s being used by Kubernetes. My point is that even if Keycloak<br>
>>> devs build some sort of way of picking up changes from the filesystem on<br>
>>> startup, that doesn’t solve all use cases.<br>
>><br>
>><br>
>> The problem is continuous integration right, and pushing changes from a<br>
>> test environment into production? So you need a reliable way to apply<br>
>> changes to both environments.<br>
>><br>
>>><br>
>>><br>
>>> It doesn’t solve issues with Dockerized deployments and it doesn’t solve<br>
>>> the issue where services have to be located at runtime<br>
>><br>
>><br>
>> What are the issues it doesn't solve?<br>
>><br>
>>><br>
>>><br>
>>> Scott Rossillo<br>
>>> Smartling | Senior Software Engineer<br>
>>> <a href="mailto:srossillo@smartling.com">srossillo@smartling.com</a><br>
>>><br>
>>> On May 26, 2016, at 2:27 AM, Stian Thorgersen <<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a>><br>
>>> wrote:<br>
>>><br>
>>><br>
>>><br>
>>> On 26 May 2016 at 02:15, Jesse Chahal <<a href="mailto:jessec@stytch.com">jessec@stytch.com</a>> wrote:<br>
>>>><br>
>>>> @Stian<br>
>>>> The approach described sounds similar to liquibase to me but with json<br>
>>>> and specific to keycloak. I feel like a lot of possible bugs could<br>
>>>> arise from this approach or at least quite a few feature requests.<br>
>>>> Would each json file only contain a single change? Would order matter<br>
>>>> in how they get applied if you put a bunch of json files in this<br>
>>>> directory at once? Can the same file be applied multiple times? These<br>
>>>> are the kind of issues I would expect to come up with this type of<br>
>>>> change management system. When I mentioned write our own tool/script<br>
>>>> to do it I was kind of thinking of a writing a liquibase like system<br>
>>>> that calls keycloak's rest api.<br>
>>><br>
>>><br>
>>> We haven't figured out all the details, but what you are proposing sounds<br>
>>> better. A single document that lists all changes, that can also import other<br>
>>> files, sorts out the ordering and we could add same type of ids as Liquibase<br>
>>> does to changesets.<br>
>>><br>
>>> You could write it to use the rest api, then use a separate db to store<br>
>>> what changes have been applied, but would be better if Keycloak deals with<br>
>>> loading the changes directly as it can write to the db what changes have<br>
>>> been applied.<br>
>>><br>
>>> One big issue is what happens if manual changes are done through the<br>
>>> admin console. One though (although probably very tricky to get right) is<br>
>>> that changes done through the admin console is added to the changeset.<br>
>>><br>
>>>><br>
>>>><br>
>>>> @ Scott<br>
>>>> If I would compare the solution you mentioned to one of the options I<br>
>>>> listed in my original question "I've also considered writing my own<br>
>>>> updater tool using a scripting language (python/ruby) that calls<br>
>>>> keycloak's rest api." The worrying thing to me is that there is<br>
>>>> another piece of code that needs to maintained by our company and<br>
>>>> requires quite a bit of knowledge of keycloak's rest api. There would<br>
>>>> probably need to be some serious thought put into the architecture of<br>
>>>> the tool as well. Without a doubt it does provide the most control. We<br>
>>>> also live by a different methodology in regards to updating production<br>
>>>> clusters. From our perspective it is more of an issue to update<br>
>>>> manually as it becomes much easier to miss a step or in someway screw<br>
>>>> up if steps are performed manually. I'm not sure what the security<br>
>>>> implications would be from it occurring automatically, especially if<br>
>>>> during each step there is thorough testing (including from a security<br>
>>>> team). For our CI/CD pipeline our goal is to have it so every commit<br>
>>>> can automagically end up on production without human intervention.<br>
>>>><br>
>>>> Currently we use a combination of an initial realm file to be included<br>
>>>> on startup and also use jq to modify the keycloak-server.json for new<br>
>>>> keycloak clusters. We don't need to generate realm or client keys as<br>
>>>> it is included in the initial realm file. That doesn't work for<br>
>>>> existing systems backed by a database that cannot be thrown away. That<br>
>>>> kind of leave me with the original option (and hardest) of "write a<br>
>>>> proprietary liquibase like system built ontop of keycloaks rest api".<br>
>>>> This is a hard problem to solve<br>
>>><br>
>>><br>
>>> Why proprietary? If we can agree on a design we'll happily accept a<br>
>>> contribution and maintain it as well.<br>
>>><br>
>>>><br>
>>>><br>
>>>> On Mon, May 23, 2016 at 1:49 PM, Anthony Fryer <<a href="mailto:anthony.fryer@gmail.com">anthony.fryer@gmail.com</a>><br>
>>>> wrote:<br>
>>>> > Thanks, I'll check it out.<br>
>>>> ><br>
>>>> ><br>
>>>> > On 05:38, Tue, 24/05/2016 Scott Rossillo <<a href="mailto:srossillo@smartling.com">srossillo@smartling.com</a>><br>
>>>> > wrote:<br>
>>>> >><br>
>>>> >> We use Jose4J[0] to create the keys and then jq[1] to modify the<br>
>>>> >> realm<br>
>>>> >> file.<br>
>>>> >><br>
>>>> >> See the first line of code here for a super simple example of how to<br>
>>>> >> generate realm keys:<br>
>>>> >> <a href="https://bitbucket.org/b_c/jose4j/wiki/JWT%20Examples" rel="noreferrer" target="_blank">https://bitbucket.org/b_c/jose4j/wiki/JWT%20Examples</a><br>
>>>> >><br>
>>>> >> PS - this may be doable with Keycloak but Jose4J is very lightweight<br>
>>>> >> for<br>
>>>> >> writing a simple script on a CI server.<br>
>>>> >><br>
>>>> >> [0]: <a href="https://bitbucket.org/b_c/jose4j" rel="noreferrer" target="_blank">https://bitbucket.org/b_c/jose4j</a><br>
>>>> >> [1]: <a href="https://stedolan.github.io/jq/" rel="noreferrer" target="_blank">https://stedolan.github.io/jq/</a><br>
>>>> >><br>
>>>> >><br>
>>>> >> Scott Rossillo<br>
>>>> >> Smartling | Senior Software Engineer<br>
>>>> >> <a href="mailto:srossillo@smartling.com">srossillo@smartling.com</a><br>
>>>> >><br>
>>>> >> On May 21, 2016, at 10:20 PM, Anthony Fryer <<a href="mailto:anthony.fryer@gmail.com">anthony.fryer@gmail.com</a>><br>
>>>> >> wrote:<br>
>>>> >><br>
>>>> >> Hi Scott,<br>
>>>> >><br>
>>>> >> How do you generate the realm keys when creating the new keycloak dev<br>
>>>> >> instances? Do you use a keycloak api or some other way? I'm<br>
>>>> >> interested in<br>
>>>> >> having a standard realm template that is used to create new realms<br>
>>>> >> but would<br>
>>>> >> need to change the realm keys when importing this template into<br>
>>>> >> keycloak.<br>
>>>> >><br>
>>>> >> Cheers,<br>
>>>> >><br>
>>>> >> Anthony<br>
>>>> >><br>
>>>> >> On Sat, May 21, 2016 at 3:43 AM, Scott Rossillo<br>
>>>> >> <<a href="mailto:srossillo@smartling.com">srossillo@smartling.com</a>><br>
>>>> >> wrote:<br>
>>>> >>><br>
>>>> >>> We’re using Keycloak on production, stage/QA, development<br>
>>>> >>> environments<br>
>>>> >>> and every developer’s workstation / laptop.<br>
>>>> >>><br>
>>>> >>> While there will always be differing options on how to successfully<br>
>>>> >>> do<br>
>>>> >>> change management, we’ve found a very effective method for handling<br>
>>>> >>> Keycloak<br>
>>>> >>> provisioning in all environments so that developers don’t need to<br>
>>>> >>> mess<br>
>>>> >>> around with. We’re a continuous integration / deployment shop using<br>
>>>> >>> micro<br>
>>>> >>> services and everything has to “just work” … I’ll give an overview<br>
>>>> >>> of our<br>
>>>> >>> process here but please keep in mind a few things:<br>
>>>> >>><br>
>>>> >>> 1. This approach works for us, I’m not saying it’s the best way<br>
>>>> >>> 2. We do _not_ allow production config changes to be automated due<br>
>>>> >>> to<br>
>>>> >>> security implications<br>
>>>> >>> 3. We're very opinionated in our approach to configuration<br>
>>>> >>> management and<br>
>>>> >>> we don’t ever modify 3rd party software databases directly. We<br>
>>>> >>> always use<br>
>>>> >>> APIs.<br>
>>>> >>><br>
>>>> >>> We deploy Keycloak to all environments using Docker images. On<br>
>>>> >>> developer<br>
>>>> >>> workstations we use Docker Compose to orchestrate bringing up all<br>
>>>> >>> services a<br>
>>>> >>> developer may need, including Keycloak.<br>
>>>> >>><br>
>>>> >>> We have 4 docker images for Keycloak:<br>
>>>> >>> - Keycloak Base<br>
>>>> >>> \- Keycloak HA<br>
>>>> >>> \- Keycloak Dev<br>
>>>> >>> - Keycloak config manager*<br>
>>>> >>><br>
>>>> >>> The base image includes all customizations necessary to bring up a<br>
>>>> >>> Keycloak instance configured with our modules and themes installed.<br>
>>>> >>> The HA instance builds off base and configures Keycloak to run as a<br>
>>>> >>> cluster node. This is used on stage and prod.<br>
>>>> >>> The dev instance builds off base and includes our realm file. On<br>
>>>> >>> startup,<br>
>>>> >>> this instance loads our realm configuration if it’s not already<br>
>>>> >>> loaded.<br>
>>>> >>><br>
>>>> >>> All docker images are built and published by the CI server and<br>
>>>> >>> Keycloak<br>
>>>> >>> HA can be deployed to stage and prod after a clean CI build.<br>
>>>> >>><br>
>>>> >>> Developers are free to add clients for testing, do whatever they<br>
>>>> >>> want,<br>
>>>> >>> etc. to their running dev instance. If they want to get back to our<br>
>>>> >>> stock<br>
>>>> >>> build, they pull the latest Docker image from our private Docker<br>
>>>> >>> repo and<br>
>>>> >>> restart it.<br>
>>>> >>><br>
>>>> >>> Adding clients to stage and prod requires approval and is done by a<br>
>>>> >>> hand.<br>
>>>> >>> This is for security reasons. Once a configuration change is<br>
>>>> >>> detected on<br>
>>>> >>> stage - say a client is added - our CI server exports the realm from<br>
>>>> >>> stage,<br>
>>>> >>> changes the realm keys, and creates a new Keycloak Dev instance with<br>
>>>> >>> the<br>
>>>> >>> updated realm file.<br>
>>>> >>><br>
>>>> >>> *A word about configuration management:<br>
>>>> >>><br>
>>>> >>> Obviously, the realm file we generate knows the URLs of staging<br>
>>>> >>> services,<br>
>>>> >>> not local or development environment URLs. To overcome this we<br>
>>>> >>> introduced<br>
>>>> >>> another Docker based service called the Keycloak configuration<br>
>>>> >>> manger. It<br>
>>>> >>> runs on development environments and workstations. It’s responsible<br>
>>>> >>> for<br>
>>>> >>> discovering running services and updating Keycloak via its admin<br>
>>>> >>> endpoints<br>
>>>> >>> to reflect the proper configuration for the given environment.<br>
>>>> >>><br>
>>>> >>> That’s it. The whole process is automated with the exception of<br>
>>>> >>> configuration changes to stage and prod which require a security<br>
>>>> >>> review.<br>
>>>> >>><br>
>>>> >>> Hope this helps. Let me know if you’d like me to elaborate on<br>
>>>> >>> anything.<br>
>>>> >>><br>
>>>> >>> Best,<br>
>>>> >>> Scott<br>
>>>> >>><br>
>>>> >>> Scott Rossillo<br>
>>>> >>> Smartling | Senior Software Engineer<br>
>>>> >>> <a href="mailto:srossillo@smartling.com">srossillo@smartling.com</a><br>
>>>> >>><br>
>>>> >>> On May 20, 2016, at 1:46 AM, Stian Thorgersen <<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a>><br>
>>>> >>> wrote:<br>
>>>> >>><br>
>>>> >>> Firstly, just wanted to highlight that core Keycloak team are devs,<br>
>>>> >>> not<br>
>>>> >>> sysadmins/ops guys, so we have limited experience in continuous<br>
>>>> >>> delivery and<br>
>>>> >>> maintenance of real production systems. Hence, we'd love input from<br>
>>>> >>> the<br>
>>>> >>> community on this.<br>
>>>> >>><br>
>>>> >>> As it stands we don't really have a proper solution. I believe the<br>
>>>> >>> best<br>
>>>> >>> you can do at the moment is either using import feature, partial<br>
>>>> >>> import or<br>
>>>> >>> admin rest endpoints. Import is not going to work IMO as it requires<br>
>>>> >>> re-creating the whole realm. Partial import may work, but would work<br>
>>>> >>> best<br>
>>>> >>> for new resources rather than modifying existing resources as it<br>
>>>> >>> does a<br>
>>>> >>> delete/create operation rather than attempt to modify. With the<br>
>>>> >>> admin rest<br>
>>>> >>> endpoints you'd get the best control of what's going on, but<br>
>>>> >>> obviously that<br>
>>>> >>> leaves a fair amount of the work.<br>
>>>> >>><br>
>>>> >>> In the future we have an idea of introducing an "import directory"<br>
>>>> >>> it<br>
>>>> >>> would be possible to drop json files in here that would add, modify<br>
>>>> >>> or<br>
>>>> >>> delete resources (realms, clients, roles, users, whatever). This<br>
>>>> >>> would allow<br>
>>>> >>> dropping json files before the server starts and the server would<br>
>>>> >>> then<br>
>>>> >>> import on startup. It would also be possible to do this at runtime<br>
>>>> >>> and new<br>
>>>> >>> files would be detected at runtime. Finally, we also had an idea of<br>
>>>> >>> an<br>
>>>> >>> offline mode to run import of this (it would basically start the<br>
>>>> >>> server<br>
>>>> >>> without http listener, import files, then stop, so it could be used<br>
>>>> >>> in a<br>
>>>> >>> script/tool). Import is probably not the best name for it, as it<br>
>>>> >>> would<br>
>>>> >>> support modify and delete as well as "importing" new things.<br>
>>>> >>><br>
>>>> >>> On 19 May 2016 at 19:53, Jesse Chahal <<a href="mailto:jessec@stytch.com">jessec@stytch.com</a>> wrote:<br>
>>>> >>>><br>
>>>> >>>> Following some of the best practices for continuous Integration and<br>
>>>> >>>> continuous delivery there needs to be environments for build, test,<br>
>>>> >>>> and production. This would mean that following these practices<br>
>>>> >>>> would<br>
>>>> >>>> require you to have multiple versions of keycloak at different<br>
>>>> >>>> stages<br>
>>>> >>>> of development cycle. Some of these environments might not have<br>
>>>> >>>> important persistent data while others might. In order to have<br>
>>>> >>>> builds<br>
>>>> >>>> transition from one environment to another there may be<br>
>>>> >>>> configuration<br>
>>>> >>>> changes required for a build to be valid. This is especially true<br>
>>>> >>>> when<br>
>>>> >>>> new services (openid clients) are being added or "default"<br>
>>>> >>>> accounts.<br>
>>>> >>>> I'm trying to come up with a scripted way of updating keycloak<br>
>>>> >>>> instances that are backed up by an RDMS. This may include adding<br>
>>>> >>>> new<br>
>>>> >>>> clients, adding new users, updating realm config, etc... Originally<br>
>>>> >>>> I<br>
>>>> >>>> was planning on simply exporting the realm config and importing it<br>
>>>> >>>> every time keycloak starts. If I enabled the OVERWRITE option I<br>
>>>> >>>> might<br>
>>>> >>>> overwrite things that I do not want overridden. This is especially<br>
>>>> >>>> true if there is some config that differ's based on whether it is a<br>
>>>> >>>> build, test, or production instance. If I don't enable it then it<br>
>>>> >>>> is<br>
>>>> >>>> only useful for new/blank keycloak environments. I considered using<br>
>>>> >>>> liquibase but since I do not have control of schema changes created<br>
>>>> >>>> by<br>
>>>> >>>> the keycloak team I might run into issues with my liquibase file<br>
>>>> >>>> not<br>
>>>> >>>> being valid after a migration/liquibase update by the keycloak team<br>
>>>> >>>> as<br>
>>>> >>>> my liquibase file would run after keycloak's does. There might also<br>
>>>> >>>> be<br>
>>>> >>>> some other unknown issues our liquibase changes conflicting somehow<br>
>>>> >>>> with keycloak's liquibase changes. I've also considered writing my<br>
>>>> >>>> own<br>
>>>> >>>> updater tool using a scripting language (python/ruby) that calls<br>
>>>> >>>> keycloak's rest api. The issues with this mechanism is it feels<br>
>>>> >>>> like I<br>
>>>> >>>> am recreating the wheel as well as not being able to find good<br>
>>>> >>>> documentation on keycloak's openid endpoints/url's used for<br>
>>>> >>>> different<br>
>>>> >>>> oauth2 flows. Even if I did find this documentation it would also<br>
>>>> >>>> require me to find a good openid client for the scripting language.<br>
>>>> >>>> This doesn't matter for our normal clients as they simply use the<br>
>>>> >>>> keycloak subsystems and adapters instead. I've also looked at<br>
>>>> >>>> commonly<br>
>>>> >>>> used server configuration software such as chef, puppet, and<br>
>>>> >>>> ansible.<br>
>>>> >>>> I don't see a good solution using any of those tools yet either.<br>
>>>> >>>> What<br>
>>>> >>>> have other people done for cases like this? Please don't tell me<br>
>>>> >>>> there<br>
>>>> >>>> is someone who is doing this all manually because that doesn't work<br>
>>>> >>>> in<br>
>>>> >>>> modern software development.<br>
>>>> >>>><br>
>>>> >>>> - doesn't accidentally delete users<br>
>>>> >>>> - doesn't accidentally delete clients<br>
>>>> >>>> - doesn't invalidate sessions (optional)<br>
>>>> >>>> - works to bring up new, correctly configured, keycloak instances<br>
>>>> >>>> - handles applying updates to existing keycloak instances<br>
>>>> >>>> - can handle minor differences between keycloak instances (build,<br>
>>>> >>>> test, production) when updating<br>
>>>> >>>> - preferably can work well in rolling deployment scenario's.<br>
>>>> >>>> -- I hope the keycloak team is taking these into consideration when<br>
>>>> >>>> doing database migration between 1-2 releases. It would be nice if<br>
>>>> >>>> they set some specific rules for rolling updates between versions<br>
>>>> >>>> (aka<br>
>>>> >>>> backwards breaking changes)<br>
>>>> >>>> _______________________________________________<br>
>>>> >>>> keycloak-user mailing list<br>
>>>> >>>> <a href="mailto:keycloak-user@lists.jboss.org">keycloak-user@lists.jboss.org</a><br>
>>>> >>>> <a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a><br>
>>>> >>><br>
>>>> >>><br>
>>>> >>> _______________________________________________<br>
>>>> >>> keycloak-user mailing list<br>
>>>> >>> <a href="mailto:keycloak-user@lists.jboss.org">keycloak-user@lists.jboss.org</a><br>
>>>> >>> <a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a><br>
>>>> >>><br>
>>>> >>><br>
>>>> >>><br>
>>>> >>> _______________________________________________<br>
>>>> >>> keycloak-user mailing list<br>
>>>> >>> <a href="mailto:keycloak-user@lists.jboss.org">keycloak-user@lists.jboss.org</a><br>
>>>> >>> <a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a><br>
>>>> >><br>
>>>> >><br>
>>>> >><br>
>>>> ><br>
>>><br>
>>><br>
>>><br>
>><br>
>><br>
><br>
</div></div></blockquote></div><br></div>