[keycloak-dev] infinispan as a storage mechanism

Wed Feb 28 07:22:38 EST 2018

There are some adapter tests using undertow already [1] but they're 
excluded by default [2]. There is also ongoing work on refactor improve 
the adapter testing [3]

[1] 
https://github.com/keycloak/keycloak/tree/master/testsuite/integration-arquillian/tests/base/src/test/java/org/keycloak/testsuite/adapter/undertow/servlet

[2] 
https://github.com/keycloak/keycloak/blob/master/testsuite/integration-arquillian/tests/base/pom.xml#L47

[3] https://issues.jboss.org/browse/KEYCLOAK-6539

On 02/28/2018 11:56 AM, Pedro Igor Silva wrote:
> Another thing that might provide a better experience is run adapter tests
> using Undertow instead of booting a Wildfly app server.
>
> On Wed, Feb 28, 2018 at 6:23 AM, Marek Posolda <mposolda at redhat.com> wrote:
>
>> I am also personally not concerned about running full testsuite through
>> "mvn clean install", however I agree will be good to improve startup
>> when you run single test from IDE.
>>
>> We already use in-memory H2 by default on embedded undertow. But maybe
>> it helps if we skip liquibase at all and let Hibernate to create the
>> schema when running testsuite with in-mem H2? The problem with Liquibase
>> is, that there are more and more changelogs and more and more changes
>> and we can't get rid old changelogs. So there are many things like the
>> table is created by jpa-changelog-1.0.0, but then immediately dropped by
>> jpa-changelog-1.5.0 etc. This is not so big issue for production
>> environments when schema is created just once, but sucks during
>> development IMO.
>>
>> Question is, how much time is spent on schema initialization and how
>> much on other initialization things (EG. importing master realm and
>> "test" realm etc)... Maybe to try some startup profiling will help...
>>
>> Marek
>>
>>
>> On 27/02/18 21:08, Stian Thorgersen wrote:
>>> It's not going to help with startup time, but perhaps one simple way to
>>> make the testsuite run faster would be configuring the h2 dB as
>>> in-mem/non-persisted.
>>>
>>> On 27 Feb 2018 7:11 pm, "Stian Thorgersen" <sthorger at redhat.com> wrote:
>>>
>>>> On 27 February 2018 at 15:02, Bill Burke <bburke at redhat.com> wrote:
>>>>
>>>>> This is something I've been doing off and on for awhile.  An hour here
>>>>> or there.  Its a lot of monotonous work.
>>>>>
>>>>> Well worth it IMO if our build times drop drastically.  30 min build
>>>>> is becoming burdensome.  If the console tests get turned on combined
>>>>> with all the other tests that are added daily, I can see this turning
>>>>> into 60 min by the end of the year.  Not to mention when I run a
>>>>> single test from the IDE it takes like 10 seconds to start.  I'm just
>>>>> sick and tired of it.
>>>>>
>>>> I can only see how it saves time on starting the Keycloak server, not
>> for
>>>> individual tests. That means you're only saving a few seconds each time
>> the
>>>> server is started, which is not many times for a full run. You'll
>> probably
>>>> end up shaving 1 min of the whole 30 min at best.
>>>>
>>>> Andy also mentioned that the Hibernate guys are working on performance.
>>>> There is something weird about the Hibernate startup time in general.
>> So if
>>>> there's improvements there that 1 min you could shave would become even
>>>> less.
>>>>
>>>> We're also working towards having the full testsuite ran on PRs, which
>>>> means as a dev you would be better of picking and running only the tests
>>>> you believe may be affected and let the CI run the complete set of jobs.
>>>>
>>>> I'd never bother running the admin console tests locally unless I've
>> done
>>>> some changes to the admin console. I also found that running the tests
>> from
>>>> the IDE run significantly quicker than through Maven. The difference
>> there
>>>> is down to something else than JPA as that's used in both cases.
>> Personally
>>>> I very rarely build the full distro or run the tests from Maven at all.
>> I
>>>> just let Travis do that, while I run tests through the IDE.
>>>>
>>>>
>>>>> For migration, if you base your stored objects on Maps, you only have
>>>>> to worry about cases where you're modifying objects that are
>>>>> serialized.  These would need to be versioned as per java
>>>>> serialization.  New things
>>>>>
>>>> I really don't want us to maintain two different persistence layers.
>> It's
>>>> a lot of duplicated effort. Becomes even worse when you start thinking
>>>> about things like migration, cross-dc, rolling upgrades, etc..
>>>>
>>>> If you are suggesting to completely drop our JPA store altogether and
>> use
>>>> Infinispan with a cache store instead that could be an interesting
>> option,
>>>> but I have loads and loads of concerns around that.
>>>>
>>>>
>>>>> On Tue, Feb 27, 2018 at 1:53 AM, Stian Thorgersen <sthorger at redhat.com
>>>>> wrote:
>>>>>> I'm really not convinced about this. Infinispan still needs to persist
>>>>> the
>>>>>> data. It still needs to handle migration whenever we change things.
>> It's
>>>>>> another layer to get working correctly, etc.. I think we have more
>>>>> important
>>>>>> work than to work on another data layer.
>>>>>>
>>>>>> On 26 February 2018 at 21:47, Bill Burke <bburke at redhat.com> wrote:
>>>>>>> I'm thinking of this mostly for running our testsuite.  If you're not
>>>>>>> developing on DB, much nicer if your test startup times is
>>>>>>> milliseconds rather than 5-10 secs.
>>>>>>>
>>>>>>> For production, I'm thinking more of when people need lightweight
>>>>>>> keycloak instances and are doing a lot of identity federation.  This
>>>>>>> is really long term thoughts though.
>>>>>>>
>>>>>>> On Mon, Feb 26, 2018 at 2:56 PM, Pedro Igor Silva <psilva at redhat.com
>>>>>>> wrote:
>>>>>>>> I think MongoDB will start supporting transactions very soon on v4
>>>>> ....
>>>>>>>> I'm not sure about running both app and database in the same VM
>>>>> though.
>>>>>>>> For
>>>>>>>> dev purposes that is fine, but in real world scenarios you probably
>>>>> want
>>>>>>>> to
>>>>>>>> avoid sharing resources (mem, cpu) with your DB. In your case,
>> people
>>>>>>>> will
>>>>>>>> probably need JBoss DataGrid in production.
>>>>>>>>
>>>>>>>> On Mon, Feb 26, 2018 at 4:30 PM, Bill Burke <bburke at redhat.com>
>>>>> wrote:
>>>>>>>>> Other than MongoDB not supporting transactions or even sessions?
>>>>> And
>>>>>>>>> requiring a DB to be run in a separate VM?
>>>>>>>>>
>>>>>>>>> No not really :)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Feb 26, 2018 at 12:54 PM, Pedro Igor Silva <
>>>>> psilva at redhat.com>
>>>>>>>>> wrote:
>>>>>>>>>> Isn't this somewhat related with what we used to have with
>>>>> MongoDB ?
>>>>>>>>>> On Mon, Feb 26, 2018 at 2:35 PM, Bill Burke <bburke at redhat.com>
>>>>>>>>>> wrote:
>>>>>>>>>>> If we had a built-in, clusterable storage mechanism for Keycloak
>>>>>>>>>>> using
>>>>>>>>>>> Infinispan we would:
>>>>>>>>>>> * Shorten build times drastically.  30 minutes and growing for me
>>>>>>>>>>> for
>>>>>>>>>>> JPA builds.  Liquibase + JPA startup takes 5-7 seconds on my box.
>>>>>>>>>>> * Simpler startup.  No need to start a DB.
>>>>>>>>>>> * Reduce memory footprint?  I think JPA is responsible for a lot
>>>>> of
>>>>>>>>>>> classes loaded.
>>>>>>>>>>>
>>>>>>>>>>> I've started some work on this in spare time.  I'd say I'd be
>>>>> done
>>>>>>>>>>> in
>>>>>>>>>>> like 2 months considering the other work I have in queue.
>>>>>>>>>>>
>>>>>>>>>>> Looking at FineGrainAtomicMap as an implementation.  Should make
>>>>> DB
>>>>>>>>>>> migration simple and replication quicker.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Bill Burke
>>>>>>>>>>> Red Hat
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> keycloak-dev mailing list
>>>>>>>>>>> keycloak-dev at lists.jboss.org
>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Bill Burke
>>>>>>>>> Red Hat
>>>>>>>
>>>>>>> --
>>>>>>> Bill Burke
>>>>>>> Red Hat
>>>>>>> _______________________________________________
>>>>>>> keycloak-dev mailing list
>>>>>>> keycloak-dev at lists.jboss.org
>>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>>
>>>>> --
>>>>> Bill Burke
>>>>> Red Hat
>>>>>
>>> _______________________________________________
>>> keycloak-dev mailing list
>>> keycloak-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>
>> _______________________________________________
>> keycloak-dev mailing list
>> keycloak-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev