[keycloak-dev] infinispan as a storage mechanism

Wed Feb 28 05:56:42 EST 2018

Another thing that might provide a better experience is run adapter tests
using Undertow instead of booting a Wildfly app server.

On Wed, Feb 28, 2018 at 6:23 AM, Marek Posolda <mposolda at redhat.com> wrote:

> I am also personally not concerned about running full testsuite through
> "mvn clean install", however I agree will be good to improve startup
> when you run single test from IDE.
>
> We already use in-memory H2 by default on embedded undertow. But maybe
> it helps if we skip liquibase at all and let Hibernate to create the
> schema when running testsuite with in-mem H2? The problem with Liquibase
> is, that there are more and more changelogs and more and more changes
> and we can't get rid old changelogs. So there are many things like the
> table is created by jpa-changelog-1.0.0, but then immediately dropped by
> jpa-changelog-1.5.0 etc. This is not so big issue for production
> environments when schema is created just once, but sucks during
> development IMO.
>
> Question is, how much time is spent on schema initialization and how
> much on other initialization things (EG. importing master realm and
> "test" realm etc)... Maybe to try some startup profiling will help...
>
> Marek
>
>
> On 27/02/18 21:08, Stian Thorgersen wrote:
> > It's not going to help with startup time, but perhaps one simple way to
> > make the testsuite run faster would be configuring the h2 dB as
> > in-mem/non-persisted.
> >
> > On 27 Feb 2018 7:11 pm, "Stian Thorgersen" <sthorger at redhat.com> wrote:
> >
> >>
> >> On 27 February 2018 at 15:02, Bill Burke <bburke at redhat.com> wrote:
> >>
> >>> This is something I've been doing off and on for awhile.  An hour here
> >>> or there.  Its a lot of monotonous work.
> >>>
> >>> Well worth it IMO if our build times drop drastically.  30 min build
> >>> is becoming burdensome.  If the console tests get turned on combined
> >>> with all the other tests that are added daily, I can see this turning
> >>> into 60 min by the end of the year.  Not to mention when I run a
> >>> single test from the IDE it takes like 10 seconds to start.  I'm just
> >>> sick and tired of it.
> >>>
> >> I can only see how it saves time on starting the Keycloak server, not
> for
> >> individual tests. That means you're only saving a few seconds each time
> the
> >> server is started, which is not many times for a full run. You'll
> probably
> >> end up shaving 1 min of the whole 30 min at best.
> >>
> >> Andy also mentioned that the Hibernate guys are working on performance.
> >> There is something weird about the Hibernate startup time in general.
> So if
> >> there's improvements there that 1 min you could shave would become even
> >> less.
> >>
> >> We're also working towards having the full testsuite ran on PRs, which
> >> means as a dev you would be better of picking and running only the tests
> >> you believe may be affected and let the CI run the complete set of jobs.
> >>
> >> I'd never bother running the admin console tests locally unless I've
> done
> >> some changes to the admin console. I also found that running the tests
> from
> >> the IDE run significantly quicker than through Maven. The difference
> there
> >> is down to something else than JPA as that's used in both cases.
> Personally
> >> I very rarely build the full distro or run the tests from Maven at all.
> I
> >> just let Travis do that, while I run tests through the IDE.
> >>
> >>
> >>> For migration, if you base your stored objects on Maps, you only have
> >>> to worry about cases where you're modifying objects that are
> >>> serialized.  These would need to be versioned as per java
> >>> serialization.  New things
> >>>
> >> I really don't want us to maintain two different persistence layers.
> It's
> >> a lot of duplicated effort. Becomes even worse when you start thinking
> >> about things like migration, cross-dc, rolling upgrades, etc..
> >>
> >> If you are suggesting to completely drop our JPA store altogether and
> use
> >> Infinispan with a cache store instead that could be an interesting
> option,
> >> but I have loads and loads of concerns around that.
> >>
> >>
> >>>
> >>> On Tue, Feb 27, 2018 at 1:53 AM, Stian Thorgersen <sthorger at redhat.com
> >
> >>> wrote:
> >>>> I'm really not convinced about this. Infinispan still needs to persist
> >>> the
> >>>> data. It still needs to handle migration whenever we change things.
> It's
> >>>> another layer to get working correctly, etc.. I think we have more
> >>> important
> >>>> work than to work on another data layer.
> >>>>
> >>>> On 26 February 2018 at 21:47, Bill Burke <bburke at redhat.com> wrote:
> >>>>> I'm thinking of this mostly for running our testsuite.  If you're not
> >>>>> developing on DB, much nicer if your test startup times is
> >>>>> milliseconds rather than 5-10 secs.
> >>>>>
> >>>>> For production, I'm thinking more of when people need lightweight
> >>>>> keycloak instances and are doing a lot of identity federation.  This
> >>>>> is really long term thoughts though.
> >>>>>
> >>>>> On Mon, Feb 26, 2018 at 2:56 PM, Pedro Igor Silva <psilva at redhat.com
> >
> >>>>> wrote:
> >>>>>> I think MongoDB will start supporting transactions very soon on v4
> >>> ....
> >>>>>> I'm not sure about running both app and database in the same VM
> >>> though.
> >>>>>> For
> >>>>>> dev purposes that is fine, but in real world scenarios you probably
> >>> want
> >>>>>> to
> >>>>>> avoid sharing resources (mem, cpu) with your DB. In your case,
> people
> >>>>>> will
> >>>>>> probably need JBoss DataGrid in production.
> >>>>>>
> >>>>>> On Mon, Feb 26, 2018 at 4:30 PM, Bill Burke <bburke at redhat.com>
> >>> wrote:
> >>>>>>> Other than MongoDB not supporting transactions or even sessions?
> >>> And
> >>>>>>> requiring a DB to be run in a separate VM?
> >>>>>>>
> >>>>>>> No not really :)
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Mon, Feb 26, 2018 at 12:54 PM, Pedro Igor Silva <
> >>> psilva at redhat.com>
> >>>>>>> wrote:
> >>>>>>>> Isn't this somewhat related with what we used to have with
> >>> MongoDB ?
> >>>>>>>> On Mon, Feb 26, 2018 at 2:35 PM, Bill Burke <bburke at redhat.com>
> >>>>>>>> wrote:
> >>>>>>>>> If we had a built-in, clusterable storage mechanism for Keycloak
> >>>>>>>>> using
> >>>>>>>>> Infinispan we would:
> >>>>>>>>> * Shorten build times drastically.  30 minutes and growing for me
> >>>>>>>>> for
> >>>>>>>>> JPA builds.  Liquibase + JPA startup takes 5-7 seconds on my box.
> >>>>>>>>> * Simpler startup.  No need to start a DB.
> >>>>>>>>> * Reduce memory footprint?  I think JPA is responsible for a lot
> >>> of
> >>>>>>>>> classes loaded.
> >>>>>>>>>
> >>>>>>>>> I've started some work on this in spare time.  I'd say I'd be
> >>> done
> >>>>>>>>> in
> >>>>>>>>> like 2 months considering the other work I have in queue.
> >>>>>>>>>
> >>>>>>>>> Looking at FineGrainAtomicMap as an implementation.  Should make
> >>> DB
> >>>>>>>>> migration simple and replication quicker.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Bill Burke
> >>>>>>>>> Red Hat
> >>>>>>>>> _______________________________________________
> >>>>>>>>> keycloak-dev mailing list
> >>>>>>>>> keycloak-dev at lists.jboss.org
> >>>>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> Bill Burke
> >>>>>>> Red Hat
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Bill Burke
> >>>>> Red Hat
> >>>>> _______________________________________________
> >>>>> keycloak-dev mailing list
> >>>>> keycloak-dev at lists.jboss.org
> >>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
> >>>>
> >>>
> >>>
> >>> --
> >>> Bill Burke
> >>> Red Hat
> >>>
> >>
> > _______________________________________________
> > keycloak-dev mailing list
> > keycloak-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
>
>
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>