[keycloak-dev] Improving testsuite performance

[Bartosz Majsak]

Hi Marek,

are you running drone tests in parallel? It could be possible to speed it
up with just tweaking surfire. In my last project we were using 8-12
parallel drones and this gave us quite an advantage.

I would be happy to help.

Cheers,
Bartosz.

On Fri, Feb 17, 2017 at 10:12 PM, Marek Posolda <mposolda at redhat.com> wrote:

> ATM running tests in the new arquillian testsuite (
> testsuite/integration-arquillian ) is significantly slower then running
> tests in the old testsuite ( testsuite/integration ). It was quite
> visible in some tests, which were around 3 times slower after they were
> migrated from the old testsuite.
>
> We found that there are 2 major bottlenecks causing it:
> - There is re-import of all the realms before each test method. In the
> old testsuite, it was just BeforeClass.
> - The new testsuite uses phantomjs WebDriver by default. The old one
> used htmlUnit. I've checked that phantomjs is around 15-times slower
> than htmlUnit. The main reason is that htmlUnit talks directly with the
> tested server, when phantomJS (similarly like all other real browsers)
> uses remote selenium server, which adds significant overhead. For
> example all the simple operations like "driver.getTitle()" needs to send
> HTTP request to the selenium server.
>
> Hence I've looked at the possibility to switch to htmlUnit by default
> and sent PR for it: https://github.com/keycloak/keycloak/pull/3876 .
>
> New testsuite runs on my laptop in 21 minutes with htmlUnit, when
> previously it was 36 minutes.
>
> However there is the disadvantage, that htmlUnit has limited javascript
> support and is not the best for angular apps. It seems that admin
> console tests won't run with it. I also needed to do some refactoring of
> few existing tests to have them pass with htmlUnit (eg.
> AbstractDemoServletsAdapterTest to use assertEvents instead of admin
> console etc).
>
> However this won't be an issue as browser implementation can be
> overriden with: -Dbrowser=phantomjs
>
> So it's possible to run console tests with proper browsers.
>
> Anyone seeing issues with switch to htmlUnit by default?
>
> I hope that after fix the import + maybe few other things (eg. share
> adminClient per class instead of test method etc), we will be able to
> run whole testsuite in less than 15 minutes!
>
> Thanks,
> Marek
>
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>