[keycloak-dev] Improving testsuite performance

[Bartosz Majsak]

Hi Marek,

I meant running x browser tests in parallel with Drone. That can be
achieved with surefire/failsafe. Not sure how does phantomjs nor
htmlunit handles that, but I was successfully running drone tests using
chrome and firefox this way.

Cheers,
Bartosz.

On Mon, Feb 20, 2017 at 12:19 PM, Marek Posolda <mposolda at redhat.com> wrote:

> Hi Bartosz,
>
> we are using parallel running tests on Travis (our CI). That allows test
> classes to be divided into groups and each group is running concurrently
> with other groups. Example script is here:
> https://github.com/keycloak/keycloak/blob/master/travis-run-tests.sh
>
> Did you mean something like that?
>
> Thanks,
> Marek
>
>
> On 18/02/17 17:31, Bartosz Majsak wrote:
>
> Hi Marek,
>
> are you running drone tests in parallel? It could be possible to speed it
> up with just tweaking surfire. In my last project we were using 8-12
> parallel drones and this gave us quite an advantage.
>
> I would be happy to help.
>
> Cheers,
> Bartosz.
>
> On Fri, Feb 17, 2017 at 10:12 PM, Marek Posolda <mposolda at redhat.com>
> wrote:
>
>> ATM running tests in the new arquillian testsuite (
>> testsuite/integration-arquillian ) is significantly slower then running
>> tests in the old testsuite ( testsuite/integration ). It was quite
>> visible in some tests, which were around 3 times slower after they were
>> migrated from the old testsuite.
>>
>> We found that there are 2 major bottlenecks causing it:
>> - There is re-import of all the realms before each test method. In the
>> old testsuite, it was just BeforeClass.
>> - The new testsuite uses phantomjs WebDriver by default. The old one
>> used htmlUnit. I've checked that phantomjs is around 15-times slower
>> than htmlUnit. The main reason is that htmlUnit talks directly with the
>> tested server, when phantomJS (similarly like all other real browsers)
>> uses remote selenium server, which adds significant overhead. For
>> example all the simple operations like "driver.getTitle()" needs to send
>> HTTP request to the selenium server.
>>
>> Hence I've looked at the possibility to switch to htmlUnit by default
>> and sent PR for it: https://github.com/keycloak/keycloak/pull/3876 .
>>
>> New testsuite runs on my laptop in 21 minutes with htmlUnit, when
>> previously it was 36 minutes.
>>
>> However there is the disadvantage, that htmlUnit has limited javascript
>> support and is not the best for angular apps. It seems that admin
>> console tests won't run with it. I also needed to do some refactoring of
>> few existing tests to have them pass with htmlUnit (eg.
>> AbstractDemoServletsAdapterTest to use assertEvents instead of admin
>> console etc).
>>
>> However this won't be an issue as browser implementation can be
>> overriden with: -Dbrowser=phantomjs
>>
>> So it's possible to run console tests with proper browsers.
>>
>> Anyone seeing issues with switch to htmlUnit by default?
>>
>> I hope that after fix the import + maybe few other things (eg. share
>> adminClient per class instead of test method etc), we will be able to
>> run whole testsuite in less than 15 minutes!
>>
>> Thanks,
>> Marek
>>
>> _______________________________________________
>> keycloak-dev mailing list
>> keycloak-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>
>
>
>