[keycloak-dev] Improving testsuite performance

[Marek Posolda]

ATM running tests in the new arquillian testsuite ( 
testsuite/integration-arquillian ) is significantly slower then running 
tests in the old testsuite ( testsuite/integration ). It was quite 
visible in some tests, which were around 3 times slower after they were 
migrated from the old testsuite.

We found that there are 2 major bottlenecks causing it:
- There is re-import of all the realms before each test method. In the 
old testsuite, it was just BeforeClass.
- The new testsuite uses phantomjs WebDriver by default. The old one 
used htmlUnit. I've checked that phantomjs is around 15-times slower 
than htmlUnit. The main reason is that htmlUnit talks directly with the 
tested server, when phantomJS (similarly like all other real browsers) 
uses remote selenium server, which adds significant overhead. For 
example all the simple operations like "driver.getTitle()" needs to send 
HTTP request to the selenium server.

Hence I've looked at the possibility to switch to htmlUnit by default 
and sent PR for it: https://github.com/keycloak/keycloak/pull/3876 .

New testsuite runs on my laptop in 21 minutes with htmlUnit, when 
previously it was 36 minutes.

However there is the disadvantage, that htmlUnit has limited javascript 
support and is not the best for angular apps. It seems that admin 
console tests won't run with it. I also needed to do some refactoring of 
few existing tests to have them pass with htmlUnit (eg. 
AbstractDemoServletsAdapterTest to use assertEvents instead of admin 
console etc).

However this won't be an issue as browser implementation can be 
overriden with: -Dbrowser=phantomjs

So it's possible to run console tests with proper browsers.

Anyone seeing issues with switch to htmlUnit by default?

I hope that after fix the import + maybe few other things (eg. share 
adminClient per class instead of test method etc), we will be able to 
run whole testsuite in less than 15 minutes!

Thanks,
Marek