ATM running tests in the new arquillian testsuite ( testsuite/integration-arquillian ) is significantly slower then running tests in the old testsuite ( testsuite/integration ). It was quite visible in some tests, which were around 3 times slower after they were migrated from the old testsuite. We found that there are 2 major bottlenecks causing it: - There is re-import of all the realms before each test method. In the old testsuite, it was just BeforeClass. - The new testsuite uses phantomjs WebDriver by default. The old one used htmlUnit. I've checked that phantomjs is around 15-times slower than htmlUnit. The main reason is that htmlUnit talks directly with the tested server, when phantomJS (similarly like all other real browsers) uses remote selenium server, which adds significant overhead. For example all the simple operations like "driver.getTitle()" needs to send HTTP request to the selenium server. Hence I've looked at the possibility to switch to htmlUnit by default and sent PR for it: https://github.com/keycloak/keycloak/pull/3876 . New testsuite runs on my laptop in 21 minutes with htmlUnit, when previously it was 36 minutes. However there is the disadvantage, that htmlUnit has limited javascript support and is not the best for angular apps. It seems that admin console tests won't run with it. I also needed to do some refactoring of few existing tests to have them pass with htmlUnit (eg. AbstractDemoServletsAdapterTest to use assertEvents instead of admin console etc). However this won't be an issue as browser implementation can be overriden with: -Dbrowser=phantomjs So it's possible to run console tests with proper browsers. Anyone seeing issues with switch to htmlUnit by default? I hope that after fix the import + maybe few other things (eg. share adminClient per class instead of test method etc), we will be able to run whole testsuite in less than 15 minutes! Thanks, Marek _______________________________________________ keycloak-dev mailing list keycloak-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-dev

Hi Bartosz, we are using parallel running tests on Travis (our CI). That allows test classes to be divided into groups and each group is running concurrently with other groups. Example script is here: https://github.com/keycloak/keycloak/blob/master/travis-run-tests.sh Did you mean something like that? Thanks, Marek On 18/02/17 17:31, Bartosz Majsak wrote: Hi Marek, are you running drone tests in parallel? It could be possible to speed it up with just tweaking surfire. In my last project we were using 8-12 parallel drones and this gave us quite an advantage. I would be happy to help. Cheers, Bartosz. On Fri, Feb 17, 2017 at 10:12 PM, Marek Posolda <mposolda(a)redhat.com&gt; wrote: > ATM running tests in the new arquillian testsuite ( > testsuite/integration-arquillian ) is significantly slower then running > tests in the old testsuite ( testsuite/integration ). It was quite > visible in some tests, which were around 3 times slower after they were > migrated from the old testsuite. > > We found that there are 2 major bottlenecks causing it: > - There is re-import of all the realms before each test method. In the > old testsuite, it was just BeforeClass. > - The new testsuite uses phantomjs WebDriver by default. The old one > used htmlUnit. I've checked that phantomjs is around 15-times slower > than htmlUnit. The main reason is that htmlUnit talks directly with the > tested server, when phantomJS (similarly like all other real browsers) > uses remote selenium server, which adds significant overhead. For > example all the simple operations like "driver.getTitle()" needs to send > HTTP request to the selenium server. > > Hence I've looked at the possibility to switch to htmlUnit by default > and sent PR for it: https://github.com/keycloak/keycloak/pull/3876 . > > New testsuite runs on my laptop in 21 minutes with htmlUnit, when > previously it was 36 minutes. > > However there is the disadvantage, that htmlUnit has limited javascript > support and is not the best for angular apps. It seems that admin > console tests won't run with it. I also needed to do some refactoring of > few existing tests to have them pass with htmlUnit (eg. > AbstractDemoServletsAdapterTest to use assertEvents instead of admin > console etc). > > However this won't be an issue as browser implementation can be > overriden with: -Dbrowser=phantomjs > > So it's possible to run console tests with proper browsers. > > Anyone seeing issues with switch to htmlUnit by default? > > I hope that after fix the import + maybe few other things (eg. share > adminClient per class instead of test method etc), we will be able to > run whole testsuite in less than 15 minutes! > > Thanks, > Marek > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev >

Hi Marek, I meant running x browser tests in parallel with Drone. That can be achieved with surefire/failsafe. Not sure how does phantomjs nor htmlunit handles that, but I was successfully running drone tests using chrome and firefox this way. Cheers, Bartosz. On Mon, Feb 20, 2017 at 12:19 PM, Marek Posolda <mposolda(a)redhat.com <mailto:mposolda@redhat.com>> wrote: Hi Bartosz, we are using parallel running tests on Travis (our CI). That allows test classes to be divided into groups and each group is running concurrently with other groups. Example script is here: https://github.com/keycloak/keycloak/blob/master/travis-run-tests.sh <https://github.com/keycloak/keycloak/blob/master/travis-run-tests.sh> Did you mean something like that? Thanks, Marek On 18/02/17 17:31, Bartosz Majsak wrote: > Hi Marek, > > are you running drone tests in parallel? It could be possible to > speed it up with just tweaking surfire. In my last project we > were using 8-12 parallel drones and this gave us quite an advantage. > > I would be happy to help. > > Cheers, > Bartosz. > > On Fri, Feb 17, 2017 at 10:12 PM, Marek Posolda > <mposolda(a)redhat.com <mailto:mposolda@redhat.com>> wrote: > > ATM running tests in the new arquillian testsuite ( > testsuite/integration-arquillian ) is significantly slower > then running > tests in the old testsuite ( testsuite/integration ). It was > quite > visible in some tests, which were around 3 times slower after > they were > migrated from the old testsuite. > > We found that there are 2 major bottlenecks causing it: > - There is re-import of all the realms before each test > method. In the > old testsuite, it was just BeforeClass. > - The new testsuite uses phantomjs WebDriver by default. The > old one > used htmlUnit. I've checked that phantomjs is around 15-times > slower > than htmlUnit. The main reason is that htmlUnit talks > directly with the > tested server, when phantomJS (similarly like all other real > browsers) > uses remote selenium server, which adds significant overhead. For > example all the simple operations like "driver.getTitle()" > needs to send > HTTP request to the selenium server. > > Hence I've looked at the possibility to switch to htmlUnit by > default > and sent PR for it: > https://github.com/keycloak/keycloak/pull/3876 > <https://github.com/keycloak/keycloak/pull/3876> . > > New testsuite runs on my laptop in 21 minutes with htmlUnit, when > previously it was 36 minutes. > > However there is the disadvantage, that htmlUnit has limited > javascript > support and is not the best for angular apps. It seems that admin > console tests won't run with it. I also needed to do some > refactoring of > few existing tests to have them pass with htmlUnit (eg. > AbstractDemoServletsAdapterTest to use assertEvents instead > of admin > console etc). > > However this won't be an issue as browser implementation can be > overriden with: -Dbrowser=phantomjs > > So it's possible to run console tests with proper browsers. > > Anyone seeing issues with switch to htmlUnit by default? > > I hope that after fix the import + maybe few other things > (eg. share > adminClient per class instead of test method etc), we will be > able to > run whole testsuite in less than 15 minutes! > > Thanks, > Marek > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev(a)lists.jboss.org > <mailto:keycloak-dev@lists.jboss.org> > https://lists.jboss.org/mailman/listinfo/keycloak-dev > <https://lists.jboss.org/mailman/listinfo/keycloak-dev> > >