I agree to set runOrder to "alphabetical". This will unify test runs on local machines and Jenkins which is clear benefit for us. Peter On Wed, Aug 14, 2019 at 10:35 PM Marek Posolda <mposolda(a)redhat.com&gt; wrote: > > It seems it may be possible to configure maven-surefire-plugin to use > "static" order of test classes when executing testsuite. Should be > doable with the "runOrder" parameter switched for example to > "alphabetical" [1]. > > In theory, order of the tests shouldn't be important as each test should > cleanup after itself. Reality is, that we often have bugs due that some > test class didn't properly clean everything after itself. And when some > following test fails, it may not be so trivial to find the original test > class, which caused this. Static order should help with having the > testsuite more deterministic and ensure that you can simulate same > issues locally and in jenkins - see for example this comment for some > details [2]. > > WDYT? > > [1] > http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#run... > [2] > https://issues.jboss.org/browse/KEYCLOAK-10904?focusedCommentId=13770826&... > > Marek > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev _______________________________________________ keycloak-dev mailing list keycloak-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-dev

+1 on this. That may also help us also with CrossDC tests. On Wed, Aug 14, 2019 at 5:33 PM Marek Posolda <mposolda(a)redhat.com&gt; wrote: > It seems it may be possible to configure maven-surefire-plugin to use > "static" order of test classes when executing testsuite. Should be > doable with the "runOrder" parameter switched for example to > "alphabetical" [1]. > > In theory, order of the tests shouldn't be important as each test should > cleanup after itself. Reality is, that we often have bugs due that some > test class didn't properly clean everything after itself. And when some > following test fails, it may not be so trivial to find the original test > class, which caused this. Static order should help with having the > testsuite more deterministic and ensure that you can simulate same > issues locally and in jenkins - see for example this comment for some > details [2]. > > WDYT? > > [1] > http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#run... > [2] > https://issues.jboss.org/browse/KEYCLOAK-10904?focusedCommentId=13770826&... > > Marek > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev