Right now, the admin console tests are not being run as part of CI. As a result, they have become neglected and broken. The console bugs I am fixing right now are minor and for now we can get away with merging fixes that don't include unit tests, but the longer we wait, the worse this will get. Can we come up with a plan to get these tests running again? FYI, when I try to run a console test, I get: Caused by: java.lang.IllegalStateException: Password blacklists location does not exist: c:\GitHub\keycloak\testsuite\integration-arquillian\tests\other\console\target\test-classes\password-blacklists I can manually create the password-blacklists directory, but then I get: 08:51:31,786 ERROR [io.undertow.request] UT005023: Exception handling request to /auth/realms/master/protocol/openid-connect/token org.jboss.resteasy.spi.UnhandledException: java.lang.RuntimeException: Uncompilable source code - Erroneous sym type: org.keycloak.authentication.AuthenticationFlowContext.getAuthenticationSession.setAuthNote ... ... Caused by: java.lang.RuntimeException: Uncompilable source code - Erroneous sym type: org.keycloak.authentication.AuthenticationFlowContext.getAuthenticationSession.setAuthNote _______________________________________________ keycloak-dev mailing list keycloak-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-dev

Yes, Console UI tests are run in a CI job (with FF, Chrome and IE + Safari will be added in the near future). I'm currently working on fixing them as they are yet again broken/unstable (UsersInRoleTest which was merged already broken + AuthZ tests recently started to be randomly unstable again, mainly in the IE). This is the main problem of those tests - they're very prone to get unstable, and this "instability ratio" differs from browser to browser. But I believe if we managed to get those tests running in Travis with at least one of the browsers (e.g. Chrome headless which is most stable from them all), we'd prevented at lest merging broken tests and UI changes as well as prevented most of the instability issues. The typical current scenario is: 1) Someone breaks something in the UI, or breaks/makes unstable just the tests either for one browser or even all of them 2) Those changes are merged since Travis passed 3) The CI job starts to be unstable or broken completely 4) I then need to go, find the problem (if it's just a test issue or a real bug) and identify the affected browsers, and then fix the test, or file a JIRA for the bug and ignore the broken test I believe this is not very efficient and imho I'd be perfect if we managed to discover at least some of those problems before merging (it'd saved me some headaches :)). @Stan Regarding the "Password blacklists location". Should've been fixed with https://github.com/keycloak/keycloak/pull/4646 V. On Mon, Nov 13, 2017 at 11:45 PM, Bruno Oliveira <bruno(a)abstractj.org <mailto:bruno@abstractj.org>> wrote: I believe they are part of central CI, just look for keycloak-ui job. On Mon, Nov 13, 2017 at 12:53 PM Stan Silvert <ssilvert(a)redhat.com <mailto:ssilvert@redhat.com>> wrote: > Right now, the admin console tests are not being run as part of CI. As a > result, they have become neglected and broken. The console bugs I am > fixing right now are minor and for now we can get away with merging > fixes that don't include unit tests, but the longer we wait, the worse > this will get. > > Can we come up with a plan to get these tests running again? > > FYI, when I try to run a console test, I get: > Caused by: java.lang.IllegalStateException: Password blacklists location > does not exist: > > c:\GitHub\keycloak\testsuite\integration-arquillian\tests\other\console\target\test-classes\password-blacklists > > I can manually create the password-blacklists directory, but then I get: > 08:51:31,786 ERROR [io.undertow.request] UT005023: Exception handling > request to /auth/realms/master/protocol/openid-connect/token > org.jboss.resteasy.spi.UnhandledException: java.lang.RuntimeException: > Uncompilable source code - Erroneous sym type: > > org.keycloak.authentication.AuthenticationFlowContext.getAuthenticationSession.setAuthNote > ... > ... > Caused by: java.lang.RuntimeException: Uncompilable source code - > Erroneous sym type: > > org.keycloak.authentication.AuthenticationFlowContext.getAuthenticationSession.setAuthNote > _______________________________________________ > keycloak-dev mailing list > keycloak-dev(a)lists.jboss.org <mailto:keycloak-dev@lists.jboss.org> > https://lists.jboss.org/mailman/listinfo/keycloak-dev <https://lists.jboss.org/mailman/listinfo/keycloak-dev> > _______________________________________________ keycloak-dev mailing list keycloak-dev(a)lists.jboss.org <mailto:keycloak-dev@lists.jboss.org> https://lists.jboss.org/mailman/listinfo/keycloak-dev <https://lists.jboss.org/mailman/listinfo/keycloak-dev> -- Václav Muzikář Quality Engineer Keycloak / Red Hat Single Sign-On Red Hat Czech s.r.o.