[keycloak-dev] Console Tests

[Stan Silvert]

Good news.  I'm glad to hear you are actively working on it.  Please 
keep us updated on your progress and let me know if I can help.  It's 
becoming more and more clear that the stability of the admin console is 
suffering because these tests aren't run before merging.

On 11/14/2017 1:51 AM, Vaclav Muzikar wrote:
> Yes, Console UI tests are run in a CI job (with FF, Chrome and IE + 
> Safari will be added in the near future).
> I'm currently working on fixing them as they are yet again 
> broken/unstable (UsersInRoleTest which was merged already broken + 
> AuthZ tests recently started to be randomly unstable again, mainly in 
> the IE).
> This is the main problem of those tests - they're very prone to get 
> unstable, and this "instability ratio" differs from browser to browser.
>
> But I believe if we managed to get those tests running in Travis with 
> at least one of the browsers (e.g. Chrome headless which is most 
> stable from them all), we'd prevented at lest merging broken tests and 
> UI changes as well as prevented most of the instability issues.
> The typical current scenario is:
> 1) Someone breaks something in the UI, or breaks/makes unstable just 
> the tests either for one browser or even all of them
> 2) Those changes are merged since Travis passed
> 3) The CI job starts to be unstable or broken completely
> 4) I then need to go, find the problem (if it's just a test issue or a 
> real bug) and identify the affected browsers, and then fix the test, 
> or file a JIRA for the bug and ignore the broken test
> I believe this is not very efficient and imho I'd be perfect if we 
> managed to discover at least some of those problems before merging 
> (it'd saved me some headaches :)).
>
> @Stan Regarding the "Password blacklists location". Should've been 
> fixed with https://github.com/keycloak/keycloak/pull/4646
>
> V.
>
> On Mon, Nov 13, 2017 at 11:45 PM, Bruno Oliveira <bruno at abstractj.org 
> <mailto:bruno at abstractj.org>> wrote:
>
>     I believe they are part of central CI, just look for keycloak-ui job.
>
>     On Mon, Nov 13, 2017 at 12:53 PM Stan Silvert <ssilvert at redhat.com
>     <mailto:ssilvert at redhat.com>> wrote:
>
>     > Right now, the admin console tests are not being run as part of
>     CI. As a
>     > result, they have become neglected and broken.  The console bugs
>     I am
>     > fixing right now are minor and for now we can get away with merging
>     > fixes that don't include unit tests, but the longer we wait, the
>     worse
>     > this will get.
>     >
>     > Can we come up with a plan to get these tests running again?
>     >
>     > FYI, when I try to run a console test, I get:
>     > Caused by: java.lang.IllegalStateException: Password blacklists
>     location
>     > does not exist:
>     >
>     >
>     c:\GitHub\keycloak\testsuite\integration-arquillian\tests\other\console\target\test-classes\password-blacklists
>     >
>     > I can manually create the password-blacklists directory, but
>     then I get:
>     > 08:51:31,786 ERROR [io.undertow.request] UT005023: Exception
>     handling
>     > request to /auth/realms/master/protocol/openid-connect/token
>     > org.jboss.resteasy.spi.UnhandledException:
>     java.lang.RuntimeException:
>     > Uncompilable source code - Erroneous sym type:
>     >
>     >
>     org.keycloak.authentication.AuthenticationFlowContext.getAuthenticationSession.setAuthNote
>     > ...
>     > ...
>     > Caused by: java.lang.RuntimeException: Uncompilable source code -
>     > Erroneous sym type:
>     >
>     >
>     org.keycloak.authentication.AuthenticationFlowContext.getAuthenticationSession.setAuthNote
>     > _______________________________________________
>     > keycloak-dev mailing list
>     > keycloak-dev at lists.jboss.org <mailto:keycloak-dev at lists.jboss.org>
>     > https://lists.jboss.org/mailman/listinfo/keycloak-dev
>     <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
>     >
>     _______________________________________________
>     keycloak-dev mailing list
>     keycloak-dev at lists.jboss.org <mailto:keycloak-dev at lists.jboss.org>
>     https://lists.jboss.org/mailman/listinfo/keycloak-dev
>     <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
>
>
>
>
> -- 
> Václav Muzikář
> Quality Engineer
> Keycloak / Red Hat Single Sign-On
> Red Hat Czech s.r.o.