[keycloak-dev] Console Tests

[Vaclav Muzikar]

Yes, Console UI tests are run in a CI job (with FF, Chrome and IE + Safari
will be added in the near future).
I'm currently working on fixing them as they are yet again broken/unstable
(UsersInRoleTest which was merged already broken + AuthZ tests recently
started to be randomly unstable again, mainly in the IE).
This is the main problem of those tests - they're very prone to get
unstable, and this "instability ratio" differs from browser to browser.

But I believe if we managed to get those tests running in Travis with at
least one of the browsers (e.g. Chrome headless which is most stable from
them all), we'd prevented at lest merging broken tests and UI changes as
well as prevented most of the instability issues.
The typical current scenario is:
1) Someone breaks something in the UI, or breaks/makes unstable just the
tests either for one browser or even all of them
2) Those changes are merged since Travis passed
3) The CI job starts to be unstable or broken completely
4) I then need to go, find the problem (if it's just a test issue or a real
bug) and identify the affected browsers, and then fix the test, or file a
JIRA for the bug and ignore the broken test
I believe this is not very efficient and imho I'd be perfect if we managed
to discover at least some of those problems before merging (it'd saved me
some headaches :)).

@Stan Regarding the "Password blacklists location". Should've been fixed
with https://github.com/keycloak/keycloak/pull/4646

V.

On Mon, Nov 13, 2017 at 11:45 PM, Bruno Oliveira <bruno at abstractj.org>
wrote:

> I believe they are part of central CI, just look for keycloak-ui job.
>
> On Mon, Nov 13, 2017 at 12:53 PM Stan Silvert <ssilvert at redhat.com> wrote:
>
> > Right now, the admin console tests are not being run as part of CI. As a
> > result, they have become neglected and broken.  The console bugs I am
> > fixing right now are minor and for now we can get away with merging
> > fixes that don't include unit tests, but the longer we wait, the worse
> > this will get.
> >
> > Can we come up with a plan to get these tests running again?
> >
> > FYI, when I try to run a console test, I get:
> > Caused by: java.lang.IllegalStateException: Password blacklists location
> > does not exist:
> >
> > c:\GitHub\keycloak\testsuite\integration-arquillian\tests\
> other\console\target\test-classes\password-blacklists
> >
> > I can manually create the password-blacklists directory, but then I get:
> > 08:51:31,786 ERROR [io.undertow.request] UT005023: Exception handling
> > request to /auth/realms/master/protocol/openid-connect/token
> > org.jboss.resteasy.spi.UnhandledException: java.lang.RuntimeException:
> > Uncompilable source code - Erroneous sym type:
> >
> > org.keycloak.authentication.AuthenticationFlowContext.
> getAuthenticationSession.setAuthNote
> > ...
> > ...
> > Caused by: java.lang.RuntimeException: Uncompilable source code -
> > Erroneous sym type:
> >
> > org.keycloak.authentication.AuthenticationFlowContext.
> getAuthenticationSession.setAuthNote
> > _______________________________________________
> > keycloak-dev mailing list
> > keycloak-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
> >
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>



-- 
Václav Muzikář
Quality Engineer
Keycloak / Red Hat Single Sign-On
Red Hat Czech s.r.o.