[keycloak-dev] Failing integration-arquillian tests when using -Pauth-server-wildfly

[Marko Strukelj]

I fixed the tests: https://issues.jboss.org/browse/KEYCLOAK-2993

There were two basic issues:

1) Usage of Time object on the client. We added a mechanism to set offset
on the server - it's done by using setTimeOffset() of AbstractKeycloakTest
rather than Time.setOffset().
However, we forgot that client can also sometimes directly rely on Time
object - for one, admin-client uses it to check if token needs to be
refreshed before doing a remote call. Also, JsonWebToken has validation
methods that rely on Time object, and is also used inside tests. Thus,
client-side Time.offset, and server-side Time.offset have to be in sync for
things to work properly. Since Undertow runs embedded in the same JVM where
tests execute there was no problem, but for Wildfly there are two JVMs
then, and any Time.offset mismatch starts wreaking havoc.

2) Installation of some testing providers was not done through
testsuite-providers module, but instead used tests/base directly - which
works for embedded Undertow which loads from classpath, but not for running
with Wildfly which loads from modules.




On Wed, May 11, 2016 at 2:33 PM, Marko Strukelj <mstrukel at redhat.com> wrote:

> There are currently many tests failing when running:
>
> mvn clean install -Pdistribution -DskipTests
> cd testsuite/integration-arquillian
> mvn clean install -Pauth-server-wildfly
>
> Failed tests:
>   ProvidersTest.testClientAuthenticatorProviders:85->compareProviders:178
> Providers count expected:<3> but was:<2>
>
> ProvidersTest.testInitialAuthenticationProviders:130->compareProviders:178
> Providers count expected:<20> but was:<18>
>   ProvidersTest.testPerClientConfigDescriptions:93 null
>   CustomFlowTest.clientAuthTest:209->grantAccessToken:235 expected:<200>
> but was:<400>
>   CustomFlowTest.grantTest:202->grantAccessToken:235 expected:<200> but
> was:<400>
>   RefreshTokenTest.refreshTokenRequest:154
> Expected: (a value equal to or greater than <1799> and a value less than
> or equal to <1800>)
>      but: a value equal to or greater than <1799> <1798> was less than
> <1799>
>   RefreshTokenTest.refreshTokenReuseTokenWithRefreshTokensRevoked:261
> expected:<400> but was:<200>
>   RefreshTokenTest.refreshTokenUserSessionMaxLifespan:456 expected:<400>
> but was:<200>
>   RefreshTokenTest.testUserSessionRefreshAndIdle:398 Values should be
> different. Actual: 1462968567
>
> Tests in error:
>   CustomFlowTest.loginSuccess:193 » IllegalArgument No enum constant
> org.keycloa...
>   CustomRegistrationFlowTest.registerUserSuccess:98 » IllegalArgument No
> enum co...
>   ResetPasswordTest.resetPasswordExpiredCode:386 » NotAuthorized HTTP 401
> Unauth...
>   ResetPasswordTest.resetPasswordExpiredCodeShort:430 » NotAuthorized HTTP
> 401 U...
>
> ResetPasswordTest.resetPasswordWithPasswordHisoryPolicy:575->resetPassword:267
> » NotAuthorized
>
> OfflineTokenTest.offlineTokenAllowedWithCompositeRole:428->offlineTokenDirectGrantFlow:290
> » Runtime
>   OfflineTokenTest.offlineTokenBrowserFlow:210 » Runtime Failed to verify
> token
>
> OfflineTokenTest.offlineTokenDirectGrantFlow:311->testRefreshWithOfflineToken:255
> » Runtime
>   OfflineTokenTest.offlineTokenDirectGrantFlowWithRefreshTokensRevoked:325
> » Runtime
>   OfflineTokenTest.offlineTokenServiceAccountFlow:371 » Runtime Failed to
> verify...
>
> Tests run: 480, Failures: 9, Errors: 10, Skipped: 3
>
> It's probably just server-side changes that require maintenance in the
> tests themselves.
>
> We should maybe configure Travis CI with -Pauth-server-wildfly option so
> that we detect these right away.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-dev/attachments/20160512/17709899/attachment.html