I fixed the tests: https://issues.jboss.org/browse/KEYCLOAK-2993

There were two basic issues:

1) Usage of Time object on the client. We added a mechanism to set offset on the server - it's done by using setTimeOffset() of AbstractKeycloakTest rather than Time.setOffset().

However, we forgot that client can also sometimes directly rely on Time object - for one, admin-client uses it to check if token needs to be refreshed before doing a remote call. Also, JsonWebToken has validation methods that rely on Time object, and is also used inside tests. Thus, client-side Time.offset, and server-side Time.offset have to be in sync for things to work properly. Since Undertow runs embedded in the same JVM where tests execute there was no problem, but for Wildfly there are two JVMs then, and any Time.offset mismatch starts wreaking havoc.

2) Installation of some testing providers was not done through testsuite-providers module, but instead used tests/base directly - which works for embedded Undertow which loads from classpath, but not for running with Wildfly which loads from modules.

On Wed, May 11, 2016 at 2:33 PM, Marko Strukelj <mstrukel@redhat.com> wrote:

There are currently many tests failing when running:

mvn clean install -Pdistribution -DskipTests

cd testsuite/integration-arquillian

mvn clean install -Pauth-server-wildfly

Failed tests:
ProvidersTest.testClientAuthenticatorProviders:85->compareProviders:178 Providers count expected:<3> but was:<2>
ProvidersTest.testInitialAuthenticationProviders:130->compareProviders:178 Providers count expected:<20> but was:<18>
ProvidersTest.testPerClientConfigDescriptions:93 null
CustomFlowTest.clientAuthTest:209->grantAccessToken:235 expected:<200> but was:<400>
CustomFlowTest.grantTest:202->grantAccessToken:235 expected:<200> but was:<400>
RefreshTokenTest.refreshTokenRequest:154
Expected: (a value equal to or greater than <1799> and a value less than or equal to <1800>)
but: a value equal to or greater than <1799> <1798> was less than <1799>
RefreshTokenTest.refreshTokenReuseTokenWithRefreshTokensRevoked:261 expected:<400> but was:<200>
RefreshTokenTest.refreshTokenUserSessionMaxLifespan:456 expected:<400> but was:<200>
RefreshTokenTest.testUserSessionRefreshAndIdle:398 Values should be different. Actual: 1462968567

Tests in error:
CustomFlowTest.loginSuccess:193 » IllegalArgument No enum constant org.keycloa...
CustomRegistrationFlowTest.registerUserSuccess:98 » IllegalArgument No enum co...
ResetPasswordTest.resetPasswordExpiredCode:386 » NotAuthorized HTTP 401 Unauth...
ResetPasswordTest.resetPasswordExpiredCodeShort:430 » NotAuthorized HTTP 401 U...
ResetPasswordTest.resetPasswordWithPasswordHisoryPolicy:575->resetPassword:267 » NotAuthorized
OfflineTokenTest.offlineTokenAllowedWithCompositeRole:428->offlineTokenDirectGrantFlow:290 » Runtime
OfflineTokenTest.offlineTokenBrowserFlow:210 » Runtime Failed to verify token
OfflineTokenTest.offlineTokenDirectGrantFlow:311->testRefreshWithOfflineToken:255 » Runtime
OfflineTokenTest.offlineTokenDirectGrantFlowWithRefreshTokensRevoked:325 » Runtime
OfflineTokenTest.offlineTokenServiceAccountFlow:371 » Runtime Failed to verify...

Tests run: 480, Failures: 9, Errors: 10, Skipped: 3

It's probably just server-side changes that require maintenance in the tests themselves.

We should maybe configure Travis CI with -Pauth-server-wildfly option so that we detect these right away.