[keycloak-user] Uncaught server error: org.keycloak.models.ModelException: Could not find UserStorageProviderFactory

[Joy Kent]

We recently noticed an exception in our keycloak 3.3.4 3-node cluster
running in standalone HA mode:

ESC[0mESC[31m14:10:28,640 ERROR
[org.keycloak.services.error.KeycloakErrorHandler] (default task-25)
Uncaught server error: org.keycloak.models.ModelException: Could not find
UserStorageProviderFactory for: MyUserStorage
        at
org.keycloak.storage.UserStorageManager.getStorageProvider(UserStorageManager.java:164)
        at
org.keycloak.storage.UserStorageManager.getUserById(UserStorageManager.java:370)
        at
org.keycloak.storage.UserStorageManager.getUserByFederatedIdentity(UserStorageManager.java:422)
        at
org.keycloak.models.cache.infinispan.UserCacheSession.getUserByFederatedIdentity(UserCacheSession.java:504)
        at
org.keycloak.protocol.oidc.endpoints.TokenEndpoint.importUserFromExternalIdentity(TokenEndpoint.java:894)
        at
org.keycloak.protocol.oidc.endpoints.TokenEndpoint.exchangeExternalToken(TokenEndpoint.java:857)
        at
org.keycloak.protocol.oidc.endpoints.TokenEndpoint.tokenExchange(TokenEndpoint.java:644)
        at
org.keycloak.protocol.oidc.endpoints.TokenEndpoint.processGrantRequest(TokenEndpoint.java:180)
        at sun.reflect.GeneratedMethodAccessor531.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        .....

Our UserStorageProvider was constructed based on the
`EjbExampleUserStorageProvider` as described here:
https://www.keycloak.org/docs/3.3/server_development/topics/providers.html,
except for the following. The `EjbExampleUserStorageProvider` has these
annotations:

@Stateful
> @Local(EjbExampleUserStorageProvider.class)
> public class EjbExampleUserStorageProvider implements UserStorageProvider,


While our UserStorageProvider has these annotations:

@Stateful(passivationCapable=false)
> @Local(MyUserStorageProvider.class)
> public class MyUserStorageProvider implements UserStorageProvider,


The `(passivationCapable=false)` was added to bypass an exception we hit
when running this in HA mode. It was suggested by this thread:
http://lists.jboss.org/pipermail/keycloak-user/2018-March/013442.html.

There is a corresponding MyUserStorageProviderFactory class for this new
MyUserStorageProvider.

Things worked out fine until recently. From the stack trace above, it seems
like the new MyUserStorageProviderFactory was removed from
keycloakSession's keycloakSessionFactory.

Does anyone know what might cause this?

Thanks,
Joy