[keycloak-user] Keycloak 3.4.0 cross datacenter issue with external Infinispan

Tue Nov 28 14:32:02 EST 2017

Hi All,

We are running Cross dc setup with 2 data centers. In this setup we are
using external infinispan server 9.1.2.Final. Each dc will have its own
external infinispan cluster. And cross dc infinispan cluster is formed
using Relay2 protocol.

We haven't faced any issues till 3.2.1.Final. When we trying to upgrade
Keycloak to 3.4.0.Final we are facing the following issue: Keycloak in
first data center are starting up fine but keycloak in second data center
fails to start with below error

16:18:58,473 WARN  [org.infinispan.client.hotrod.impl.protocol.Codec21]
(pool-13-thread-1) ISPN004005: Error received from the server:
java.lang.ClassNotFoundException:
org.keycloak.models.sessions.infinispan.changes.SessionEntityWrapper from
[Module "org.infinispan.commons:main" from local module loader @1c2c22f3
(finder: local module finder @18e8568 (roots:
/opt/jboss/infinispan-server/modules,/opt/jboss/infinispan-server/modules/system/layers/base))]
2017-11-28 16:18:58,473 DEBG 'Keycloak' stdout output:
16:18:58,473 WARN  [org.infinispan.client.hotrod.impl.protocol.Codec21]
(pool-13-thread-1) ISPN004005: Error received from the server:
java.lang.ClassNotFoundException:
org.keycloak.models.sessions.infinispan.changes.SessionEntityWrapper from
[Module "org.infinispan.commons:main" from local module loader @1c2c22f3
(finder: local module finder @18e8568 (roots:
/opt/jboss/infinispan-server/modules,/opt/jboss/infinispan-server/modules/system/layers/base))]

16:18:58,474 ERROR
[org.keycloak.models.sessions.infinispan.initializer.InfinispanCacheInitializer]
(ServerService Thread Pool -- 57) ExecutionException when computed future.
Errors: 1: java.util.concurrent.ExecutionException:
org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for
messageId=23 returned server error (status=0x85):
java.lang.ClassNotFoundException:
org.keycloak.models.sessions.infinispan.changes.SessionEntityWrapper from
[Module "org.infinispan.commons:main" from local module loader @1c2c22f3
(finder: local module finder @18e8568 (roots:
/opt/jboss/infinispan-server/modules,/opt/jboss/infinispan-server/modules/system/layers/base))]
 at java.util.concurrent.FutureTask.report(Unknown Source)
 at java.util.concurrent.FutureTask.get(Unknown Source)
 at
org.infinispan.commons.util.concurrent.NotifyingFutureImpl.get(NotifyingFutureImpl.java:88)
 at
org.infinispan.distexec.DefaultExecutorService$LocalDistributedTaskPart.getResult(DefaultExecutorService.java:1083)
 at
org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868)
 at
org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:848)
 at
org.keycloak.models.sessions.infinispan.initializer.InfinispanCacheInitializer.startLoading(InfinispanCacheInitializer.java:102)
 at
org.keycloak.models.sessions.infinispan.initializer.CacheInitializer.loadSessions(CacheInitializer.java:41)
 at
org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory$7.run(InfinispanUserSessionProviderFactory.java:273)
 at
org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:229)
 at
org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory.loadSessionsFromRemoteCache(InfinispanUserSessionProviderFactory.java:263)
 at
org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory.loadSessionsFromRemoteCaches(InfinispanUserSessionProviderFactory.java:255)
 at
org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory.access$200(InfinispanUserSessionProviderFactory.java:62)
 at
org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory$1.onEvent(InfinispanUserSessionProviderFactory.java:110)
 at
org.keycloak.services.DefaultKeycloakSessionFactory.publish(DefaultKeycloakSessionFactory.java:68)
 at
org.keycloak.services.resources.KeycloakApplication$2.run(KeycloakApplication.java:165)
 at
org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:229)
 at
org.keycloak.services.resources.KeycloakApplication.<init>(KeycloakApplication.java:158)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source)
 at java.lang.reflect.Constructor.newInstance(Unknown Source)
 at
org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:150)
 at
org.jboss.resteasy.spi.ResteasyProviderFactory.createProviderInstance(ResteasyProviderFactory.java:2298)
 at
org.jboss.resteasy.spi.ResteasyDeployment.createApplication(ResteasyDeployment.java:340)
 at
org.jboss.resteasy.spi.ResteasyDeployment.start(ResteasyDeployment.java:253)
 at
org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.init(ServletContainerDispatcher.java:120)
 at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.init(HttpServletDispatcher.java:36)
 at
io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:117)
 at
org.wildfly.extension.undertow.security.RunAsLifecycleInterceptor.init(RunAsLifecycleInterceptor.java:78)
 at
io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:103)
 at
io.undertow.servlet.core.ManagedServlet$DefaultInstanceStrategy.start(ManagedServlet.java:265)
 at
io.undertow.servlet.core.ManagedServlet.createServlet(ManagedServlet.java:133)
 at
io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:565)
 at
io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:536)
 at
io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
 at
io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
 at
org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
 at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
 at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
 at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
 at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
 at
io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:578)
 at
org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:100)
 at
org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:81)
 at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
 at java.util.concurrent.FutureTask.run(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)
 at org.jboss.threads.JBossThread.run(JBossThread.java:320)
Caused by:
org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for
messageId=23 returned server error (status=0x85):
java.lang.ClassNotFoundException:
org.keycloak.models.sessions.infinispan.changes.SessionEntityWrapper from
[Module "org.infinispan.commons:main" from local module loader @1c2c22f3
(finder: local module finder @18e8568 (roots:
/opt/jboss/infinispan-server/modules,/opt/jboss/infinispan-server/modules/system/layers/base))]
 at
org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:350)
 at
org.infinispan.client.hotrod.impl.protocol.Codec20.readPartialHeader(Codec20.java:139)
 at
org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:125)
 at
org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:56)
 at
org.infinispan.client.hotrod.impl.operations.ExecuteOperation.executeOperation(ExecuteOperation.java:48)
 at
org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:54)
 at
org.infinispan.client.hotrod.impl.RemoteCacheImpl.execute(RemoteCacheImpl.java:724)
 at
org.keycloak.models.sessions.infinispan.remotestore.RemoteCacheSessionsLoader.loadSessions(RemoteCacheSessionsLoader.java:109)
 at
org.keycloak.models.sessions.infinispan.initializer.SessionInitializerWorker$1.run(SessionInitializerWorker.java:74)
 at
org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:229)
 at
org.keycloak.models.sessions.infinispan.initializer.SessionInitializerWorker.call(SessionInitializerWorker.java:70)
 at
org.keycloak.models.sessions.infinispan.initializer.SessionInitializerWorker.call(SessionInitializerWorker.java:34)
 at
org.infinispan.commands.read.DistributedExecuteCommand.perform(DistributedExecuteCommand.java:107)
 at
org.infinispan.distexec.DefaultExecutorService$LocalDistributedTaskPart$1.doLocalInvoke(DefaultExecutorService.java:1112)
 at
org.infinispan.distexec.DefaultExecutorService$LocalDistributedTaskPart$1.call(DefaultExecutorService.java:1102)
 at java.util.concurrent.FutureTask.run(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

At the same time we are getting below in external Infinispan

2017-11-28 16:51:37,357 DEBUG
[org.infinispan.server.hotrod.HotRodExceptionHandler]
(HotRod-ServerWorker-9-3) Exception caught:
org.infinispan.commons.CacheException: java.lang.ClassNotFoundException:
org.keycloak.models.sessions.infinispan.changes.SessionEntityWrapper from
[Module "org.infinispan.commons:main" from local module loader @1c2c22f3
(finder: local module finder @18e8568 (roo
ts:
/opt/jboss/infinispan-server/modules,/opt/jboss/infinispan-server/modules/system/layers/base))]
at
org.infinispan.commons.dataconversion.MarshallerEncoder.fromStorage(MarshallerEncoder.java:36)
at
org.infinispan.cache.impl.EncoderEntryMapper.decode(EncoderEntryMapper.java:43)
at
org.infinispan.cache.impl.EncoderEntryMapper.apply(EncoderEntryMapper.java:57)
at
org.infinispan.cache.impl.EncoderEntryMapper.apply(EncoderEntryMapper.java:23)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at
java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1812)
at
org.infinispan.commons.util.Closeables$SpliteratorAsCloseableSpliterator.tryAdvance(Closeables.java:143)
at
java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
at
java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at
org.infinispan.stream.impl.local.LocalCacheStream.collect(LocalCacheStream.java:258)
at
org.infinispan.util.AbstractDelegatingCacheStream.collect(AbstractDelegatingCacheStream.java:273)
at
jdk.nashorn.internal.scripts.Script$Recompilation$1$58$\^eval\_.loadSessions(<eval>:5)
at jdk.nashorn.internal.scripts.Script$\^eval\_.:program(<eval>:22)
at
jdk.nashorn.internal.runtime.ScriptFunctionData.invoke(ScriptFunctionData.java:637)
at
jdk.nashorn.internal.runtime.ScriptFunction.invoke(ScriptFunction.java:494)
at jdk.nashorn.internal.runtime.ScriptRuntime.apply(ScriptRuntime.java:393)
at
jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:421)
at
jdk.nashorn.api.scripting.NashornScriptEngine.access$300(NashornScriptEngine.java:73)
at
jdk.nashorn.api.scripting.NashornScriptEngine$3.eval(NashornScriptEngine.java:514)
at javax.script.CompiledScript.eval(CompiledScript.java:92)
at
org.infinispan.scripting.impl.ScriptingManagerImpl.execute(ScriptingManagerImpl.java:239)
at org.infinispan.scripting.impl.LocalRunner.runScript(LocalRunner.java:19)
at
org.infinispan.scripting.impl.ScriptingManagerImpl.runScript(ScriptingManagerImpl.java:222)
at
org.infinispan.scripting.impl.ScriptingTaskEngine.runTask(ScriptingTaskEngine.java:44)
at
org.infinispan.tasks.impl.TaskManagerImpl.runTask(TaskManagerImpl.java:99)
at
org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:120)
at
org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$0(ContextHandler.java:52)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException:
org.keycloak.models.sessions.infinispan.changes.SessionEntityWrapper from
[Module "org.infinispan.commons:main" from local module loader @1c2c22f3
(finder: local module finder @18e8568 (roots:
/opt/jboss/infinispan-server/modules,/opt/jboss/infinispan-server/modules/system/layers/base))]
at
org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:198)
at
org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:363)
at
org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:351)
at
org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:93)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at
org.jboss.marshalling.AbstractClassResolver.loadClass(AbstractClassResolver.java:131)
at
org.jboss.marshalling.AbstractClassResolver.resolveClass(AbstractClassResolver.java:112)
at
org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:1087)
at
org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1354)
at
org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:275)
at
org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:208)
at
org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
at
org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:134)
at
org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromByteBuffer(AbstractJBossMarshaller.java:112)
at
org.infinispan.commons.marshall.AbstractMarshaller.objectFromByteBuffer(AbstractMarshaller.java:82)
at
org.infinispan.commons.dataconversion.MarshallerEncoder.fromStorage(MarshallerEncoder.java:34)
... 35 more

I believe the error might be due to the latest code addition for supporting
cross dc. Any help on this issue is appreciated.

Thanks,
Vikrant