10:30 a.m.
Hi all,
I'm trying to deploy KeyCloak (v 4.8.3-final) in Standalone Cluster mode in order to
answer a very specific requirement: get continuity of service 1/ in case of crash of a
KeyCloak instance, 2/ during the upgrade of my solution.
However, I unfortunately don't get such results :(, despite the fact the cluster looks
to be properly configured.
First, as I'm deploying KeyCloak in Kubernetes, I configured KeyCloak in Standalone
Cluster by using the DNS_PING and a TCP transport for JGroups. Instances of the cluster
can discover/see each other and the cluster is working fine as soon as it is used a couple
of seconds/minutes after starting the instances.
I've created a simple script that just get in a loop the JWT tokens using the Token
endpoint of a Realm, and I always have a 200 Ok status code whatever the KeyCloak instance
that is hit through the Kubernetes Service.
However, coming back to the 2 uses cases I'm interested in, it looks the KeyCloak
instances are getting crazy as soon as the cluster is not "stable". By
"cluster is not stable", I mean:
- When scaling down the number of Keycloak instances (whatever it is by killing a
Container or by a smart scale down of the Kubernetes Deployment)
- When performing a rolling update of the Pods
In both cases, during a particular time, most of the calls to get a JWT return a HTTP 499
status code and KeyCloak logs show the following:
08:14:56,785 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor]
(timeout-thread--p13-t1) ISPN000136: Error executing command GetKeyValueCommand, writing
keys ISPN000476: Timed out waiting for responses for request 1883 from
id-provider-5c55bbd99d-kqr8v
at
org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
at
org.infinispan.remoting.transport.jgroups.StaggeredRequest.onTimeout(StaggeredRequest.java:64)
at
org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
at
org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
08:14:56,790 ERROR [org.keycloak.services.error.KeycloakErrorHandler] (default task-992)
Uncaught server error: org.infinispan.util.concurrent.TimeoutException: ISPN000476:
Time-provider-5c55bbd99d-kqr8v
at
org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:472)
at
org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
at
org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
at
org.keycloak.models.sessions.infinispan.changes.InfinispanChangelogBasedTransaction.get(InfinispanChangelogBasedTransaction.java:120)
at
org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider.getLoginFailureEntity(InfinispanUserSessionProvider.java:678)
at
org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider.getUserLoginFailure(InfinispanUserSessionProvider.java:672)
at
org.keycloak.services.managers.DefaultBruteForceProtector.isTemporarilyDisabled(DefaultBruteForceProtector.java:306)
at
org.keycloak.authentication.authenticators.directgrant.ValidateUsername.authenticate(ValidateUsername.java:85)
at
org.keycloak.authentication.DefaultAuthenticationFlow.processFlow(DefaultAuthenticationFlow.java:221)
at
org.keycloak.authentication.DefaultAuthenticationFlow.processFlow(DefaultAuthenticationFlow.java:148)
at
org.keycloak.authentication.AuthenticationProcessor.authenticateOnly(AuthenticationProcessor.java:910)
at
org.keycloak.protocol.oidc.endpoints.TokenEndpoint.resourceOwnerPasswordCredentialsGrant(TokenEndpoint.java:554)
at
org.keycloak.protocol.oidc.endpoints.TokenEndpoint.processGrantRequest(TokenEndpoint.java:187)
at sun.reflect.GeneratedMethodAccessor449.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:140)
at
org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:509)
at
org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:399)
at
org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$0(ResourceMethodInvoker.java:363)
at
org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
at
org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:365)
at
org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:337)
at
org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:137)
at
org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:106)
at
org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:132)
at
org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:100)
at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:443)
at
org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:233)
at
org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:139)
at
org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
at
org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:142)
at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:219)
at
org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:791)
at
io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
at
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at
org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:90)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at
io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at
io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at
io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
at
io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at
org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
at
io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at
io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at
io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at
io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at
io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
at
io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
at
io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
at
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
at
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
at
io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at
io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at
org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at
io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
at
io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at
io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
at
org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at
org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting
for responses for request 1883 from id-provider-5c55bbd99d-kqr8v
at
org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
at
org.infinispan.remoting.transport.jgroups.StaggeredRequest.onTimeout(StaggeredRequest.java:64)
at
org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
at
org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Suppressed: org.infinispan.util.logging.TraceException
at
org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41)
at
org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:472)
at
org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
at
org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
at
org.keycloak.models.sessions.infinispan.changes.InfinispanChangelogBasedTransaction.get(InfinispanChangelogBasedTransaction.java:120)
at
org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider.getLoginFailureEntity(InfinispanUserSessionProvider.java:678)
at
org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider.getUserLoginFailure(InfinispanUserSessionProvider.java:672)
at
org.keycloak.services.managers.DefaultBruteForceProtector.isTemporarilyDisabled(DefaultBruteForceProtector.java:306)
at
org.keycloak.authentication.authenticators.directgrant.ValidateUsername.authenticate(ValidateUsername.java:85)
at
org.keycloak.authentication.DefaultAuthenticationFlow.processFlow(DefaultAuthenticationFlow.java:221)
at
org.keycloak.authentication.DefaultAuthenticationFlow.processFlow(DefaultAuthenticationFlow.java:148)
at
org.keycloak.authentication.AuthenticationProcessor.authenticateOnly(AuthenticationProcessor.java:910)
at
org.keycloak.protocol.oidc.endpoints.TokenEndpoint.resourceOwnerPasswordCredentialsGrant(TokenEndpoint.java:554)
at
org.keycloak.protocol.oidc.endpoints.TokenEndpoint.processGrantRequest(TokenEndpoint.java:187)
at sun.reflect.GeneratedMethodAccessor449.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:140)
at
org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:509)
at
org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:399)
at
org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$0(ResourceMethodInvoker.java:363)
at
org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
at
org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:365)
at
org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:337)
at
org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:137)
at
org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:106)
at
org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:132)
at
org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:100)
at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:443)
at
org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:233)
at
org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:139)
at
org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
at
org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:142)
at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:219)
at
org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:791)
at
io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
at
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at
org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:90)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at
io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at
io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at
io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
at
io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at
org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
at
io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at
io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at
io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at
io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at
io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
at
io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
at
io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
at
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
at
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
at
io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at
io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at
org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at
io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
at
io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at
io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
at
io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
at
org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at
org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
... 1 more
It looks that the KeyCloak instances that is still alive and receiving the call for
generation of the JWT is trying to contact the other instances in order to get some data
from the distributed cache... In particular, it tries to contact the instance that is no
longer here (because killed, shutdown, rolled...), received a Timeout error, and then
terminates the incoming request in error.
As I'm making the same request within a loop, I see the error happening during a
couple of seconds (around 10 to 15 secs), i.e. during the time the cluster composition is
not stabilized yet. When the re-discovery of the cluster has been performed, the new
composition of the cluster is updated, and everything goes back to normal!
I can understand, from KeyCloak documentation on server caches
(https://www.keycloak.org/docs/4.8/server_installation/index.html#_replica...) that
"By default, Keycloak only specifies one owner for data. So if that one node goes
down that data is lost. This usually means that users will be logged out and will have to
login again. You can change the number of nodes that replicate a piece of data by change
the owners attribute in the distributed-cache declaration.", but unfortunately,
setting the "owner" field to 2 (or more !) for all distributed caches does not
remove the issue.
I even tried to change the type of cache from "distributed-cache" to
"replicated-cache", but then KeyCloak is not starting:
<replicated-cache name="sessions"/>
<replicated-cache name="authenticationSessions"/>
<replicated-cache name="offlineSessions"/>
<replicated-cache name="clientSessions"/>
<replicated-cache name="offlineClientSessions"/>
<replicated-cache name="loginFailures"/>
<replicated-cache name="actionTokens">
<object-memory size="-1"/>
<expiration interval="300000" max-idle="-1"/>
</replicated-cache>
Error during startup:
15:07:13,942 ERROR [org.infinispan.topology.LocalTopologyManagerImpl]
(transport-thread--p14-t10) ISPN000230: Failed to start rebalance for cache
authenticationSessions: java.lang.ClassCastException:
org.infinispan.distribution.ch.impl.DefaultConsistentHash cannot be cast to
org.infinispan.distribution.ch.impl.ReplicatedConsistentHash
at
org.infinispan.distribution.ch.impl.SyncReplicatedConsistentHashFactory.union(SyncReplicatedConsistentHashFactory.java:26)
at
org.infinispan.topology.LocalTopologyManagerImpl.doHandleRebalance(LocalTopologyManagerImpl.java:512)
at
org.infinispan.topology.LocalTopologyManagerImpl.lambda$handleRebalance$3(LocalTopologyManagerImpl.java:475)
at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175)
at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:37)
at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:227)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
at java.lang.Thread.run(Thread.java:748)
Did I forgot one thing? How to get a real continuity of service with KeyCloak? Is
Standalone Cluster mode the good way, and how?
Thanks a lot for your help.
Patrice
________________________________
This message and any attachments are intended solely for the addressees and may contain
confidential information. Any unauthorized use or disclosure, either whole or partial, is
prohibited.
E-mails are susceptible to alteration. Our company shall not be liable for the message if
altered, changed or falsified. If you are not the intended recipient of this message,
please delete it and notify the sender.
Although all reasonable efforts have been made to keep this transmission free from
viruses, the sender will not be liable for damages caused by a transmitted virus.
12:18 p.m.
Hello Patrice,
Do you experience this even if the node is properly shutdown, i.e. via SIGTERM rather than
SIGKILL? Keycloak does seem to properly shutdown caches [1], which, according to
Infinispan doc [2], should result in a graceful leave.
Do you see messages like this in the log after shutdown has been triggered?
20:14:26,598 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 62)
WFLYCLINF0003: Stopped XXX cache from keycloak container
As for replicated caches, the issue might be triggered by mixing caches with different
synchronization modes. Could you try to explicitly specify mode="SYNC", like
here [3]?
[1]
https://github.com/keycloak/keycloak/blob/4.8.3.Final/model/infinispan/sr...
[2] https://infinispan.org/docs/stable/user_guide/user_guide.html#cache_manager
[3]
https://infinispan.org/docs/stable/user_guide/user_guide.html#cache_confi...
Regards,
Dmitry Telegin
Carretti Consulting OÜ | Keycloak Consulting and Training
Sepapaja 6, Tallinn 15551, Estonia | info(a)carretti.pro
On Mon, 2019-06-24 at 15:30 +0000, AMIEL Patrice wrote:
Hi all,
I'm trying to deploy KeyCloak (v 4.8.3-final) in Standalone Cluster mode in order to
answer a very specific requirement: get continuity of service 1/ in case of crash of a
KeyCloak instance, 2/ during the upgrade of my solution.
However, I unfortunately don't get such results :(, despite the fact the cluster
looks to be properly configured.
First, as I'm deploying KeyCloak in Kubernetes, I configured KeyCloak in Standalone
Cluster by using the DNS_PING and a TCP transport for JGroups. Instances of the cluster
can discover/see each other and the cluster is working fine as soon as it is used a couple
of seconds/minutes after starting the instances.
I've created a simple script that just get in a loop the JWT tokens using the Token
endpoint of a Realm, and I always have a 200 Ok status code whatever the KeyCloak instance
that is hit through the Kubernetes Service.
However, coming back to the 2 uses cases I'm interested in, it looks the KeyCloak
instances are getting crazy as soon as the cluster is not "stable". By
"cluster is not stable", I mean:
- When scaling down the number of Keycloak instances (whatever it is by killing
a Container or by a smart scale down of the Kubernetes Deployment)
- When performing a rolling update of the Pods
In both cases, during a particular time, most of the calls to get a JWT return a HTTP 499
status code and KeyCloak logs show the following:
08:14:56,785 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor]
(timeout-thread--p13-t1) ISPN000136: Error executing command GetKeyValueCommand, writing
keys ISPN000476: Timed out waiting for responses for request 1883 from
id-provider-5c55bbd99d-kqr8v
at
org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
at
org.infinispan.remoting.transport.jgroups.StaggeredRequest.onTimeout(StaggeredRequest.java:64)
at
org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
at
org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
08:14:56,790 ERROR [org.keycloak.services.error.KeycloakErrorHandler] (default task-992)
Uncaught server error: org.infinispan.util.concurrent.TimeoutException: ISPN000476:
Time-provider-5c55bbd99d-kqr8v
at
org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:472)
at
org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
at
org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
at
org.keycloak.models.sessions.infinispan.changes.InfinispanChangelogBasedTransaction.get(InfinispanChangelogBasedTransaction.java:120)
at
org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider.getLoginFailureEntity(InfinispanUserSessionProvider.java:678)
at
org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider.getUserLoginFailure(InfinispanUserSessionProvider.java:672)
at
org.keycloak.services.managers.DefaultBruteForceProtector.isTemporarilyDisabled(DefaultBruteForceProtector.java:306)
at
org.keycloak.authentication.authenticators.directgrant.ValidateUsername.authenticate(ValidateUsername.java:85)
at
org.keycloak.authentication.DefaultAuthenticationFlow.processFlow(DefaultAuthenticationFlow.java:221)
at
org.keycloak.authentication.DefaultAuthenticationFlow.processFlow(DefaultAuthenticationFlow.java:148)
at
org.keycloak.authentication.AuthenticationProcessor.authenticateOnly(AuthenticationProcessor.java:910)
at
org.keycloak.protocol.oidc.endpoints.TokenEndpoint.resourceOwnerPasswordCredentialsGrant(TokenEndpoint.java:554)
at
org.keycloak.protocol.oidc.endpoints.TokenEndpoint.processGrantRequest(TokenEndpoint.java:187)
at sun.reflect.GeneratedMethodAccessor449.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:140)
at
org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:509)
at
org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:399)
at
org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$0(ResourceMethodInvoker.java:363)
at
org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
at
org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:365)
at
org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:337)
at
org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:137)
at
org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:106)
at
org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:132)
at
org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:100)
at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:443)
at
org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:233)
at
org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:139)
at
org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
at
org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:142)
at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:219)
at
org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:791)
at
io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
at
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at
org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:90)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at
io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at
io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at
io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
at
io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at
org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
at
io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at
io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at
io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at
io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at
io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
at
io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
at
io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
at
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
at
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
at
io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at
io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at
org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at
io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
at
io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at
io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
at
org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at
org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting
for responses for request 1883 from id-provider-5c55bbd99d-kqr8v
at
org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
at
org.infinispan.remoting.transport.jgroups.StaggeredRequest.onTimeout(StaggeredRequest.java:64)
at
org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
at
org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Suppressed: org.infinispan.util.logging.TraceException
at
org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41)
at
org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:472)
at
org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
at
org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
at
org.keycloak.models.sessions.infinispan.changes.InfinispanChangelogBasedTransaction.get(InfinispanChangelogBasedTransaction.java:120)
at
org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider.getLoginFailureEntity(InfinispanUserSessionProvider.java:678)
at
org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider.getUserLoginFailure(InfinispanUserSessionProvider.java:672)
at
org.keycloak.services.managers.DefaultBruteForceProtector.isTemporarilyDisabled(DefaultBruteForceProtector.java:306)
at
org.keycloak.authentication.authenticators.directgrant.ValidateUsername.authenticate(ValidateUsername.java:85)
at
org.keycloak.authentication.DefaultAuthenticationFlow.processFlow(DefaultAuthenticationFlow.java:221)
at
org.keycloak.authentication.DefaultAuthenticationFlow.processFlow(DefaultAuthenticationFlow.java:148)
at
org.keycloak.authentication.AuthenticationProcessor.authenticateOnly(AuthenticationProcessor.java:910)
at
org.keycloak.protocol.oidc.endpoints.TokenEndpoint.resourceOwnerPasswordCredentialsGrant(TokenEndpoint.java:554)
at
org.keycloak.protocol.oidc.endpoints.TokenEndpoint.processGrantRequest(TokenEndpoint.java:187)
at sun.reflect.GeneratedMethodAccessor449.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:140)
at
org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:509)
at
org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:399)
at
org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$0(ResourceMethodInvoker.java:363)
at
org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
at
org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:365)
at
org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:337)
at
org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:137)
at
org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:106)
at
org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:132)
at
org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:100)
at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:443)
at
org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:233)
at
org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:139)
at
org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
at
org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:142)
at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:219)
at
org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:791)
at
io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
at
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at
org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:90)
at
io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at
io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at
io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at
io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
at
io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at
org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
at
io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at
io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at
io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at
io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at
io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
at
io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
at
io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
at
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
at
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
at
io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at
io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at
org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at
io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
at
io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at
io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
at
io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
at
org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at
org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
... 1 more
It looks that the KeyCloak instances that is still alive and receiving the call for
generation of the JWT is trying to contact the other instances in order to get some data
from the distributed cache... In particular, it tries to contact the instance that is no
longer here (because killed, shutdown, rolled...), received a Timeout error, and then
terminates the incoming request in error.
As I'm making the same request within a loop, I see the error happening during a
couple of seconds (around 10 to 15 secs), i.e. during the time the cluster composition is
not stabilized yet. When the re-discovery of the cluster has been performed, the new
composition of the cluster is updated, and everything goes back to normal!
I can understand, from KeyCloak documentation on server caches
(https://www.keycloak.org/docs/4.8/server_installation/index.html#_replica...) that
"By default, Keycloak only specifies one owner for data. So if that one node goes
down that data is lost. This usually means that users will be logged out and will have to
login again. You can change the number of nodes that replicate a piece of data by change
the owners attribute in the distributed-cache declaration.", but unfortunately,
setting the "owner" field to 2 (or more !) for all distributed caches does not
remove the issue.
I even tried to change the type of cache from "distributed-cache" to
"replicated-cache", but then KeyCloak is not starting:
<replicated-cache name="sessions"/>
<replicated-cache name="authenticationSessions"/>
<replicated-cache name="offlineSessions"/>
<replicated-cache name="clientSessions"/>
<replicated-cache name="offlineClientSessions"/>
<replicated-cache name="loginFailures"/>
<replicated-cache name="actionTokens">
<object-memory size="-1"/>
<expiration interval="300000" max-idle="-1"/>
</replicated-cache>
Error during startup:
15:07:13,942 ERROR [org.infinispan.topology.LocalTopologyManagerImpl]
(transport-thread--p14-t10) ISPN000230: Failed to start rebalance for cache
authenticationSessions: java.lang.ClassCastException:
org.infinispan.distribution.ch.impl.DefaultConsistentHash cannot be cast to
org.infinispan.distribution.ch.impl.ReplicatedConsistentHash
at
org.infinispan.distribution.ch.impl.SyncReplicatedConsistentHashFactory.union(SyncReplicatedConsistentHashFactory.java:26)
at
org.infinispan.topology.LocalTopologyManagerImpl.doHandleRebalance(LocalTopologyManagerImpl.java:512)
at
org.infinispan.topology.LocalTopologyManagerImpl.lambda$handleRebalance$3(LocalTopologyManagerImpl.java:475)
at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175)
at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:37)
at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:227)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
at java.lang.Thread.run(Thread.java:748)
Did I forgot one thing? How to get a real continuity of service with KeyCloak? Is
Standalone Cluster mode the good way, and how?
Thanks a lot for your help.
Patrice
________________________________
This message and any attachments are intended solely for the addressees and may contain
confidential information. Any unauthorized use or disclosure, either whole or partial, is
prohibited.
E-mails are susceptible to alteration. Our company shall not be liable for the message if
altered, changed or falsified. If you are not the intended recipient of this message,
please delete it and notify the sender.
Although all reasonable efforts have been made to keep this transmission free from
viruses, the sender will not be liable for damages caused by a transmitted virus.
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user
8:31 a.m.
Thanks Dimitri,
You are right, the shutdown was not working properly: the SIGTERM signal was not reaching
the JBoss process. To help other members of the community, here is what I did: I first set
the LAUNCH_JBOSS_IN_BACKGROUND envvar so that the JBoss java process that is launched by
the standalone.sh script is run in background. The "trap" commands are then
automatically set to forward the various signals received by the standalone.sh process to
the JBoss process.
I then updated my own Docker entrypoint shell script to have the same behavior (run
standalone.sh un background, get the process id, and define the traps). At the end,
SIGTERM sent by Kubernetes is correctly received by JBoss that exists properly the
cluster.
Now, the cluster is more stable (much less errors encountered when scaling down, no more
exception stack traces, and the cluster is not de-stabilized when an instance is stopped:
less API calls falling into error), but it is still not perfect : I still encounter some
errors during the scaling down transitions, even when setting the "owner" field
of the "distributed-cache" descriptions to "2".
Indeed, I realizes that most (almost all) of the requests are processed by a single
KeyCloak instance (often the oldest one) while I don't have any session affinity
(Session Affinity = None) on the Kubernetes Service: why ?
As a consequence, when killing ("kubectl delete pod") the instance that is
receiving (almost) none of the request, I don't have errors; but when deleting the
instance that is processing (almost) all the requests, then I have errors! Again, setting
the "owner" field of the "distributed-cache" description to
"2" (expecting the cache data are duplicated on the 2 nodes) does not change
things: I still have errors...
I tries to set the 'mode="SYNC"' like this:
<distributed-cache name="sessions" mode="SYNC"
owners="2"/>
But the process does not start, with the following error:
13:17:32,080 ERROR [org.jboss.as.controller] (Controller Boot Thread)
OPVDX001: Validation error in standalone-ha.xml --------------------------------
|
| 338: </local-cache>
| 339: <replicated-cache name="work"/>
| 340: <distributed-cache name="sessions" mode="SYNC"
owners="2"/>
| ^^^^ 'mode' isn't an allowed
attribute for the 'distributed-cache' element
|
| Attributes allowed here are:
capacity-factor,
| consistent-hash-strategy, l1-lifespan,
owners, segments
|
| 341: <distributed-cache name="authenticationSessions"
mode="SYNC" owners="2"/>
| 342: <distributed-cache name="offlineSessions" mode="SYNC"
owners="2"/>
| 343: <distributed-cache name="clientSessions" mode="SYNC"
owners="2"/>
|
| 'mode' is allowed on elements:
| - server > profile > {urn:jboss:domain:infinispan:7.0}subsystem >
cache-container > local-cache > transaction
Where should I put the "mode" instead?
Any idea on why I still have some errors?
Thanks for your valuable help.
Patrice
PS: By the way, I fear there is a bug in the standalone.sh (but I'm not sure): the
trap definitions are the following:
trap "kill -HUP $JBOSS_PID" HUP
trap "kill -TERM $JBOSS_PID" INT
trap "kill -QUIT $JBOSS_PID" QUIT
trap "kill -PIPE $JBOSS_PID" PIPE
trap "kill -TERM $JBOSS_PID" TERM
Each signal is forwarded to the JBoss process except the INT signal that is changed into a
TERM signal ? Shouldn't we have
trap "kill -INT $JBOSS_PID" INT
instead ?
-----Original Message-----
From: Dmitry Telegin [mailto:demetrio@carretti.pro]
Sent: lundi 24 juin 2019 19:19
To: AMIEL Patrice <Patrice.Amiel(a)gemalto.com>; keycloak-user(a)lists.jboss.org
Subject: Re: [keycloak-user] How to achieve "service continuity" with KeyCloak
in Standalone Cluster mode ?
Hello Patrice,
Do you experience this even if the node is properly shutdown, i.e. via SIGTERM rather than
SIGKILL? Keycloak does seem to properly shutdown caches [1], which, according to
Infinispan doc [2], should result in a graceful leave.
Do you see messages like this in the log after shutdown has been triggered?
20:14:26,598 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 62)
WFLYCLINF0003: Stopped XXX cache from keycloak container
As for replicated caches, the issue might be triggered by mixing caches with different
synchronization modes. Could you try to explicitly specify mode="SYNC", like
here [3]?
[1]
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub....
[2]
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Finfinis...
[3]
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Finfinis...
Regards,
Dmitry Telegin
Carretti Consulting OÜ | Keycloak Consulting and Training Sepapaja 6, Tallinn 15551,
Estonia | info(a)carretti.pro
On Mon, 2019-06-24 at 15:30 +0000, AMIEL Patrice wrote:
Hi all,
I'm trying to deploy KeyCloak (v 4.8.3-final) in Standalone Cluster mode in order to
answer a very specific requirement: get continuity of service 1/ in case of crash of a
KeyCloak instance, 2/ during the upgrade of my solution.
However, I unfortunately don't get such results :(, despite the fact the cluster
looks to be properly configured.
First, as I'm deploying KeyCloak in Kubernetes, I configured KeyCloak in Standalone
Cluster by using the DNS_PING and a TCP transport for JGroups. Instances of the cluster
can discover/see each other and the cluster is working fine as soon as it is used a couple
of seconds/minutes after starting the instances.
I've created a simple script that just get in a loop the JWT tokens using the Token
endpoint of a Realm, and I always have a 200 Ok status code whatever the KeyCloak instance
that is hit through the Kubernetes Service.
However, coming back to the 2 uses cases I'm interested in, it looks the KeyCloak
instances are getting crazy as soon as the cluster is not "stable". By
"cluster is not stable", I mean:
- When scaling down the number of Keycloak instances (whatever it is by killing
a Container or by a smart scale down of the Kubernetes Deployment)
- When performing a rolling update of the Pods
In both cases, during a particular time, most of the calls to get a JWT return a HTTP 499
status code and KeyCloak logs show the following:
08:14:56,785 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor]
(timeout-thread--p13-t1) ISPN000136: Error executing command GetKeyValueCommand, writing
keys ISPN000476: Timed out waiting for responses for request 1883 from
id-provider-5c55bbd99d-kqr8v
at
org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
at
org.infinispan.remoting.transport.jgroups.StaggeredRequest.onTimeout(StaggeredRequest.java:64)
at
org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
at
org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
08:14:56,790 ERROR [org.keycloak.services.error.KeycloakErrorHandler] (default task-992)
Uncaught server error: org.infinispan.util.concurrent.TimeoutException: ISPN000476:
Time-provider-5c55bbd99d-kqr8v
at
org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:472)
at
org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
at
org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
at
org.keycloak.models.sessions.infinispan.changes.InfinispanChangelogBasedTransaction.get(InfinispanChangelogBasedTransaction.java:120)
at
org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider.getLoginFailureEntity(InfinispanUserSessionProvider.java:678)
at
org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider.getUserLoginFailure(InfinispanUserSessionProvider.java:672)
at
org.keycloak.services.managers.DefaultBruteForceProtector.isTemporarilyDisabled(DefaultBruteForceProtector.java:306)
at
org.keycloak.authentication.authenticators.directgrant.ValidateUsername.authenticate(ValidateUsername.java:85)
at
org.keycloak.authentication.DefaultAuthenticationFlow.processFlow(DefaultAuthenticationFlow.java:221)
at
org.keycloak.authentication.DefaultAuthenticationFlow.processFlow(DefaultAuthenticationFlow.java:148)
at
org.keycloak.authentication.AuthenticationProcessor.authenticateOnly(AuthenticationProcessor.java:910)
at
org.keycloak.protocol.oidc.endpoints.TokenEndpoint.resourceOwnerPasswordCredentialsGrant(TokenEndpoint.java:554)
at
org.keycloak.protocol.oidc.endpoints.TokenEndpoint.processGrantRequest(TokenEndpoint.java:187)
at sun.reflect.GeneratedMethodAccessor449.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:140)
at
org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:509)
at
org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:399)
at
org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$0(ResourceMethodInvoker.java:363)
at
org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
at
org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:365)
at
org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:337)
at
org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:137)
at
org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:106)
at
org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:132)
at
org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:100)
at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:443)
at
org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:233)
at
org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:139)
at
org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
at
org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:142)
at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:219)
at
org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:791)
at
io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
at
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at
org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:90)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at
io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at
io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at
io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
at
io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at
org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
at
io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at
io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at
io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at
io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at
io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
at
io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
at
io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
at
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
at
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
at
io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at
io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at
org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at
io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
at
io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at
io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
at
org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at
org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting
for responses for request 1883 from id-provider-5c55bbd99d-kqr8v
at
org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
at
org.infinispan.remoting.transport.jgroups.StaggeredRequest.onTimeout(StaggeredRequest.java:64)
at
org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
at
org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Suppressed: org.infinispan.util.logging.TraceException
at
org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41)
at
org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:472)
at
org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
at
org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
at
org.keycloak.models.sessions.infinispan.changes.InfinispanChangelogBasedTransaction.get(InfinispanChangelogBasedTransaction.java:120)
at
org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider.getLoginFailureEntity(InfinispanUserSessionProvider.java:678)
at
org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider.getUserLoginFailure(InfinispanUserSessionProvider.java:672)
at
org.keycloak.services.managers.DefaultBruteForceProtector.isTemporarilyDisabled(DefaultBruteForceProtector.java:306)
at
org.keycloak.authentication.authenticators.directgrant.ValidateUsername.authenticate(ValidateUsername.java:85)
at
org.keycloak.authentication.DefaultAuthenticationFlow.processFlow(DefaultAuthenticationFlow.java:221)
at
org.keycloak.authentication.DefaultAuthenticationFlow.processFlow(DefaultAuthenticationFlow.java:148)
at
org.keycloak.authentication.AuthenticationProcessor.authenticateOnly(AuthenticationProcessor.java:910)
at
org.keycloak.protocol.oidc.endpoints.TokenEndpoint.resourceOwnerPasswordCredentialsGrant(TokenEndpoint.java:554)
at
org.keycloak.protocol.oidc.endpoints.TokenEndpoint.processGrantRequest(TokenEndpoint.java:187)
at sun.reflect.GeneratedMethodAccessor449.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:140)
at
org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:509)
at
org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:399)
at
org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$0(ResourceMethodInvoker.java:363)
at
org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
at
org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:365)
at
org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:337)
at
org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:137)
at
org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:106)
at
org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:132)
at
org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:100)
at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:443)
at
org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:233)
at
org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:139)
at
org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
at
org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:142)
at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:219)
at
org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:791)
at
io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
at
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at
org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:90)
at
io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at
io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at
io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at
io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
at
io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at
org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
at
io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at
io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at
io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at
io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at
io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
at
io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
at
io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
at
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
at
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
at
io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at
io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at
org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at
io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
at
io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at
io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
at
io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
at
org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at
org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
... 1 more
It looks that the KeyCloak instances that is still alive and receiving the call for
generation of the JWT is trying to contact the other instances in order to get some data
from the distributed cache... In particular, it tries to contact the instance that is no
longer here (because killed, shutdown, rolled...), received a Timeout error, and then
terminates the incoming request in error.
As I'm making the same request within a loop, I see the error happening during a
couple of seconds (around 10 to 15 secs), i.e. during the time the cluster composition is
not stabilized yet. When the re-discovery of the cluster has been performed, the new
composition of the cluster is updated, and everything goes back to normal!
I can understand, from KeyCloak documentation on server caches
(https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.key...)
that "By default, Keycloak only specifies one owner for data. So if that one node
goes down that data is lost. This usually means that users will be logged out and will
have to login again. You can change the number of nodes that replicate a piece of data by
change the owners attribute in the distributed-cache declaration.", but
unfortunately, setting the "owner" field to 2 (or more !) for all distributed
caches does not remove the issue.
I even tried to change the type of cache from "distributed-cache" to
"replicated-cache", but then KeyCloak is not starting:
<replicated-cache name="sessions"/>
<replicated-cache name="authenticationSessions"/> <replicated-cache
name="offlineSessions"/> <replicated-cache
name="clientSessions"/>
<replicated-cache name="offlineClientSessions"/> <replicated-cache
name="loginFailures"/> <replicated-cache
name="actionTokens">
<object-memory size="-1"/>
<expiration interval="300000" max-idle="-1"/>
</replicated-cache>
Error during startup:
15:07:13,942 ERROR [org.infinispan.topology.LocalTopologyManagerImpl]
(transport-thread--p14-t10) ISPN000230: Failed to start rebalance for cache
authenticationSessions: java.lang.ClassCastException:
org.infinispan.distribution.ch.impl.DefaultConsistentHash cannot be cast to
org.infinispan.distribution.ch.impl.ReplicatedConsistentHash
at
org.infinispan.distribution.ch.impl.SyncReplicatedConsistentHashFactory.union(SyncReplicatedConsistentHashFactory.java:26)
at
org.infinispan.topology.LocalTopologyManagerImpl.doHandleRebalance(LocalTopologyManagerImpl.java:512)
at
org.infinispan.topology.LocalTopologyManagerImpl.lambda$handleRebalance$3(LocalTopologyManagerImpl.java:475)
at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175)
at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:37)
at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:227)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
at java.lang.Thread.run(Thread.java:748)
Did I forgot one thing? How to get a real continuity of service with KeyCloak? Is
Standalone Cluster mode the good way, and how?
Thanks a lot for your help.
Patrice
________________________________
This message and any attachments are intended solely for the addressees and may contain
confidential information. Any unauthorized use or disclosure, either whole or partial, is
prohibited.
E-mails are susceptible to alteration. Our company shall not be liable for the message if
altered, changed or falsified. If you are not the intended recipient of this message,
please delete it and notify the sender.
Although all reasonable efforts have been made to keep this transmission free from
viruses, the sender will not be liable for damages caused by a transmitted virus.
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
s.jboss.org%2Fmailman%2Flistinfo%2Fkeycloak-user&data=02%7C01%7Cpa
trice.amiel%40gemalto.com%7Cd1eda3ff5bf74dbc390408d6f8c80987%7C37d0a9d
b7c464096bfe31add5b495d6d%7C0%7C1%7C636969935378606026&sdata=l0rOr
rqLoxwu3rTB0UJ4G6Cdqsf8mzzUt6VQWxx6%2FfU%3D&reserved=0
________________________________
This message and any attachments are intended solely for the addressees and may contain
confidential information. Any unauthorized use or disclosure, either whole or partial, is
prohibited.
E-mails are susceptible to alteration. Our company shall not be liable for the message if
altered, changed or falsified. If you are not the intended recipient of this message,
please delete it and notify the sender.
Although all reasonable efforts have been made to keep this transmission free from
viruses, the sender will not be liable for damages caused by a transmitted virus.
1784
days inactive
1791
days old
2 comments
2 participants
participants (2)
-
AMIEL Patrice -
Dmitry Telegin