[keycloak-user] Lost session when removing an instance off cluster
Stian Thorgersen
sthorger at redhat.com
Tue Jul 5 02:58:52 EDT 2016
There probably will be some errors from Infinispan when you remove a node
at least if there's ongoing replication/messages being sent to the cluster.
Can you try with the latest Keycloak and with the default clustering config
and only increasing owners? We have tested session failover ourselves and
it was working fine. So could be down to the way you've configured it or
could be an issue due to your network setup.
On 5 July 2016 at 08:50, Sarp Kaya <akaya at expedia.com> wrote:
> Hi Stian,
>
> You are right, it needs to have owner. By default the configuration had
> owners but as I experimented I forgot to add it. However adding owners does
> not fix the problem.
>
> Initially I also experimented this with offlineSessions and loginFailures
> being distributed-cache so logins were already failing with that (same
> issue, Internal Server Error being presented).
>
> Anyway, if worse case user needs to re-authenticate, then Internal Server
> Error problem should be fixed right? (it’s the below exception I have
> pasted) Because whenever there is org.jboss.resteasy.spi.UnhandledException
> then browsers will see 500, instead of logging in again (there could be
> something added as such as handling this exception and asking user to log
> in again).
>
> Kind Regards,
> Sarp Kaya
>
> From: Stian Thorgersen <sthorger at redhat.com>
> Reply-To: "stian at redhat.com" <stian at redhat.com>
> Date: Tuesday, July 5, 2016 at 4:41 PM
> To: Abdullah Sarp Kaya <akaya at expedia.com>
> Cc: "keycloak-user at lists.jboss.org" <keycloak-user at lists.jboss.org>
> Subject: Re: [keycloak-user] Lost session when removing an instance off
> cluster
>
> The sessions cache is a distributed-cache, but by default it only has 1
> owner of each segment. This means that sessions are not replicated to
> multiple nodes. So it's expected that you loose sessions when a node is
> removed. If you want sessions to survive the removal of a node you should
> increase the number of owners for the cache. For example:
>
> <distributed-cache name="sessions" mode="SYNC" owners="2"/>
>
> We've not actually tried with enabling transactions on the sessions, but I
> would say performance is more important. Worst case scenario is that a
> session is lost and a user has to re-authenticate.
>
> BTW the config you have for offlineSessions and loginFailures are wrong
> they both need to be distributed caches as well in clustered mode:
>
> <distributed-cache name="offlineSessions" mode="SYNC" owners="1"/>
> <distributed-cache name="loginFailures" mode="SYNC" owners="1"/>
>
>
> On 4 July 2016 at 15:45, Sarp Kaya <akaya at expedia.com> wrote:
>
>> Just to add on this.
>>
>> I think I misunderstood something or Keycloak does not really implement
>> the infinispan caching lifetime correctly.
>>
>> So first thing I did is to TRACE the infinispan logs. So I added this
>> logger in standalone.xml:
>>
>> <logger category="org.infinispan">
>> <level name="TRACE"/>
>> </logger>
>>
>>
>> The next thing I did is to do the timing. Basically my test is that what
>> happens when I log in. I also enabled events and printing the events as
>> well.
>>
>> What I found is:
>>
>>
>> 1. Login event is triggered at 13:18:35,193 in Keycloak.
>> 2. The browser got the full response at 13:18:35,273
>> 3. Now the surprising part is the fact that there are whole a lot of
>> infinispan logging out occurring for “sessions” cache after the browser got
>> the response. The last log message occurring is like this:
>>
>> 2016-07-04 13:18:35,345 TRACE [org.infinispan.remoting.rpc.RpcManagerImpl] (default task-4) Response(s) to TxCompletionNotificationCommand{ xid=null, internalId=0, topologyId=19, gtx=GlobalTransaction:<kc1>:239:local, cacheName=sessions} is {}
>>
>>
>> To see the all the logs after event trigger please check here:
>> <http://pastebin.com/pkvYP07T> <http://pastebin.com/pkvYP07T>
>> <http://pastebin.com/pkvYP07T> <http://pastebin.com/pkvYP07T>
>> http://pastebin.com/p4K2Ghff
>>
>> So my understanding was that I marked sessions as SYNC, which meant to me
>> that it is supposed to do all of the synchronization before it sends the
>> response back to the browser. If it does the other way around then it
>> should be ASYNC mode. Could you please clarify the intended/expected
>> behavior?
>>
>> Kind Regards,
>> Sarp Kaya
>>
>>
>> From: Abdullah Sarp Kaya <akaya at expedia.com>
>> Date: Monday, July 4, 2016 at 9:59 PM
>> To: "stian at redhat.com" <stian at redhat.com>
>> Cc: "keycloak-user at lists.jboss.org" <keycloak-user at lists.jboss.org>
>> Subject: Re: [keycloak-user] Lost session when removing an instance off
>> cluster
>>
>> Hello,
>>
>> Below is exactly how Infinispan is configured:
>>
>> <subsystem xmlns="urn:jboss:domain:infinispan:4.0">
>> <cache-container name="keycloak" jndi-name="infinispan/Keycloak">
>> <transport lock-timeout="60000"/>
>> <invalidation-cache name="realms" mode="SYNC"/>
>> <invalidation-cache name="users" mode="SYNC">
>> <eviction max-entries="10000" strategy="LRU"/>
>> </invalidation-cache>
>> <distributed-cache xmlns="" name="sessions" mode="SYNC">
>> <transaction mode="NON_DURABLE_XA"/>
>> </distributed-cache>
>> <local-cache xmlns="" name="offlineSessions"/>
>> <local-cache xmlns="" name="loginFailures"/>
>> <replicated-cache name="work" mode="SYNC"/>
>> <local-cache name="realmVersions">
>> <transaction mode="BATCH" locking="PESSIMISTIC"/>
>> </local-cache>
>> </cache-container>
>> <cache-container name="server" aliases="singleton cluster" default-cache="default" module="org.wildfly.clustering.server">
>> <transport lock-timeout="60000"/>
>> <replicated-cache name="default" mode="SYNC">
>> <transaction mode="BATCH"/>
>> </replicated-cache>
>> </cache-container>
>> <cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">
>> <transport lock-timeout="60000"/>
>> <distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
>> <locking isolation="REPEATABLE_READ"/>
>> <transaction mode="BATCH"/>
>> <file-store/>
>> </distributed-cache>
>> </cache-container>
>> <cache-container name="ejb" aliases="sfsb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan">
>> <transport lock-timeout="60000"/>
>> <distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
>> <locking isolation="REPEATABLE_READ"/>
>> <transaction mode="BATCH"/>
>> <file-store/>
>> </distributed-cache>
>> </cache-container>
>> <cache-container name="hibernate" default-cache="local-query" module="org.hibernate.infinispan">
>> <transport lock-timeout="60000"/>
>> <invalidation-cache name="entity" mode="SYNC">
>> <transaction mode="NON_XA"/>
>> <eviction strategy="LRU" max-entries="10000"/>
>> <expiration max-idle="100000"/>
>> </invalidation-cache>
>> <local-cache name="local-query">
>> <eviction strategy="LRU" max-entries="10000"/>
>> <expiration max-idle="100000"/>
>> </local-cache>
>> <replicated-cache name="timestamps" mode="ASYNC"/>
>> </cache-container>
>> </subsystem>
>>
>>
>>
>>
>> This is the complete exception:
>>
>> 03:49:40,241 ERROR
>> [org.infinispan.interceptors.InvocationContextInterceptor] (default
>> task-12) ISPN000136: Error executing command PutKeyValueCommand, writing
>> keys [ba74e523-3299-42b3-801e-9f8cb5a0d81d]:
>> org.infinispan.util.concurrent.TimeoutException: Replication timeout for kc2
>> at
>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
>> at
>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612)
>> at
>> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
>> at
>> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
>> at
>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>> at
>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>> at
>> org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
>> at
>> org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> 03:49:40,250 ERROR [org.jboss.resteasy.resteasy_jaxrs.i18n] (default
>> task-12) RESTEASY002025: Unknown exception while executing GET
>> /realms/partner-dmz/protocol/openid-connect/auth:
>> org.infinispan.util.concurrent.TimeoutException: Replication timeout for kc2
>> at
>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
>> at
>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612)
>> at
>> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
>> at
>> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
>> at
>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>> at
>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>> at
>> org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
>> at
>> org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> 03:49:40,257 ERROR [io.undertow.request] (default task-12) UT005023:
>> Exception handling request to
>> /auth/realms/partner-dmz/protocol/openid-connect/auth:
>> org.jboss.resteasy.spi.UnhandledException:
>> org.infinispan.util.concurrent.TimeoutException: Replication timeout for kc2
>> at
>> org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:247)
>> at
>> org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:168)
>> at
>> org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:471)
>> at
>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:415)
>> at
>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:202)
>> at
>> org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221)
>> at
>> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
>> at
>> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>> at
>> io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
>> at
>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
>> at
>> org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:88)
>> at
>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
>> at
>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
>> at
>> io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
>> at
>> io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
>> at
>> io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
>> at
>> org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
>> at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at
>> io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
>> at
>> io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
>> at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at
>> io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
>> at
>> io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
>> at
>> io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
>> at
>> io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
>> at
>> io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
>> at
>> io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
>> at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at
>> org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
>> at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at
>> io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)
>> at
>> io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
>> at
>> io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
>> at
>> io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
>> at
>> io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
>> at
>> io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.infinispan.util.concurrent.TimeoutException: Replication
>> timeout for kc2
>> at
>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
>> at
>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612)
>> at
>> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
>> at
>> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
>> at
>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>> at
>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>> at
>> org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
>> at
>> org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> ... 3 more
>>
>> 03:49:41,251 ERROR
>> [org.infinispan.interceptors.InvocationContextInterceptor] (default task-3)
>> ISPN000136: Error executing command PrepareCommand, writing keys
>> [02cff8bb-0ab9-4ab9-905f-17ffb3226050]:
>> org.infinispan.util.concurrent.TimeoutException: Replication timeout for kc2
>> at
>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
>> at
>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612)
>> at
>> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
>> at
>> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
>> at
>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>> at
>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>> at
>> org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
>> at
>> org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> 03:49:41,258 ERROR
>> [org.infinispan.transaction.impl.TransactionCoordinator] (default task-3)
>> ISPN000097: Error while processing a prepare in a single-phase transaction:
>> org.infinispan.util.concurrent.TimeoutException: Replication timeout for kc2
>> at
>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
>> at
>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612)
>> at
>> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
>> at
>> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
>> at
>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>> at
>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>> at
>> org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
>> at
>> org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> 03:49:45,504 INFO
>> [org.infinispan.remoting.transport.jgroups.JGroupsTransport]
>> (Incoming-3,dev,kc1) ISPN000094: Received new cluster view for channel
>> server: [kc1|4] (1) [kc1]
>> 03:49:45,507 INFO
>> [org.infinispan.remoting.transport.jgroups.JGroupsTransport]
>> (Incoming-3,dev,kc1) ISPN000094: Received new cluster view for channel
>> keycloak: [kc1|4] (1) [kc1]
>> 03:49:45,534 INFO
>> [org.infinispan.remoting.transport.jgroups.JGroupsTransport]
>> (Incoming-3,dev,kc1) ISPN000094: Received new cluster view for channel web:
>> [kc1|4] (1) [kc1]
>> 03:49:45,538 INFO
>> [org.infinispan.remoting.transport.jgroups.JGroupsTransport]
>> (Incoming-3,dev,kc1) ISPN000094: Received new cluster view for channel ejb:
>> [kc1|4] (1) [kc1]
>> 03:49:45,548 WARN [com.arjuna.ats.jta] (default task-3) ARJUNA016039:
>> onePhaseCommit on < formatId=131077, gtrid_length=29, bqual_length=36,
>> tx_uid=0:ffffac110003:-7fcaa87f:5779dc60:f01, node_name=1,
>> branch_uid=0:ffffac110003:-7fcaa87f:5779dc60:f02, subordinatenodename=null,
>> eis_name=unknown eis name >
>> (TransactionXaAdapter{localTransaction=LocalXaTransaction{xid=<
>> formatId=131077, gtrid_length=29, bqual_length=36,
>> tx_uid=0:ffffac110003:-7fcaa87f:5779dc60:f01, node_name=1,
>> branch_uid=0:ffffac110003:-7fcaa87f:5779dc60:f02, subordinatenodename=null,
>> eis_name=unknown eis name >} LocalTransaction{remoteLockedNodes=null,
>> isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null,
>> topologyId=6, stateTransferFlag=null}
>> org.infinispan.transaction.xa.LocalXaTransaction at 3ef}) failed with
>> exception XAException.XA_HEURRB: javax.transaction.xa.XAException
>> at
>> org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:213)
>> at
>> org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159)
>> at
>> org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:112)
>> at
>> com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:704)
>> at
>> com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2366)
>> at
>> com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1495)
>> at
>> com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96)
>> at
>> com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
>> at
>> com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)
>> at
>> com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
>> at
>> com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
>> at
>> org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1722)
>> at
>> org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1679)
>> at
>> org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1121)
>> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1111)
>> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1742)
>> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:248)
>> at
>> org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:291)
>> at
>> org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider$InfinispanKeycloakTransaction$CacheTask.execute(InfinispanUserSessionProvider.java:854)
>> at
>> org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider$InfinispanKeycloakTransaction.commit(InfinispanUserSessionProvider.java:750)
>> at
>> org.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:104)
>> at
>> org.keycloak.services.filters.KeycloakTransactionCommitter.filter(KeycloakTransactionCommitter.java:43)
>> at
>> org.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:121)
>> at
>> org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:48)
>> at
>> org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:466)
>> at
>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:415)
>> at
>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:202)
>> at
>> org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221)
>> at
>> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
>> at
>> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>> at
>> io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
>> at
>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
>> at
>> org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:88)
>> at
>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
>> at
>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
>> at
>> io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
>> at
>> io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
>> at
>> io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
>> at
>> org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
>> at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at
>> io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
>> at
>> io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
>> at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at
>> io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
>> at
>> io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
>> at
>> io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
>> at
>> io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
>> at
>> io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
>> at
>> io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
>> at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at
>> org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
>> at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at
>> io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)
>> at
>> io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
>> at
>> io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
>> at
>> io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
>> at
>> io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
>> at
>> io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.infinispan.util.concurrent.TimeoutException: Replication
>> timeout for kc2
>> at
>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
>> at
>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612)
>> at
>> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
>> at
>> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
>> at
>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>> at
>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>> at
>> org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
>> at
>> org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> ... 3 more
>>
>> 03:49:45,563 WARN [org.infinispan.cache.impl.CacheImpl] (default task-3)
>> ISPN000160: Could not complete injected transaction.:
>> javax.transaction.RollbackException: ARJUNA016053: Could not commit
>> transaction.
>> at
>> com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1212)
>> at
>> com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
>> at
>> com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
>> at
>> org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1722)
>> at
>> org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1679)
>> at
>> org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1121)
>> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1111)
>> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1742)
>> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:248)
>> at
>> org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:291)
>> at
>> org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider$InfinispanKeycloakTransaction$CacheTask.execute(InfinispanUserSessionProvider.java:854)
>> at
>> org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider$InfinispanKeycloakTransaction.commit(InfinispanUserSessionProvider.java:750)
>> at
>> org.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:104)
>> at
>> org.keycloak.services.filters.KeycloakTransactionCommitter.filter(KeycloakTransactionCommitter.java:43)
>> at
>> org.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:121)
>> at
>> org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:48)
>> at
>> org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:466)
>> at
>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:415)
>> at
>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:202)
>> at
>> org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221)
>> at
>> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
>> at
>> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>> at
>> io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
>> at
>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
>> at
>> org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:88)
>> at
>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
>> at
>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
>> at
>> io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
>> at
>> io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
>> at
>> io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
>> at
>> org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
>> at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at
>> io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
>> at
>> io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
>> at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at
>> io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
>> at
>> io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
>> at
>> io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
>> at
>> io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
>> at
>> io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
>> at
>> io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
>> at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at
>> org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
>> at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at
>> io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)
>> at
>> io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
>> at
>> io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
>> at
>> io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
>> at
>> io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
>> at
>> io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> Suppressed: javax.transaction.xa.XAException
>> at
>> org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:213)
>> at
>> org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159)
>> at
>> org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:112)
>> at
>> com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:704)
>> at
>> com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2366)
>> at
>> com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1495)
>> at
>> com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96)
>> at
>> com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
>> at
>> com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)
>> ... 54 more
>> Caused by: org.infinispan.util.concurrent.TimeoutException:
>> Replication timeout for kc2
>> at
>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
>> at
>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612)
>> at
>> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
>> at
>> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
>> at
>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>> at
>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>> at
>> org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
>> at
>> org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
>> at
>> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> ... 3 more
>>
>> 03:49:45,575 INFO
>> [org.infinispan.remoting.transport.jgroups.JGroupsTransport]
>> (Incoming-3,dev,kc1) ISPN000094: Received new cluster view for channel
>> hibernate: [kc1|4] (1) [kc1]
>> 03:49:45,575 ERROR [org.jboss.resteasy.resteasy_jaxrs.i18n] (default
>> task-3) RESTEASY002025: Unknown exception while executing GET
>> /realms/partner-dmz/protocol/openid-connect/auth:
>> org.infinispan.commons.CacheException: Could not commit implicit transaction
>> at
>> org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1725)
>> at
>> org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1679)
>> at
>> org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1121)
>> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1111)
>> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1742)
>> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:248)
>> at
>> org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:291)
>> at
>> org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider$InfinispanKeycloakTransaction$CacheTask.execute(InfinispanUserSessionProvider.java:854)
>> at
>> org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider$InfinispanKeycloakTransaction.commit(InfinispanUserSessionProvider.java:750)
>> at
>> org.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:104)
>> at
>> org.keycloak.services.filters.KeycloakTransactionCommitter.filter(KeycloakTransactionCommitter.java:43)
>> at
>> org.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:121)
>> at
>> org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:48)
>> at
>> org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:466)
>> at
>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:415)
>> at
>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:202)
>> at
>> org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221)
>> at
>> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
>> at
>> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>> at
>> io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
>> at
>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
>> at
>> org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:88)
>> at
>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
>> at
>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
>> at
>> io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
>> at
>> io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
>> at
>> io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
>> at
>> org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
>> at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at
>> io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
>> at
>> io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
>> at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at
>> io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
>> at
>> io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
>> at
>> io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
>> at
>> io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
>> at
>> io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
>> at
>> io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
>> at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at
>> org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
>> at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at
>> io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)
>> at
>> io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
>> at
>> io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
>> at
>> io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
>> at
>> io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
>> at
>> io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not
>> commit transaction.
>> at
>> com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1212)
>> at
>> com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
>> at
>> com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
>> at
>> org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1722)
>> ... 51 more
>> Suppressed: javax.transaction.xa.XAException
>> at
>> org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:213)
>> at
>> org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159)
>> at
>> org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:112)
>> at
>> com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:704)
>> at
>> com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2366)
>> at
>> com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1495)
>> at
>> com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96)
>> at
>> com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
>> at
>> com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)
>> ... 54 more
>> Caused by: org.infinispan.util.concurrent.TimeoutException:
>> Replication timeout for kc2
>> at
>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
>> at
>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612)
>> at
>> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
>> at
>> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
>> at
>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>> at
>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>> at
>> org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
>> at
>> org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
>> at
>> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> ... 3 more
>>
>> 03:49:45,596 ERROR [io.undertow.request] (default task-3) UT005023:
>> Exception handling request to
>> /auth/realms/partner-dmz/protocol/openid-connect/auth:
>> org.jboss.resteasy.spi.UnhandledException:
>> org.infinispan.commons.CacheException: Could not commit implicit transaction
>> at
>> org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:247)
>> at
>> org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:168)
>> at
>> org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:471)
>> at
>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:415)
>> at
>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:202)
>> at
>> org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221)
>> at
>> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
>> at
>> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>> at
>> io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
>> at
>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
>> at
>> org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:88)
>> at
>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
>> at
>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
>> at
>> io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
>> at
>> io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
>> at
>> io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
>> at
>> org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
>> at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at
>> io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
>> at
>> io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
>> at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at
>> io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
>> at
>> io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
>> at
>> io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
>> at
>> io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
>> at
>> io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
>> at
>> io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
>> at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at
>> org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
>> at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at
>> io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)
>> at
>> io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
>> at
>> io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
>> at
>> io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
>> at
>> io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
>> at
>> io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.infinispan.commons.CacheException: Could not commit
>> implicit transaction
>> at
>> org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1725)
>> at
>> org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1679)
>> at
>> org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1121)
>> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1111)
>> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1742)
>> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:248)
>> at
>> org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:291)
>> at
>> org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider$InfinispanKeycloakTransaction$CacheTask.execute(InfinispanUserSessionProvider.java:854)
>> at
>> org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider$InfinispanKeycloakTransaction.commit(InfinispanUserSessionProvider.java:750)
>> at
>> org.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:104)
>> at
>> org.keycloak.services.filters.KeycloakTransactionCommitter.filter(KeycloakTransactionCommitter.java:43)
>> at
>> org.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:121)
>> at
>> org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:48)
>> at
>> org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:466)
>> ... 38 more
>> Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not
>> commit transaction.
>> at
>> com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1212)
>> at
>> com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
>> at
>> com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
>> at
>> org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1722)
>> ... 51 more
>> Suppressed: javax.transaction.xa.XAException
>> at
>> org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:213)
>> at
>> org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159)
>> at
>> org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:112)
>> at
>> com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:704)
>> at
>> com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2366)
>> at
>> com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1495)
>> at
>> com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96)
>> at
>> com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
>> at
>> com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)
>> ... 54 more
>> Caused by: org.infinispan.util.concurrent.TimeoutException:
>> Replication timeout for kc2
>> at
>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
>> at
>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612)
>> at
>> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
>> at
>> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
>> at
>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>> at
>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>> at
>> org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
>> at
>> org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
>> at
>> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> ... 3 more
>>
>>
>>
>> Here kc2 is the removed instance.
>>
>> Kind Regards,
>> Sarp Kaya
>>
>>
>> From: Stian Thorgersen <sthorger at redhat.com>
>> Reply-To: "stian at redhat.com" <stian at redhat.com>
>> Date: Friday, July 1, 2016 at 7:17 PM
>> To: Abdullah Sarp Kaya <akaya at expedia.com>
>> Cc: "keycloak-user at lists.jboss.org" <keycloak-user at lists.jboss.org>
>> Subject: Re: [keycloak-user] Lost session when removing an instance off
>> cluster
>>
>> Can you please include more details from the log if there is any, at
>> least a full stack trace and not just the bit you've included. We also need
>> to know details around how you've configured the caches and Infinispan.
>>
>> On 1 July 2016 at 10:16, Sarp Kaya <akaya at expedia.com> wrote:
>>
>>> Hello,
>>>
>>> I have tried various ways of configuring infinispan but it just seems
>>> like if I deploy a new instance to the cluster and remove one, then some
>>> sessions are lost and an exception is thrown saying that it was not
>>> handled. This is the Infinispan exception:
>>>
>>> Exceptionhandlingrequestto/auth/realms/realmname/protocol/openid-connect
>>> /auth:org.jboss.resteasy.spi.UnhandledException:org.infinispan.util.
>>> concurrent.TimeoutException:Replicationtimeoutfor79a0757ecab3atorg.jboss
>>> .resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:
>>> 247) atorg.jboss.resteasy.core.SynchronousDispatcher.writeException(
>>> SynchronousDispatcher.java:168) atorg.jboss.resteasy.core.
>>> SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:471) at
>>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(
>>> SynchronousDispatcher.java:415) atorg.jboss.resteasy.core.
>>> SynchronousDispatcher.invoke(SynchronousDispatcher.java:202) atorg.jboss
>>> .resteasy.plugins.server.servlet.ServletContainerDispatcher.service(
>>> ServletContainerDispatcher.java:221) atorg.jboss.resteasy.plugins.server
>>> .servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) at
>>> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(
>>> HttpServletDispatcher.java:51) atjavax.servlet.http.HttpServlet.service(
>>> HttpServlet.java:790) atio.undertow.servlet.handlers.ServletHandler.
>>> handleRequest(ServletHandler.java:85) atio.undertow.servlet.handlers.
>>> FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) atorg.
>>> keycloak.services.filters.KeycloakSessionServletFilter.doFilter(
>>> KeycloakSessionServletFilter.java:88) atio.undertow.servlet.core.
>>> ManagedFilter.doFilter(ManagedFilter.java:60) atio.undertow.servlet.
>>> handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
>>> atio.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler
>>> .java:84) atio.undertow.servlet.handlers.security.
>>> ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java
>>> :62) atio.undertow.servlet.handlers.ServletDispatchingHandler.
>>> handleRequest(ServletDispatchingHandler.java:36) atorg.wildfly.extension
>>> .undertow.security.SecurityContextAssociationHandler.handleRequest(
>>> SecurityContextAssociationHandler.java:78) atio.undertow.server.handlers
>>> .PredicateHandler.handleRequest(PredicateHandler.java:43) atio.undertow.
>>> servlet.handlers.security.SSLInformationAssociationHandler.handleRequest
>>> (SSLInformationAssociationHandler.java:131) atio.undertow.servlet.
>>> handlers.security.ServletAuthenticationCallHandler.handleRequest(
>>> ServletAuthenticationCallHandler.java:57) atio.undertow.server.handlers.
>>> PredicateHandler.handleRequest(PredicateHandler.java:43) atio.undertow.
>>> security.handlers.AbstractConfidentialityHandler.handleRequest(
>>> AbstractConfidentialityHandler.java:46) atio.undertow.servlet.handlers.
>>> security.ServletConfidentialityConstraintHandler.handleRequest(
>>> ServletConfidentialityConstraintHandler.java:64) atio.undertow.security.
>>> handlers.AuthenticationMechanismsHandler.handleRequest(
>>> AuthenticationMechanismsHandler.java:60) atio.undertow.servlet.handlers.
>>> security.CachedAuthenticatedSessionHandler.handleRequest(
>>> CachedAuthenticatedSessionHandler.java:77) atio.undertow.security.
>>> handlers.NotificationReceiverHandler.handleRequest(
>>> NotificationReceiverHandler.java:50) atio.undertow.security.handlers.
>>> AbstractSecurityContextAssociationHandler.handleRequest(
>>> AbstractSecurityContextAssociationHandler.java:43) atio.undertow.server.
>>> handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) atorg.
>>> wildfly.extension.undertow.security.jacc.JACCContextIdHandler.
>>> handleRequest(JACCContextIdHandler.java:61) atio.undertow.server.
>>> handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) atio.
>>> undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler
>>> .java:43) atio.undertow.servlet.handlers.ServletInitialHandler.
>>> handleFirstRequest(ServletInitialHandler.java:284) atio.undertow.servlet
>>> .handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.
>>> java:263) atio.undertow.servlet.handlers.ServletInitialHandler.access$
>>> 000(ServletInitialHandler.java:81) atio.undertow.servlet.handlers.
>>> ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174) at
>>> io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) at
>>> io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
>>> atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
>>> java:1142) atjava.util.concurrent.ThreadPoolExecutor$Worker.run(
>>> ThreadPoolExecutor.java:617) atjava.lang.Thread.run(Thread.java:745)
>>> Causedby:org.infinispan.util.concurrent.TimeoutException:Replication
>>> timeoutfor79a0757ecab3atorg.infinispan.remoting.transport.jgroups.
>>> JGroupsTransport.checkRsp(JGroupsTransport.java:765) atorg.infinispan.
>>> remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$
>>> 72(JGroupsTransport.java:599) atjava.util.concurrent.CompletableFuture.
>>> uniApply(CompletableFuture.java:602) atjava.util.concurrent.
>>> CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) atjava.
>>> util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:
>>> 474) atjava.util.concurrent.CompletableFuture.complete(CompletableFuture
>>> .java:1962) atorg.infinispan.remoting.transport.jgroups.
>>> SingleResponseFuture.call(SingleResponseFuture.java:46) atorg.infinispan
>>> .remoting.transport.jgroups.SingleResponseFuture.call(
>>> SingleResponseFuture.java:17) atjava.util.concurrent.FutureTask.run(
>>> FutureTask.java:266) atjava.util.concurrent.ScheduledThreadPoolExecutor$
>>> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>
>>> This causes browsers to see Internal Server Error. Shouldn’t that be
>>> handled in Keycloak as lost session, therefore KC should try to handle it
>>> rather than showing that it’s an Internal Server Error?
>>>
>>> My current infinispan configuration looks like this:
>>>
>>> <distributed-cache name="sessions" mode="SYNC">
>>> <transaction mode="NON_DURABLE_XA"/>
>>> </distributed-cache>
>>>
>>>
>>> I use Keycloak version 1.9.5. My question is am I doing something wrong
>>> with my configuration? I tried both replicated-cache and distributed-cache
>>> and tried all transaction mode on both of them. None of them seems to solve
>>> the error that I’ve had above.
>>>
>>> Kind Regards,
>>> Sarp Kaya
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> keycloak-user mailing list
>>> keycloak-user at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/keycloak-user
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-user/attachments/20160705/dd12c2d3/attachment-0001.html
More information about the keycloak-user
mailing list