[keycloak-user] Lost session when removing an instance off cluster
Sarp Kaya
akaya at expedia.com
Tue Jul 5 03:10:41 EDT 2016
Hi Stian,
I have tested with owners being 4 and had 4 instances, removed 1 off and still had the issue. I’ll try it with version 2.0.0 with 4 owners in sessions and the rest with 1 owners. I don’t have any network issues because it’s all tested locally.
Kind Regards,
Sarp Kaya
From: Stian Thorgersen <sthorger at redhat.com<mailto:sthorger at redhat.com>>
Reply-To: "stian at redhat.com<mailto:stian at redhat.com>" <stian at redhat.com<mailto:stian at redhat.com>>
Date: Tuesday, July 5, 2016 at 4:58 PM
To: Abdullah Sarp Kaya <akaya at expedia.com<mailto:akaya at expedia.com>>
Cc: "keycloak-user at lists.jboss.org<mailto:keycloak-user at lists.jboss.org>" <keycloak-user at lists.jboss.org<mailto:keycloak-user at lists.jboss.org>>
Subject: Re: [keycloak-user] Lost session when removing an instance off cluster
There probably will be some errors from Infinispan when you remove a node at least if there's ongoing replication/messages being sent to the cluster.
Can you try with the latest Keycloak and with the default clustering config and only increasing owners? We have tested session failover ourselves and it was working fine. So could be down to the way you've configured it or could be an issue due to your network setup.
On 5 July 2016 at 08:50, Sarp Kaya <akaya at expedia.com<mailto:akaya at expedia.com>> wrote:
Hi Stian,
You are right, it needs to have owner. By default the configuration had owners but as I experimented I forgot to add it. However adding owners does not fix the problem.
Initially I also experimented this with offlineSessions and loginFailures being distributed-cache so logins were already failing with that (same issue, Internal Server Error being presented).
Anyway, if worse case user needs to re-authenticate, then Internal Server Error problem should be fixed right? (it’s the below exception I have pasted) Because whenever there is org.jboss.resteasy.spi.UnhandledException then browsers will see 500, instead of logging in again (there could be something added as such as handling this exception and asking user to log in again).
Kind Regards,
Sarp Kaya
From: Stian Thorgersen <sthorger at redhat.com<mailto:sthorger at redhat.com>>
Reply-To: "stian at redhat.com<mailto:stian at redhat.com>" <stian at redhat.com<mailto:stian at redhat.com>>
Date: Tuesday, July 5, 2016 at 4:41 PM
To: Abdullah Sarp Kaya <akaya at expedia.com<mailto:akaya at expedia.com>>
Cc: "keycloak-user at lists.jboss.org<mailto:keycloak-user at lists.jboss.org>" <keycloak-user at lists.jboss.org<mailto:keycloak-user at lists.jboss.org>>
Subject: Re: [keycloak-user] Lost session when removing an instance off cluster
The sessions cache is a distributed-cache, but by default it only has 1 owner of each segment. This means that sessions are not replicated to multiple nodes. So it's expected that you loose sessions when a node is removed. If you want sessions to survive the removal of a node you should increase the number of owners for the cache. For example:
<distributed-cache name="sessions" mode="SYNC" owners="2"/>
We've not actually tried with enabling transactions on the sessions, but I would say performance is more important. Worst case scenario is that a session is lost and a user has to re-authenticate.
BTW the config you have for offlineSessions and loginFailures are wrong they both need to be distributed caches as well in clustered mode:
<distributed-cache name="offlineSessions" mode="SYNC" owners="1"/>
<distributed-cache name="loginFailures" mode="SYNC" owners="1"/>
On 4 July 2016 at 15:45, Sarp Kaya <akaya at expedia.com<mailto:akaya at expedia.com>> wrote:
Just to add on this.
I think I misunderstood something or Keycloak does not really implement the infinispan caching lifetime correctly.
So first thing I did is to TRACE the infinispan logs. So I added this logger in standalone.xml:
<logger category="org.infinispan">
<level name="TRACE"/>
</logger>
The next thing I did is to do the timing. Basically my test is that what happens when I log in. I also enabled events and printing the events as well.
What I found is:
1. Login event is triggered at 13:18:35,193 in Keycloak.
2. The browser got the full response at 13:18:35,273
3. Now the surprising part is the fact that there are whole a lot of infinispan logging out occurring for “sessions” cache after the browser got the response. The last log message occurring is like this:
2016-07-04 13:18:35,345 TRACE [org.infinispan.remoting.rpc.RpcManagerImpl] (default task-4) Response(s) to TxCompletionNotificationCommand{ xid=null, internalId=0, topologyId=19, gtx=GlobalTransaction:<kc1>:239:local, cacheName=sessions} is {}
To see the all the logs after event trigger please check here:
<http://pastebin.com/pkvYP07T><http://pastebin.com/pkvYP07T><http://pastebin.com/pkvYP07T><http://pastebin.com/pkvYP07T>http://pastebin.com/p4K2Ghff
So my understanding was that I marked sessions as SYNC, which meant to me that it is supposed to do all of the synchronization before it sends the response back to the browser. If it does the other way around then it should be ASYNC mode. Could you please clarify the intended/expected behavior?
Kind Regards,
Sarp Kaya
From: Abdullah Sarp Kaya <akaya at expedia.com<mailto:akaya at expedia.com>>
Date: Monday, July 4, 2016 at 9:59 PM
To: "stian at redhat.com<mailto:stian at redhat.com>" <stian at redhat.com<mailto:stian at redhat.com>>
Cc: "keycloak-user at lists.jboss.org<mailto:keycloak-user at lists.jboss.org>" <keycloak-user at lists.jboss.org<mailto:keycloak-user at lists.jboss.org>>
Subject: Re: [keycloak-user] Lost session when removing an instance off cluster
Hello,
Below is exactly how Infinispan is configured:
<subsystem xmlns="urn:jboss:domain:infinispan:4.0">
<cache-container name="keycloak" jndi-name="infinispan/Keycloak">
<transport lock-timeout="60000"/>
<invalidation-cache name="realms" mode="SYNC"/>
<invalidation-cache name="users" mode="SYNC">
<eviction max-entries="10000" strategy="LRU"/>
</invalidation-cache>
<distributed-cache xmlns="" name="sessions" mode="SYNC">
<transaction mode="NON_DURABLE_XA"/>
</distributed-cache>
<local-cache xmlns="" name="offlineSessions"/>
<local-cache xmlns="" name="loginFailures"/>
<replicated-cache name="work" mode="SYNC"/>
<local-cache name="realmVersions">
<transaction mode="BATCH" locking="PESSIMISTIC"/>
</local-cache>
</cache-container>
<cache-container name="server" aliases="singleton cluster" default-cache="default" module="org.wildfly.clustering.server">
<transport lock-timeout="60000"/>
<replicated-cache name="default" mode="SYNC">
<transaction mode="BATCH"/>
</replicated-cache>
</cache-container>
<cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
<cache-container name="ejb" aliases="sfsb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
<cache-container name="hibernate" default-cache="local-query" module="org.hibernate.infinispan">
<transport lock-timeout="60000"/>
<invalidation-cache name="entity" mode="SYNC">
<transaction mode="NON_XA"/>
<eviction strategy="LRU" max-entries="10000"/>
<expiration max-idle="100000"/>
</invalidation-cache>
<local-cache name="local-query">
<eviction strategy="LRU" max-entries="10000"/>
<expiration max-idle="100000"/>
</local-cache>
<replicated-cache name="timestamps" mode="ASYNC"/>
</cache-container>
</subsystem>
This is the complete exception:
03:49:40,241 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-12) ISPN000136: Error executing command PutKeyValueCommand, writing keys [ba74e523-3299-42b3-801e-9f8cb5a0d81d]: org.infinispan.util.concurrent.TimeoutException: Replication timeout for kc2
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
03:49:40,250 ERROR [org.jboss.resteasy.resteasy_jaxrs.i18n] (default task-12) RESTEASY002025: Unknown exception while executing GET /realms/partner-dmz/protocol/openid-connect/auth: org.infinispan.util.concurrent.TimeoutException: Replication timeout for kc2
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
03:49:40,257 ERROR [io.undertow.request] (default task-12) UT005023: Exception handling request to /auth/realms/partner-dmz/protocol/openid-connect/auth: org.jboss.resteasy.spi.UnhandledException: org.infinispan.util.concurrent.TimeoutException: Replication timeout for kc2
at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:247)
at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:168)
at org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:471)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:415)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:202)
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:88)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.infinispan.util.concurrent.TimeoutException: Replication timeout for kc2
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
... 3 more
03:49:41,251 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-3) ISPN000136: Error executing command PrepareCommand, writing keys [02cff8bb-0ab9-4ab9-905f-17ffb3226050]: org.infinispan.util.concurrent.TimeoutException: Replication timeout for kc2
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
03:49:41,258 ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (default task-3) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.util.concurrent.TimeoutException: Replication timeout for kc2
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
03:49:45,504 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-3,dev,kc1) ISPN000094: Received new cluster view for channel server: [kc1|4] (1) [kc1]
03:49:45,507 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-3,dev,kc1) ISPN000094: Received new cluster view for channel keycloak: [kc1|4] (1) [kc1]
03:49:45,534 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-3,dev,kc1) ISPN000094: Received new cluster view for channel web: [kc1|4] (1) [kc1]
03:49:45,538 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-3,dev,kc1) ISPN000094: Received new cluster view for channel ejb: [kc1|4] (1) [kc1]
03:49:45,548 WARN [com.arjuna.ats.jta] (default task-3) ARJUNA016039: onePhaseCommit on < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffffac110003:-7fcaa87f:5779dc60:f01, node_name=1, branch_uid=0:ffffac110003:-7fcaa87f:5779dc60:f02, subordinatenodename=null, eis_name=unknown eis name > (TransactionXaAdapter{localTransaction=LocalXaTransaction{xid=< formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffffac110003:-7fcaa87f:5779dc60:f01, node_name=1, branch_uid=0:ffffac110003:-7fcaa87f:5779dc60:f02, subordinatenodename=null, eis_name=unknown eis name >} LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=6, stateTransferFlag=null} org.infinispan.transaction.xa.LocalXaTransaction at 3ef}) failed with exception XAException.XA_HEURRB: javax.transaction.xa.XAException
at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:213)
at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159)
at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:112)
at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:704)
at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2366)
at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1495)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1722)
at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1679)
at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1121)
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1111)
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1742)
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:248)
at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:291)
at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider$InfinispanKeycloakTransaction$CacheTask.execute(InfinispanUserSessionProvider.java:854)
at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider$InfinispanKeycloakTransaction.commit(InfinispanUserSessionProvider.java:750)
at org.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:104)
at org.keycloak.services.filters.KeycloakTransactionCommitter.filter(KeycloakTransactionCommitter.java:43)
at org.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:121)
at org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:48)
at org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:466)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:415)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:202)
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:88)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.infinispan.util.concurrent.TimeoutException: Replication timeout for kc2
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
... 3 more
03:49:45,563 WARN [org.infinispan.cache.impl.CacheImpl] (default task-3) ISPN000160: Could not complete injected transaction.: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1212)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1722)
at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1679)
at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1121)
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1111)
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1742)
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:248)
at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:291)
at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider$InfinispanKeycloakTransaction$CacheTask.execute(InfinispanUserSessionProvider.java:854)
at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider$InfinispanKeycloakTransaction.commit(InfinispanUserSessionProvider.java:750)
at org.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:104)
at org.keycloak.services.filters.KeycloakTransactionCommitter.filter(KeycloakTransactionCommitter.java:43)
at org.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:121)
at org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:48)
at org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:466)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:415)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:202)
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:88)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Suppressed: javax.transaction.xa.XAException
at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:213)
at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159)
at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:112)
at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:704)
at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2366)
at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1495)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)
... 54 more
Caused by: org.infinispan.util.concurrent.TimeoutException: Replication timeout for kc2
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
... 3 more
03:49:45,575 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-3,dev,kc1) ISPN000094: Received new cluster view for channel hibernate: [kc1|4] (1) [kc1]
03:49:45,575 ERROR [org.jboss.resteasy.resteasy_jaxrs.i18n] (default task-3) RESTEASY002025: Unknown exception while executing GET /realms/partner-dmz/protocol/openid-connect/auth: org.infinispan.commons.CacheException: Could not commit implicit transaction
at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1725)
at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1679)
at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1121)
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1111)
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1742)
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:248)
at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:291)
at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider$InfinispanKeycloakTransaction$CacheTask.execute(InfinispanUserSessionProvider.java:854)
at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider$InfinispanKeycloakTransaction.commit(InfinispanUserSessionProvider.java:750)
at org.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:104)
at org.keycloak.services.filters.KeycloakTransactionCommitter.filter(KeycloakTransactionCommitter.java:43)
at org.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:121)
at org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:48)
at org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:466)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:415)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:202)
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:88)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1212)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1722)
... 51 more
Suppressed: javax.transaction.xa.XAException
at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:213)
at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159)
at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:112)
at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:704)
at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2366)
at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1495)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)
... 54 more
Caused by: org.infinispan.util.concurrent.TimeoutException: Replication timeout for kc2
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
... 3 more
03:49:45,596 ERROR [io.undertow.request] (default task-3) UT005023: Exception handling request to /auth/realms/partner-dmz/protocol/openid-connect/auth: org.jboss.resteasy.spi.UnhandledException: org.infinispan.commons.CacheException: Could not commit implicit transaction
at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:247)
at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:168)
at org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:471)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:415)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:202)
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:88)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.infinispan.commons.CacheException: Could not commit implicit transaction
at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1725)
at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1679)
at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1121)
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1111)
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1742)
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:248)
at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:291)
at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider$InfinispanKeycloakTransaction$CacheTask.execute(InfinispanUserSessionProvider.java:854)
at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider$InfinispanKeycloakTransaction.commit(InfinispanUserSessionProvider.java:750)
at org.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:104)
at org.keycloak.services.filters.KeycloakTransactionCommitter.filter(KeycloakTransactionCommitter.java:43)
at org.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:121)
at org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:48)
at org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:466)
... 38 more
Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1212)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1722)
... 51 more
Suppressed: javax.transaction.xa.XAException
at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:213)
at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159)
at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:112)
at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:704)
at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2366)
at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1495)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)
... 54 more
Caused by: org.infinispan.util.concurrent.TimeoutException: Replication timeout for kc2
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
... 3 more
Here kc2 is the removed instance.
Kind Regards,
Sarp Kaya
From: Stian Thorgersen <sthorger at redhat.com<mailto:sthorger at redhat.com>>
Reply-To: "stian at redhat.com<mailto:stian at redhat.com>" <stian at redhat.com<mailto:stian at redhat.com>>
Date: Friday, July 1, 2016 at 7:17 PM
To: Abdullah Sarp Kaya <akaya at expedia.com<mailto:akaya at expedia.com>>
Cc: "keycloak-user at lists.jboss.org<mailto:keycloak-user at lists.jboss.org>" <keycloak-user at lists.jboss.org<mailto:keycloak-user at lists.jboss.org>>
Subject: Re: [keycloak-user] Lost session when removing an instance off cluster
Can you please include more details from the log if there is any, at least a full stack trace and not just the bit you've included. We also need to know details around how you've configured the caches and Infinispan.
On 1 July 2016 at 10:16, Sarp Kaya <akaya at expedia.com<mailto:akaya at expedia.com>> wrote:
Hello,
I have tried various ways of configuring infinispan but it just seems like if I deploy a new instance to the cluster and remove one, then some sessions are lost and an exception is thrown saying that it was not handled. This is the Infinispan exception:
Exceptionhandlingrequestto/auth/realms/realmname/protocol/openid-connect/auth:org.jboss.resteasy.spi.UnhandledException:org.infinispan.util.concurrent.TimeoutException:Replicationtimeoutfor79a0757ecab3atorg.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:247) atorg.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:168) atorg.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:471) atorg.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:415) atorg.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:202) atorg.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221) atorg.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) atorg.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) atjavax.servlet.http.HttpServlet.service(HttpServlet.java:790) atio.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) atio.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) atorg.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:88) atio.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) atio.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) atio.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) atio.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) atio.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) atorg.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) atio.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) atio.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) atio.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) atio.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) atio.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) atio.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) atio.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) atio.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) atorg.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) atio.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284) atio.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263) atio.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) atio.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174) atio.undertow.server.Connectors.executeRootHandler(Connectors.java:202) atio.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793) atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) atjava.lang.Thread.run(Thread.java:745) Causedby:org.infinispan.util.concurrent.TimeoutException:Replicationtimeoutfor79a0757ecab3atorg.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765) atorg.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$72(JGroupsTransport.java:599) atjava.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) atjava.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) atjava.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) atjava.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) atorg.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) atorg.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) atjava.util.concurrent.FutureTask.run(FutureTask.java:266) atjava.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
This causes browsers to see Internal Server Error. Shouldn’t that be handled in Keycloak as lost session, therefore KC should try to handle it rather than showing that it’s an Internal Server Error?
My current infinispan configuration looks like this:
<distributed-cache name="sessions" mode="SYNC">
<transaction mode="NON_DURABLE_XA"/>
</distributed-cache>
I use Keycloak version 1.9.5. My question is am I doing something wrong with my configuration? I tried both replicated-cache and distributed-cache and tried all transaction mode on both of them. None of them seems to solve the error that I’ve had above.
Kind Regards,
Sarp Kaya
_______________________________________________
keycloak-user mailing list
keycloak-user at lists.jboss.org<mailto:keycloak-user at lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-user
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-user/attachments/20160705/0a82bb21/attachment-0001.html
More information about the keycloak-user
mailing list