[keycloak-user] cache replication problems?

Hynek Mlnarik hmlnarik at redhat.com
Thu Feb 8 03:57:31 EST 2018 

Check "Connecting to Infinispan JConsole" part of [1] for help with
connecting to JMX via jconsole in AWS environment.

[1]
http://blog.keycloak.org/2018/01/keycloak-cross-data-center-setup-in-aws.html

On Wed, Feb 7, 2018 at 4:43 PM, Angel Abella <aabella at bkool.com> wrote:

> It would be great to be able to access JMX, but I am not able to connect to
> the JMX subsystem from my local machine to the servers on AWS.
> I have this
>
>  <subsystem xmlns="urn:jboss:domain:jmx:1.3">
>   <expose-resolved-model/>
>   <expose-expression-model/>
>   <remoting-connector use-management-endpoint="false"/>
> </subsystem>
>
> In the standalone-ha.xml file, and ports 9090 an 8081 open, but I cannot
> connect from my machine to the remote server using jconsole with:
>
> service:jmx:http-remoting-jmx://servername:9990  OR
> service:jmx:http-remoting-jmx://servername:8081
>
> :-(
>
>
> 2018-02-07 16:07 GMT+01:00 Josh Cain <jcain at redhat.com>:
>
> > I know for us, we keep a close eye on JVM statistics.  However, if your
> > infinispan cluster is on the same box as your Keycloak server (not
> > running anything like JDG) as ours is atm, then your JVM statistics are
> > only part of the story.
> >
> > Iif you can *safely* enable JMX and set up a service account
> > appropriately, MBean statistics on cache usage would give you a really
> > clear picture of what Infinispan is doing:
> >
> > <cache-container name="keycloak" jndi-name="infinispan/Keycloak"
> > statistics-enabled="true">
> >   ...
> >   <distributed-cache name="sessions" mode="SYNC" owners="1"
> > statistics-enabled="true"/>
> >   <distributed-cache name="offlineSessions" mode="SYNC" owners="1"
> > statistics-enabled="true"/>
> >   ...
> >
> > From there, you can get a better idea of how the cache-container
> > settings can be tuned in your server config.
> >
> > Josh Cain
> > Senior Software Applications Engineer, RHCE
> > Red Hat North America
> > jcain at redhat.com IRC: jcain
> >
> > On 02/07/2018 04:15 AM, Angel Abella wrote:
> > > This test are with the latest KC version.
> > > The problems starts when the number of sessions raise above 800.000. We
> > > have tried to lower the access token lifespan, sso session max and sso
> > > session idle and it helped.
> > >
> > > Sadly the docs are not as usefull as they should. :-(
> > >
> > > 2018-02-07 11:03 GMT+01:00 Marek Posolda <mposolda at redhat.com>:
> > >
> > >> What is your Keycloak version? If you used 2.X and you will migrate to
> > >> latest 3.4.3, there is some chance that issues might be solved as we
> did
> > >> some performance improvements.
> > >>
> > >> Overally, it depends on number of sessions, network connection between
> > >> cluster servers etc. If network can't be tweaked, then maybe it's
> > possible
> > >> to increase replication timeout? See infinispan and Wildfly Infinispan
> > >> Subsystem docs for how to do it.
> > >>
> > >> Marek
> > >>
> > >>
> > >> On 06/02/18 12:09, Angel Abella wrote:
> > >>
> > >>> Hello,
> > >>>
> > >>> We have a 2 server standalone-ha installation. When the number of
> > sessions
> > >>> alive increases we get this errors:
> > >>>
> > >>>
> > >>> 2018-02-06 11:42:07,161 ERROR
> > >>> [org.infinispan.interceptors.InvocationContextInterceptor] (default
> > >>> task-22) ISPN000136: Error executing command PutKeyValueCommand,
> > writing
> > >>> keys [f75b436f-d316-4442-8d9b-c7313647c5b8]:
> > >>> org.infinispan.util.concurrent.TimeoutException: Replication timeout
> > for
> > >>> sson2
> > >>>          at
> > >>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.
> > >>> checkRsp(JGroupsTransport.java:827)
> > >>>          at
> > >>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.
> > >>> lambda$invokeRemotelyAsync$0(JGroupsTransport.java:628)
> > >>>          at
> > >>> java.util.concurrent.CompletableFuture.uniApply(CompletableF
> > >>> uture.java:602)
> > >>>          at
> > >>> java.util.concurrent.CompletableFuture$UniApply.tryFire(
> > >>> CompletableFuture.java:577)
> > >>>          at
> > >>> java.util.concurrent.CompletableFuture.postComplete(Completa
> > >>> bleFuture.java:474)
> > >>>          at
> > >>> java.util.concurrent.CompletableFuture.complete(CompletableF
> > >>> uture.java:1962)
> > >>>          at
> > >>> org.infinispan.remoting.transport.jgroups.SingleResponseFutu
> > >>> re.call(SingleResponseFuture.java:46)
> > >>>          at
> > >>> org.infinispan.remoting.transport.jgroups.SingleResponseFutu
> > >>> re.call(SingleResponseFuture.java:17)
> > >>>          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > >>>          at
> > >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
> > >>> tureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > >>>          at
> > >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
> > >>> tureTask.run(ScheduledThreadPoolExecutor.java:293)
> > >>>          at
> > >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> > >>> Executor.java:1142)
> > >>>          at
> > >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> > >>> lExecutor.java:617)
> > >>>          at java.lang.Thread.run(Thread.java:745)
> > >>>
> > >>> 2018-02-06 11:42:07,162 ERROR
> > >>> [org.keycloak.services.error.KeycloakErrorHandler] (default task-22)
> > >>> Uncaught server error: org.infinispan.util.
> > concurrent.TimeoutException:
> > >>> Replication timeout for sson2
> > >>>          at
> > >>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.
> > >>> checkRsp(JGroupsTransport.java:827)
> > >>>          at
> > >>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.
> > >>> lambda$invokeRemotelyAsync$0(JGroupsTransport.java:628)
> > >>>          at
> > >>> java.util.concurrent.CompletableFuture.uniApply(CompletableF
> > >>> uture.java:602)
> > >>>          at
> > >>> java.util.concurrent.CompletableFuture$UniApply.tryFire(
> > >>> CompletableFuture.java:577)
> > >>>          at
> > >>> java.util.concurrent.CompletableFuture.postComplete(Completa
> > >>> bleFuture.java:474)
> > >>>          at
> > >>> java.util.concurrent.CompletableFuture.complete(CompletableF
> > >>> uture.java:1962)
> > >>>          at
> > >>> org.infinispan.remoting.transport.jgroups.SingleResponseFutu
> > >>> re.call(SingleResponseFuture.java:46)
> > >>>          at
> > >>> org.infinispan.remoting.transport.jgroups.SingleResponseFutu
> > >>> re.call(SingleResponseFuture.java:17)
> > >>>          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > >>>          at
> > >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
> > >>> tureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > >>>          at
> > >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
> > >>> tureTask.run(ScheduledThreadPoolExecutor.java:293)
> > >>>          at
> > >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> > >>> Executor.java:1142)
> > >>>          at
> > >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> > >>> lExecutor.java:617)
> > >>>          at java.lang.Thread.run(Thread.java:745)
> > >>>
> > >>> 2018-02-06 11:42:07,166 ERROR
> > >>> [org.infinispan.interceptors.InvocationContextInterceptor] (default
> > >>> task-17) ISPN000136: Error executing command RemoveCommand, writing
> > keys
> > >>> [0d8d4c5c-7971-46dd-b414-cb5f16862085]:
> > >>> org.infinispan.util.concurrent.TimeoutException: Replication timeout
> > for
> > >>> sson2
> > >>>          at
> > >>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.
> > >>> checkRsp(JGroupsTransport.java:827)
> > >>>          at
> > >>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.
> > >>> lambda$invokeRemotelyAsync$0(JGroupsTransport.java:628)
> > >>>          at
> > >>> java.util.concurrent.CompletableFuture.uniApply(CompletableF
> > >>> uture.java:602)
> > >>>          at
> > >>> java.util.concurrent.CompletableFuture$UniApply.tryFire(
> > >>> CompletableFuture.java:577)
> > >>>          at
> > >>> java.util.concurrent.CompletableFuture.postComplete(Completa
> > >>> bleFuture.java:474)
> > >>>          at
> > >>> java.util.concurrent.CompletableFuture.complete(CompletableF
> > >>> uture.java:1962)
> > >>>          at
> > >>> org.infinispan.remoting.transport.jgroups.SingleResponseFutu
> > >>> re.call(SingleResponseFuture.java:46)
> > >>>          at
> > >>> org.infinispan.remoting.transport.jgroups.SingleResponseFutu
> > >>> re.call(SingleResponseFuture.java:17)
> > >>>          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > >>>          at
> > >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
> > >>> tureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > >>>          at
> > >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
> > >>> tureTask.run(ScheduledThreadPoolExecutor.java:293)
> > >>>          at
> > >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> > >>> Executor.java:1142)
> > >>>          at
> > >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> > >>> lExecutor.java:617)
> > >>>          at java.lang.Thread.run(Thread.java:745)
> > >>>
> > >>> 2018-02-06 11:42:07,171 ERROR
> > >>> [org.infinispan.interceptors.InvocationContextInterceptor] (default
> > >>> task-11) ISPN000136: Error executing command PutKeyValueCommand,
> > writing
> > >>> keys [dfd69644-e241-465c-8a92-ef84e76caf62]:
> > >>> org.infinispan.util.concurrent.TimeoutException: Replication timeout
> > for
> > >>> sson2
> > >>>          at
> > >>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.
> > >>> checkRsp(JGroupsTransport.java:827)
> > >>>          at
> > >>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.
> > >>> lambda$invokeRemotelyAsync$0(JGroupsTransport.java:628)
> > >>>          at
> > >>> java.util.concurrent.CompletableFuture.uniApply(CompletableF
> > >>> uture.java:602)
> > >>>          at
> > >>> java.util.concurrent.CompletableFuture$UniApply.tryFire(
> > >>> CompletableFuture.java:577)
> > >>>          at
> > >>> java.util.concurrent.CompletableFuture.postComplete(Completa
> > >>> bleFuture.java:474)
> > >>>          at
> > >>> java.util.concurrent.CompletableFuture.complete(CompletableF
> > >>> uture.java:1962)
> > >>>          at
> > >>> org.infinispan.remoting.transport.jgroups.SingleResponseFutu
> > >>> re.call(SingleResponseFuture.java:46)
> > >>>          at
> > >>> org.infinispan.remoting.transport.jgroups.SingleResponseFutu
> > >>> re.call(SingleResponseFuture.java:17)
> > >>>          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > >>>          at
> > >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
> > >>> tureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > >>>          at
> > >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
> > >>> tureTask.run(ScheduledThreadPoolExecutor.java:293)
> > >>>          at
> > >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> > >>> Executor.java:1142)
> > >>>          at
> > >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> > >>> lExecutor.java:617)
> > >>>          at java.lang.Thread.run(Thread.java:745)
> > >>>
> > >>> 2018-02-06 11:42:07,173 ERROR
> > >>> [org.keycloak.services.error.KeycloakErrorHandler] (default task-11)
> > >>> Uncaught server error: org.infinispan.util.
> > concurrent.TimeoutException:
> > >>> Replication timeout for sson2
> > >>>          at
> > >>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.
> > >>> checkRsp(JGroupsTransport.java:827)
> > >>>          at
> > >>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.
> > >>> lambda$invokeRemotelyAsync$0(JGroupsTransport.java:628)
> > >>>          at
> > >>> java.util.concurrent.CompletableFuture.uniApply(CompletableF
> > >>> uture.java:602)
> > >>>          at
> > >>> java.util.concurrent.CompletableFuture$UniApply.tryFire(
> > >>> CompletableFuture.java:577)
> > >>>          at
> > >>> java.util.concurrent.CompletableFuture.postComplete(Completa
> > >>> bleFuture.java:474)
> > >>>          at
> > >>> java.util.concurrent.CompletableFuture.complete(CompletableF
> > >>> uture.java:1962)
> > >>>          at
> > >>> org.infinispan.remoting.transport.jgroups.SingleResponseFutu
> > >>> re.call(SingleResponseFuture.java:46)
> > >>>          at
> > >>> org.infinispan.remoting.transport.jgroups.SingleResponseFutu
> > >>> re.call(SingleResponseFuture.java:17)
> > >>>          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > >>>          at
> > >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
> > >>> tureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > >>>          at
> > >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
> > >>> tureTask.run(ScheduledThreadPoolExecutor.java:293)
> > >>>          at
> > >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> > >>> Executor.java:1142)
> > >>>          at
> > >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> > >>> lExecutor.java:617)
> > >>>          at java.lang.Thread.run(Thread.java:745)
> > >>>
> > >>> 2018-02-06 11:42:07,205 ERROR
> > >>> [org.keycloak.services.error.KeycloakErrorHandler] (default task-17)
> > >>> Uncaught server error: org.infinispan.util.
> > concurrent.TimeoutException:
> > >>> Replication timeout for sson2
> > >>>          at
> > >>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.
> > >>> checkRsp(JGroupsTransport.java:827)
> > >>>          at
> > >>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.
> > >>> lambda$invokeRemotelyAsync$0(JGroupsTransport.java:628)
> > >>>          at
> > >>> java.util.concurrent.CompletableFuture.uniApply(CompletableF
> > >>> uture.java:602)
> > >>>          at
> > >>> java.util.concurrent.CompletableFuture$UniApply.tryFire(
> > >>> CompletableFuture.java:577)
> > >>>          at
> > >>> java.util.concurrent.CompletableFuture.postComplete(Completa
> > >>> bleFuture.java:474)
> > >>>          at
> > >>> java.util.concurrent.CompletableFuture.complete(CompletableF
> > >>> uture.java:1962)
> > >>>          at
> > >>> org.infinispan.remoting.transport.jgroups.SingleResponseFutu
> > >>> re.call(SingleResponseFuture.java:46)
> > >>>          at
> > >>> org.infinispan.remoting.transport.jgroups.SingleResponseFutu
> > >>> re.call(SingleResponseFuture.java:17)
> > >>>          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > >>>          at
> > >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
> > >>> tureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > >>>          at
> > >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
> > >>> tureTask.run(ScheduledThreadPoolExecutor.java:293)
> > >>>          at
> > >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> > >>> Executor.java:1142)
> > >>>          at
> > >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> > >>> lExecutor.java:617)
> > >>>          at java.lang.Thread.run(Thread.java:745)
> > >>>
> > >>>
> > >>>
> > >>> Any idea of what's going on?
> > >>>
> > >>>
> > >>
> > >
> > >
> >
> >
> > _______________________________________________
> > keycloak-user mailing list
> > keycloak-user at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/keycloak-user
> >
>
>
>
> --
> Angel Abella
> *IT *
> *BKOOL* *Connect* *| Sport*
> mail: aabella at bkool.com
> mob: +34 691 77 18 98
> add: C/ San Joaquín 3 - 28231 Las Rozas - Madrid
> www.bkool.com
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
>



-- 

--Hynek

More information about the keycloak-user mailing list