[keycloak-user] how to clean-up OFFLINE_CLIENT_SESSION, OFFLINE_USER_SESSION tables

[i.pop at centurylink.net]

I have created a JIRA: KEYCLOAK-4066
Thanks,
Ioan

----- Original Message -----
From: "Marek Posolda" <mposolda at redhat.com>
To: "i pop" <i.pop at centurylink.net>
Cc: "keycloak-user" <keycloak-user at lists.jboss.org>
Sent: Monday, December 12, 2016 2:15:57 AM
Subject: Re: [keycloak-user] how to clean-up OFFLINE_CLIENT_SESSION, OFFLINE_USER_SESSION tables

Could you create JIRA and set fix version to 3.0 ? We plan to do some 
refactoring for userSessions in next release and better handling of this 
usecase can be added as part of it too IMO.

I think that each periodic cleaner (ClearExpiredUserSessions task) will 
just lookup the sessions, which are saved on the particular server and 
will cleanup just those. Currently cleaner tries to lookup for all 
sessions within whole cluster, which probably has some communication 
overhead. Fact is, that we didn't yet try to test with 1.25 M sessions.

The workaround for you can be to add your own implementation of 
UserSessionProvider.removeExpired and handle it somehow by yourself 
until we add our solution.

The easier workaround may be to shorten offline tokens timeout in admin 
console from 30 days to something shorter (eg. 1 day?) as then the count 
of offline sessions probably won't grow so much. This is an option just 
if you have an opportunity to cleanup your DB session tables first and 
restart cluster. When all 1.25M sessions are already in infinispan 
cluster, the exceptions will be always thrown.

Marek

On 09/12/16 20:55, i.pop at centurylink.net wrote:
> Thank you Marek for your response.
> Unfortunately, this bug still exists in the KC-2.4.0.Final release, as well(I have 1.25 M offline-user&client sessions in my 2-tables):
> [Server:server-one] 13:48:49,541 ERROR [org.keycloak.services] (Timer-2) KC-SERVICES0089: Failed to run scheduled task ClearExpiredUserSessions: org.infinispan.util.concurrent.TimeoutException: Replication timeout for r720xd-14:server-two
> [Server:server-one]     at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
> [Server:server-one]     at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$72(JGroupsTransport.java:599)
> [Server:server-one]     at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> [Server:server-one]     at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> [Server:server-one]     at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> [Server:server-one]     at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> [Server:server-one]     at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
> [Server:server-one]     at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
> [Server:server-one]     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [Server:server-one]     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [Server:server-one]     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [Server:server-one]     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [Server:server-one]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [Server:server-one]     at java.lang.Thread.run(Thread.java:745)
>
> Any piece of advice, please?
> Thanks,
> Ioan
> ----- Original Message -----
> From: "Marek Posolda" <mposolda at redhat.com>
> To: "i pop" <i.pop at centurylink.net>
> Cc: "keycloak-user" <keycloak-user at lists.jboss.org>
> Sent: Thursday, December 8, 2016 5:03:39 AM
> Subject: Re: [keycloak-user] how to clean-up OFFLINE_CLIENT_SESSION, OFFLINE_USER_SESSION tables
>
> Yes, there were some related fixes though. Can you try to upgrade to
> latest 2.4.0.Final and see if it helps?
>
> Thanks,
> Marek
>
> On 07/12/16 17:03, i.pop at centurylink.net wrote:
>> Thank you for your message. However, if I set parameter "Offline Session Idle" to 30 min, I am getting a replication timeout exception associated with the periodic cleaner scheduler service
>> [Server:server-one] 09:24:54,979 ERROR [org.keycloak.services] (Timer-2) KC-SERVICES0089: Failed to run scheduled task ClearExpiredUserSessions: org.infinispan.util.concurrent.TimeoutException: Replication timeout for slaveServer:server-two
>> [Server:server-one]     at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
>> [Server:server-one]     at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$72(JGroupsTransport.java:599)
>> [Server:server-one]     at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
>> [Server:server-one]     at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
>> [Server:server-one]     at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>> [Server:server-one]     at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>> [Server:server-one]     at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
>> [Server:server-one]     at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
>> [Server:server-one]     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> [Server:server-one]     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> [Server:server-one]     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> [Server:server-one]     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> [Server:server-one]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [Server:server-one]     at java.lang.Thread.run(Thread.java:745)
>>
>> This exception is thrown with a periodicity of 15 min.
>> I have found a previous reference to such exception in your "keycloak-user" customer inquiries
>> http://lists.jboss.org/pipermail/keycloak-user/2016-July/006892.html
>>
>> It looks like there was a bug in your KC software. Have you fixed this bug in your later KC releases since July 2016? Or, it may be a miss-configuration in my domain clustered configuration( I use your KC-2.1.0.Final  release) ?
>> Thanks,
>> Ioan
>>    
>> ----- Original Message -----
>> From: "Marek Posolda" <mposolda at redhat.com>
>> To: "i pop" <i.pop at centurylink.net>, "keycloak-user" <keycloak-user at lists.jboss.org>
>> Sent: Friday, November 25, 2016 3:20:18 AM
>> Subject: Re: [keycloak-user] how to clean-up OFFLINE_CLIENT_SESSION, OFFLINE_USER_SESSION tables
>>
>> It seems you are using offline tokens for some of your application
>> right? There is periodic cleaner, which will remove the records from the
>> expired offline sessions. But timeout for the offline sessions is 30
>> days by default. Also the time of "last refresh" is currently updated in
>> DB every time when you restart the server (in case that you have
>> single-server without cluster).
>>
>> In other words, if you restart the server at least once every 30 days,
>> the table will keep growing. It is probably something we can improve...
>> Feel free to create JIRA.
>>
>> Until that, your possibilities are:
>> - Decrease the timeout to shorter value than 30 days (can be done in
>> admin console)
>> - Ensure the server is not restarted within 30 days, so the outdated
>> sessions can be cleared.
>> - Use cluster with 2 nodes or more and ensure that at least 1 node is
>> always online.
>>
>> Marek
>>
>> On 24/11/16 20:11, i.pop at centurylink.net wrote:
>>> Hi,
>>> Working with a domain clustered mode and shared ORACLE db , I am noticing {OFFLINE_CLIENT_SESSION, OFFLINE_USER_SESSION} tables keep growing in size. How these tables get cleaned up?
>>>
>>>
>>> Thanks,
>>> Ioan
>>> _______________________________________________
>>> keycloak-user mailing list
>>> keycloak-user at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/keycloak-user