[keycloak-dev] HA mode with JDBC_PING shows warning in the logs after migration to 4.8.3 from 3.4.3

Mon May 6 03:17:56 EDT 2019

Adding +Bela Ban <bban at redhat.com>, just in case :)

Currently, JDBC_PING extends FILE_PING, which has some properties, that
works similarly to `clear_table_on_view_change`:
- remove_old_coords_on_view_change - If true, on a view change, the new
coordinator removes files from old coordinators
- remove_all_data_on_view_change - If true, on a view change, the new
coordinator removes all data except its own

It's also worth to mention, that the coordinator clears the table when
shutting down (being more specific, on `JDBC_PING#stop`. So unless your
cluster crashes a lot (by crashing I mean calling `kill -9` for example),
you should be fine.

Thanks,
Seb

On Mon, Apr 29, 2019 at 9:44 AM abhishek raghav <abhi.raghav007 at gmail.com>
wrote:

> Thanks Sebastian.
>
> I tried running the same setup with 5.0.0 of keycloak, I did not see any
> such errors which I reported in my first email. This was definitely a
> Wildfly issue and not keycloak.
>
> Regarding my 2nd question - i.e. support of "clear_table_on_view_change"
> property. I see that jgroups has removed support of this property. So lets
> say if JGROUPSPING table has lot stale entries, while keycloak starts
> booting up - each time keycloak node will try to JOIN with all the entries
> already present in the JGROUPSPING table and thus time taken for the
> service to start will be more. If that timeline is more than 300s, keycloak
> does not start and reports timeout error.
> This scenario is highly possible in cloud scenarios, since there the
> keycloak nodes can start on any available host/IP since no of nodes are not
> fixed.
>
> Can you suggest any workaround to fix this.
>
> *- Best Regards*
>    Abhishek Raghav
>
>
>
>
>
>
>
>
> On Fri, Apr 26, 2019 at 6:11 PM Sebastian Laskawiec <slaskawi at redhat.com>
> wrote:
>
>> There was a bunch of fixed to JGroups a while ago, including changes in
>> JDBC_PING.
>>
>> Could you please rerun your setup with Keycloak >= 5.0.0? I believe some
>> of the issues (or maybe even all of them) should be fixed.
>>
>> On Thu, Apr 25, 2019 at 7:19 PM abhishek raghav <abhi.raghav007 at gmail.com>
>> wrote:
>>
>>> Hi
>>>
>>> After the migration of keycloak HA configurations from 3.4.3.Final to
>>> 4.8.3.Final, I am seeing some WARNINGS on one of the nodes of keycloak
>>> immediately after the keycloak is started with 2 nodes. This occurs after
>>> every time when the cluster is scaled up or whenever infinispan is trying
>>> to update the cluster member list.
>>> I am using JDBC_PING to achieve clustering in keycloak.
>>>
>>> Below is the stacktrace -
>>>
>>> 2019-04-24 12:20:43,687 WARN
>>> >> [org.infinispan.topology.ClusterTopologyManagerImpl]
>>> >> (transport-thread--p18-t2) [dcidqdcosagent08] KEYCLOAK DEV 1.5.RC
>>> >> ISPN000197: Error updating cluster member list:
>>> >> org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out
>>> >> waiting for responses for request 1 from dcidqdcosagent02
>>> >
>>> >                   at
>>> >>
>>> org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
>>> >
>>> >                   at
>>> >>
>>> org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
>>> >
>>> >                   at
>>> >>
>>> org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
>>> >
>>> >                   at
>>> >> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> >
>>> >                   at
>>> >>
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>> >
>>> >                   at
>>> >>
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> >
>>> >                   at
>>> >>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>> >
>>> >                   at
>>> >>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> >
>>> >                   at java.lang.Thread.run(Thread.java:748)
>>> >
>>> >                   Suppressed:
>>> org.infinispan.util.logging.TraceException
>>> >
>>> >                                     at
>>> >>
>>> org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:75)
>>> >
>>> >                                     at
>>> >>
>>> org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:525)
>>> >
>>> >                                     at
>>> >>
>>> org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:508)
>>> >
>>> >
>>>
>>> Now after I searched, I really did not see anyone reported such error on
>>> keycloak but there is similar bug reported in WILDLFY 14 and is
>>> categorized
>>> as a blocker in WILDLFY 14.This bug is already fixed in WILDLFY 15.
>>> https://issues.jboss.org/browse/WFLY-10736?attachmentViewMode=list
>>>
>>> Now since keycloak 4.8 is also based on WILDLFY 14, these WARNINGS could
>>> be
>>> because of this blocker in WILDFLY 14.
>>>
>>> What should I do to get rid this error. Is this really a problem in
>>> keycloak 4.8.3.Final. Did anyone notice any such issue while running
>>> keycloak 4.8.3 in HA mode.
>>> Is there a workaround to fix this.
>>>
>>>
>>> One more thing we noticed is - It is regarding a property in JDBC_PING
>>> protocol we are using in our 3.4.3 setup i.e.
>>> "clear_table_on_view_change"
>>> but it is no more supported in 4.8 version. and thus the JGROUPSPING
>>> table
>>> is filled up with lot of stale entries. Is there a workaround to clear
>>> the
>>> table after view change in 4.8 also.
>>>
>>> Thanks
>>> Abhishek
>>> _______________________________________________
>>> keycloak-dev mailing list
>>> keycloak-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>
>>