Re: [keycloak-dev] HA mode with JDBC_PING shows warning in the logs after migration to 4.8.3 from 3.4.3

Adding +Bela Ban <bban(a)redhat.com&gt;, just in case :) Currently, JDBC_PING extends FILE_PING, which has some properties, that works similarly to `clear_table_on_view_change`: - remove_old_coords_on_view_change - If true, on a view change, the new coordinator removes files from old coordinators - remove_all_data_on_view_change - If true, on a view change, the new coordinator removes all data except its own It's also worth to mention, that the coordinator clears the table when shutting down (being more specific, on `JDBC_PING#stop`. So unless your cluster crashes a lot (by crashing I mean calling `kill -9` for example), you should be fine. Thanks, Seb On Mon, Apr 29, 2019 at 9:44 AM abhishek raghav <abhi.raghav007(a)gmail.com&gt; wrote: > Thanks Sebastian. > > I tried running the same setup with 5.0.0 of keycloak, I did not see any > such errors which I reported in my first email. This was definitely a > Wildfly issue and not keycloak. > > Regarding my 2nd question - i.e. support of "clear_table_on_view_change" > property. I see that jgroups has removed support of this property. So lets > say if JGROUPSPING table has lot stale entries, while keycloak starts > booting up - each time keycloak node will try to JOIN with all the entries > already present in the JGROUPSPING table and thus time taken for the > service to start will be more. If that timeline is more than 300s, keycloak > does not start and reports timeout error. > This scenario is highly possible in cloud scenarios, since there the > keycloak nodes can start on any available host/IP since no of nodes are not > fixed. > > Can you suggest any workaround to fix this. > > *- Best Regards* > Abhishek Raghav > > > > > > > > > On Fri, Apr 26, 2019 at 6:11 PM Sebastian Laskawiec <slaskawi(a)redhat.com&gt; > wrote: > >> There was a bunch of fixed to JGroups a while ago, including changes in >> JDBC_PING. >> >> Could you please rerun your setup with Keycloak >= 5.0.0? I believe some >> of the issues (or maybe even all of them) should be fixed. >> >> On Thu, Apr 25, 2019 at 7:19 PM abhishek raghav < >> abhi.raghav007(a)gmail.com&gt; wrote: >> >>> Hi >>> >>> After the migration of keycloak HA configurations from 3.4.3.Final to >>> 4.8.3.Final, I am seeing some WARNINGS on one of the nodes of keycloak >>> immediately after the keycloak is started with 2 nodes. This occurs >>> after >>> every time when the cluster is scaled up or whenever infinispan is >>> trying >>> to update the cluster member list. >>> I am using JDBC_PING to achieve clustering in keycloak. >>> >>> Below is the stacktrace - >>> >>> 2019-04-24 12:20:43,687 WARN >>> >> [org.infinispan.topology.ClusterTopologyManagerImpl] >>> >> (transport-thread--p18-t2) [dcidqdcosagent08] KEYCLOAK DEV 1.5.RC >>> >> ISPN000197: Error updating cluster member list: >>> >> org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed >>> out >>> >> waiting for responses for request 1 from dcidqdcosagent02 >>> > >>> > at >>> >> >>> org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167) >>> > >>> > at >>> >> >>> org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87) >>> > >>> > at >>> >> >>> org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22) >>> > >>> > at >>> >> java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> > >>> > at >>> >> >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) >>> > >>> > at >>> >> >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >>> > >>> > at >>> >> >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>> > >>> > at >>> >> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>> > >>> > at java.lang.Thread.run(Thread.java:748) >>> > >>> > Suppressed: >>> org.infinispan.util.logging.TraceException >>> > >>> > at >>> >> >>> org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:75) >>> > >>> > at >>> >> >>> org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:525) >>> > >>> > at >>> >> >>> org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:508) >>> > >>> > >>> >>> Now after I searched, I really did not see anyone reported such error on >>> keycloak but there is similar bug reported in WILDLFY 14 and is >>> categorized >>> as a blocker in WILDLFY 14.This bug is already fixed in WILDLFY 15. >>> https://issues.jboss.org/browse/WFLY-10736?attachmentViewMode=list >>> >>> Now since keycloak 4.8 is also based on WILDLFY 14, these WARNINGS >>> could be >>> because of this blocker in WILDFLY 14. >>> >>> What should I do to get rid this error. Is this really a problem in >>> keycloak 4.8.3.Final. Did anyone notice any such issue while running >>> keycloak 4.8.3 in HA mode. >>> Is there a workaround to fix this. >>> >>> >>> One more thing we noticed is - It is regarding a property in JDBC_PING >>> protocol we are using in our 3.4.3 setup i.e. >>> "clear_table_on_view_change" >>> but it is no more supported in 4.8 version. and thus the JGROUPSPING >>> table >>> is filled up with lot of stale entries. Is there a workaround to clear >>> the >>> table after view change in 4.8 also. >>> >>> Thanks >>> Abhishek >>> _______________________________________________ >>> keycloak-dev mailing list >>> keycloak-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>> >>