[keycloak-user] [keycloak-dev] HA mode with JDBC_PING shows warning in the logs after migration to 4.8.3 from 3.4.3

Wed May 15 04:35:48 EDT 2019

If you want to build an automated script to detect such situations and try
to fix them, you would need to:
- Identify that cluster experiences some problems by scanning the logs.
- Identify what node is a JGroups coordinator. You may do this by examining
JMX and looking for: jgroups/protocol/ee/GMS and the attribute name is coord
- Kill the node that is a coordinator. Then the cluster should elect the
new coordinator and should reconcile.

However, the proper recommendation is to do the upgrade and get the proper
fix from JGroups.

On Tue, May 14, 2019 at 4:03 PM <pkboucher801 at gmail.com> wrote:

> In the meantime (before we switch to 5.0+), is there any way to automate
> recognition when the cluster hangs (or is about to hang) as in
> https://issues.jboss.org/browse/WFLY-10736?attachmentViewMode=list
> <https://secure-web.cisco.com/1rkzdmR023GzmidXPHjPkNYaDpKjjdqm_1Dg-0EGDGbiQ3ihyMAwkODF6Qxb9zfR4nBPNb8bwSeliv847b4X0dyRDyogJhKiu-6hbcikyDnSzL-nBD2uWZvGyBvvYbsI4J3J2rdcVfV_KOPyxMCWm6eA6e8GvSZ74354QN5UnFi5Pr6n9aYqm9mPy37mFwSSrPUF2PwkhUGdrzqa-vSxgI112ykT9T9QSMmv8tXRSkb1fuNeRh5YzinMGgCDfZ9niMu5VdBKlxgco3b6g1oCXT_Fs1EDEfpIHijU9WAf9wsbPclp4KEXiTh2IFDKG8JLnCDwuJhbq6vZNlyY6r-FoOSIo6GgdSjkYYZpzlyZS7yy96liblbsUB-QhrUMyOWCm9Cxnn8O3-L6KT5SMsm13GhAV260vy-r_c59-UJQj7OMxdQ9jBBoP8o_IKSsd-YzmEhNgBZdEunVXPNaKgh-n8UsIFfo5JqwOPWhjQB0mcTh2JOxbJs5jNdrqEEkmZDR0gbqaUabNMvc0WV9jadEHBA/https%3A%2F%2Fissues.jboss.org%2Fbrowse%2FWFLY-10736%3FattachmentViewMode%3Dlist>
> ?
>
>
>
> And is it reliable to un-hang the cluster by scaling it down to zero
> instances, and then scaling back up?
>
>
>
> Thanks!
>
>
>
> Regards,
>
> Peter
>
>
>
> From: *Sebastian Laskawiec* <slaskawi at redhat.com>
> Date: Fri, Apr 26, 2019 at 6:11 PM
> Subject: Re: [keycloak-dev] HA mode with JDBC_PING shows warning in the
> logs after migration to 4.8.3 from 3.4.3
> To: abhishek raghav <abhi.raghav007 at gmail.com>
> Cc: keycloak-user <keycloak-user at lists.jboss.org>, keycloak-dev <
> keycloak-dev at lists.jboss.org>
>
>
>
> There was a bunch of fixed to JGroups a while ago, including changes in
> JDBC_PING.
>
>
>
> Could you please rerun your setup with Keycloak >= 5.0.0? I believe some
> of the issues (or maybe even all of them) should be fixed.
>
>
>
> On Thu, Apr 25, 2019 at 7:19 PM abhishek raghav <abhi.raghav007 at gmail.com>
> wrote:
>
> Hi
>
> After the migration of keycloak HA configurations from 3.4.3.Final to
> 4.8.3.Final, I am seeing some WARNINGS on one of the nodes of keycloak
> immediately after the keycloak is started with 2 nodes. This occurs after
> every time when the cluster is scaled up or whenever infinispan is trying
> to update the cluster member list.
> I am using JDBC_PING to achieve clustering in keycloak.
>
> Below is the stacktrace -
>
> 2019-04-24 12:20:43,687 WARN
> >> [org.infinispan.topology.ClusterTopologyManagerImpl]
> >> (transport-thread--p18-t2) [dcidqdcosagent08] KEYCLOAK DEV 1.5.RC
> >> ISPN000197: Error updating cluster member list:
> >> org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out
> >> waiting for responses for request 1 from dcidqdcosagent02
> >
> >                   at
> >>
> org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> >
> >                   at
> >>
> org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> >
> >                   at
> >>
> org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> >
> >                   at
> >> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >
> >                   at
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> >
> >                   at
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> >
> >                   at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >
> >                   at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >
> >                   at java.lang.Thread.run(Thread.java:748)
> >
> >                   Suppressed: org.infinispan.util.logging.TraceException
> >
> >                                     at
> >>
> org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:75)
> >
> >                                     at
> >>
> org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:525)
> >
> >                                     at
> >>
> org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:508)
> >
> >
>
> Now after I searched, I really did not see anyone reported such error on
> keycloak but there is similar bug reported in WILDLFY 14 and is categorized
> as a blocker in WILDLFY 14.This bug is already fixed in WILDLFY 15.
> https://issues.jboss.org/browse/WFLY-10736?attachmentViewMode=list
> <https://secure-web.cisco.com/1rkzdmR023GzmidXPHjPkNYaDpKjjdqm_1Dg-0EGDGbiQ3ihyMAwkODF6Qxb9zfR4nBPNb8bwSeliv847b4X0dyRDyogJhKiu-6hbcikyDnSzL-nBD2uWZvGyBvvYbsI4J3J2rdcVfV_KOPyxMCWm6eA6e8GvSZ74354QN5UnFi5Pr6n9aYqm9mPy37mFwSSrPUF2PwkhUGdrzqa-vSxgI112ykT9T9QSMmv8tXRSkb1fuNeRh5YzinMGgCDfZ9niMu5VdBKlxgco3b6g1oCXT_Fs1EDEfpIHijU9WAf9wsbPclp4KEXiTh2IFDKG8JLnCDwuJhbq6vZNlyY6r-FoOSIo6GgdSjkYYZpzlyZS7yy96liblbsUB-QhrUMyOWCm9Cxnn8O3-L6KT5SMsm13GhAV260vy-r_c59-UJQj7OMxdQ9jBBoP8o_IKSsd-YzmEhNgBZdEunVXPNaKgh-n8UsIFfo5JqwOPWhjQB0mcTh2JOxbJs5jNdrqEEkmZDR0gbqaUabNMvc0WV9jadEHBA/https%3A%2F%2Fissues.jboss.org%2Fbrowse%2FWFLY-10736%3FattachmentViewMode%3Dlist>
>
> Now since keycloak 4.8 is also based on WILDLFY 14, these WARNINGS could be
> because of this blocker in WILDFLY 14.
>
> What should I do to get rid this error. Is this really a problem in
> keycloak 4.8.3.Final. Did anyone notice any such issue while running
> keycloak 4.8.3 in HA mode.
> Is there a workaround to fix this.
>
>
> One more thing we noticed is - It is regarding a property in JDBC_PING
> protocol we are using in our 3.4.3 setup i.e. "clear_table_on_view_change"
> but it is no more supported in 4.8 version. and thus the JGROUPSPING table
> is filled up with lot of stale entries. Is there a workaround to clear the
> table after view change in 4.8 also.
>
> Thanks
> Abhishek
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
> <https://secure-web.cisco.com/17Y97-5o4x23f7qOJVpqrPG_ucegGkNCivunRvLWIYKMfct4L7jrNrcSJ-GOyXmluwLLVbrWhq8BVwk389drzM-cJXO3sT4tE4FBfMUllGuCgseOpvv4Ww4O0CjPMb2_L5oioy2XqmW6PC6KADWbqAB6IMesziLIk8E10zluypdXNUo3EgbqVWhdsyR6LjGBRJ_zvUGBF7_6-rocGWvYy6ZxXBsJsh5gfph_qtcvXbt0qSn7_ivTAuqwpwiSnnmyCifcDKKBY7035q0LDDamQjS9vEaZjcM7gsj7yWrLWPVxfU97b08C2SrkVy1pilyz65gFn4Ug1U__U1Xve-Ov-lJPsY6MFUQ9QCyJDoa49TVFRO4CCwE_4yQGJsPX0dYl9qUX3N3AhR_3VMU24Sf2QIHTxyXPwNUdvpwo3zmBIekJCg1VWuqITTHCUebr9Ul1mV1cW5omrk_UZc467XLthjQJZTcNGzdPapqQu5nBUwgNlnNKqioqdWdCi8658SAE1SXnR-WOL7VhIv6P53VVVuA/https%3A%2F%2Flists.jboss.org%2Fmailman%2Flistinfo%2Fkeycloak-dev>
>
>