[keycloak-user] Standalone HA tokens not immediately shared among nodes

[Sebastian Laskawiec]

Hmmm this sounds a bit weird... like there was some delay in the
communication path.

Could you please look through your logs and look for lines including "view"
keyword? Are there two nodes, as expected? How the timestamps relate to
your experiment?

On Wed, Sep 12, 2018 at 1:01 AM D V <dv at glyphy.com> wrote:

> Hi list,
>
> I'm trying to cluster several Keycloak 4.0.0 nodes running in docker
> containers. I'm seeing "interesting" (unexpected) behaviour when requesting
> new OIDC tokens with "grant_type=password" (Direct Grant/ROPC) and then
> attempting to get a new set with "grant_type=refresh_token" .
>
> After I start two nodes (containers), if I issue a "grant_type=password"
> request to the node that started first, requests for
> "grant_type=refresh_token" on the newer node fail. If I issue the
> "grant_type=password" request to the node that started last, requests for
> "grant_type=refresh_token" succeed on any node AND all future
> password/refresh_token requests work correctly no matter which node handles
> the request.
>
> So, let's say node1 starts first and node2 starts second:
> 1. Password auth on node1: OK
> 2. Refresh token auth on node2 with token from previous step: Error:
> invalid_grant (Invalid refresh token)
> 3. Refresh token auth on node1 with token from step 1: OK (new set of
> refresh+access tokens)
> BUT!
> 4. Password auth on node2: OK
> 5. Refresh token auth on node1 with token from previous step: OK! (new set
> of refresh+access tokens)
> 6. Refresh token auth on node2 with token from step 4: OK
> 7. Password auth sequence from steps 1-3: also OK!
>
> It's as though the node that starts most recently needs a password auth
> request to "wake up" and start communicating with the rest of the cluster.
> Once it does, everything's in sync.
>
> Some facts that are hopefully relevant:
> * Keycloak 4.0.0 docker image base
> * standalone-ha.xml from distribution with changes in JGroups subsystem.
> I'm using JDBC PING configured for the same DB as Keycloak itself, which is
> MySQL 5.7. See the subsystem config below.
> * Custom org.keycloak.storage.UserStorageProviderFactory SPI, which creates
> a provider that makes an HTTP call to an external authentication service to
> validate username/password credentials.
> * A couple of custom themes.
> * One realm with a handful of clients provisioned via a shell script that
> just calls kcadm.sh and jboss-cli.sh
> * There's a simple LB in front of both instances
>
>
> JGroups subsystem config:
>         <subsystem xmlns="urn:jboss:domain:jgroups:5.0">
>             <channels default="ee">
>                 <channel name="ee" stack="tcp" cluster="ejb"/>
>             </channels>
>             <stacks>
>                 <stack name="tcp">
>                     <transport type="TCP" socket-binding="jgroups-tcp">
>                         <property
> name="external_addr">${env.HOST}</property>
>                         <property
> name="external_port">${env.PORT_7600}</property>
>                     </transport>
>                     <protocol type="org.jgroups.protocols.JDBC_PING">
>                         <property name="datasource_jndi_name">
>                             java:jboss/datasources/KeycloakDS
>                         </property>
>                     </protocol>
>                     <protocol type="MERGE3"/>
>                     <protocol type="FD_SOCK"/>
>                     <protocol type="FD_ALL"/>
>                     <protocol type="VERIFY_SUSPECT"/>
>                     <protocol type="pbcast.NAKACK2"/>
>                     <protocol type="UNICAST3"/>
>                     <protocol type="pbcast.STABLE"/>
>                     <protocol type="pbcast.GMS"/>
>                     <protocol type="MFC"/>
>                     <protocol type="FRAG2"/>
>                 </stack>
>             </stacks>
>         </subsystem>
>
> $HOST and $PORT_7600 are set to external host:port combination that allows
> the two instances to communicate.
>
> There's also a socket-binding to a public interface:
> <socket-binding name="jgroups-tcp" port="7600"/>
>
> In the JGroups and Infinispan log entries I can see the two nodes do find
> each other and are able to communicate. I haven't been able to get
> ispn-cli.sh to connect to the internal Infinispan instances running in
> containers, so I can't confirm that they have the same entries, but as
> described in flows above they do eventually work together.
>
> Is there a configuration change I'm missing somewhere to make the new node
> joining the cluster become aware of the other one?
>
> Thanks for any help,
> DV
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
>