[keycloak-dev] Clustering configuration

Thu Sep 13 11:22:00 EDT 2018

Hi Cédric,

Thanks for your help. I tested your configuration and it seems to work but the effects are still kind of unexpected.

It looks like the cluster is formed correctly. But the way JDBC_PING itself works is completely different from before. In the past, every node removed and readded itself to the JGROUPSPING table every few seconds. Now it looks like only the master node updates the JGROUPSPING table and only when it detects a change. It also always inserts its own address for every node. That makes the JGROUPSPING table look kind of strange as it does not contain the correct addresses for all the nodes. I have not found documentation for such a big change so I am not sure this is all intended...

Best regards,
Sebastian

Mit freundlichen Grüßen / Best regards

Dr.-Ing.  Sebastian Schuster

Open Source Services (INST-CSS/BSV-OS) 
Bosch Software Innovations GmbH | Ullsteinstr. 128 | 12109 Berlin | GERMANY | www.bosch-si.com
Tel. +49 30 726112-485 | Fax +49 30 726112-100 | Sebastian.Schuster at bosch-si.com

Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411 B 
Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke; Geschäftsführung: Dr. Stefan Ferber, Michael Hahn 

-----Original Message-----
From: cedric at couralet.eu <cedric at couralet.eu> 
Sent: Mittwoch, 12. September 2018 13:33
To: Schuster Sebastian (INST-CSS/BSV-OS) <Sebastian.Schuster at bosch-si.com>
Cc: Thomas Darimont <thomas.darimont at googlemail.com>; Radoslav Husar <rhusar at redhat.com>; Bela Ban <bban at redhat.com>; Paul Ferraro <paul.ferraro at redhat.com>; keycloak-dev <keycloak-dev at lists.jboss.org>; Tarrant, Tristan <ttarrant at redhat.com>
Subject: Re: [keycloak-dev] Clustering configuration

Hi Sebastian,

Le Mercredi, Septembre 12, 2018 11:56 CEST, "Schuster Sebastian (INST-CSS/BSV-OS)" <Sebastian.Schuster at bosch-si.com> a écrit: 

> Guess what, our JDBC_PING configuration not working with 4.4.0.Final is what I am currently working on.

If it helps, I had some difficulties configuring JDBC_PING on keycloak 4.4.0.Final. 

My final working configuration is :
       <subsystem xmlns="urn:jboss:domain:jgroups:6.0">
            <channels default="ee">
                <channel name="ee" stack="tcpping"/>
            </channels>
            <stacks>
                <stack name="tcpping">
                    <transport type="TCP" socket-binding="jgroups-tcp">
                        <property name="external_addr">
                            ${jgroups.bind.address:127.0.0.1}
                        </property>
                        <property name="bind_addr">
                            ${jgroups.bind_addr:SITE_LOCAL}
                        </property>
                    </transport>
                    <jdbc-protocol type="JDBC_PING" data-source="KeycloakDS">
                        <property name="initialize_sql">
                            CREATE TABLE IF NOT EXISTS JGROUPSPING (own_addr varchar(200) NOT NULL,bind_addr varchar(200) NOT NULL,created timestamp NOT NULL,cluster_name varchar(200) NOT NULL,ping_data BYTEA,constraint PK_JGROUPSPING PRIMARY KEY (own_addr, cluster_name))
                        </property>
                        <property name="insert_single_sql">
                            INSERT INTO JGROUPSPING (own_addr, bind_addr, created, cluster_name, ping_data) values (?,'${jgroups.bind.address:127.0.0.1}',NOW(), ?, ?)
                        </property>
                        <property name="delete_single_sql">
                            DELETE FROM JGROUPSPING WHERE own_addr=? AND cluster_name=?
                        </property>
                        <property name="select_all_pingdata_sql">
                            SELECT ping_data FROM JGROUPSPING WHERE cluster_name=?;
                        </property>
                    </jdbc-protocol>
                    <protocol type="MERGE3"/>
                    <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd">
                        <property name="external_addr">
                            ${jgroups.bind.address:127.0.0.1}
                        </property>
                    </protocol>
                    <protocol type="FD"/>
                    <protocol type="VERIFY_SUSPECT"/>
                    <protocol type="pbcast.NAKACK2"/>
                    <protocol type="UNICAST3"/>
                    <protocol type="pbcast.STABLE"/>
                    <protocol type="pbcast.GMS"/>
                    <protocol type="MFC"/>
                    <protocol type="FRAG2"/>
                </stack>
            </stacks>
        </subsystem>

I do it in my own docker image where I change the configuration with jboss-cli (as in the officilal) with this file :
/subsystem=infinispan/cache-container=keycloak/distributed-cache=sessions:remove
/subsystem=infinispan/cache-container=keycloak/replicated-cache=sessions:add()
/subsystem=infinispan/cache-container=keycloak/replicated-cache=sessions:write-attribute(name="mode",value="SYNC")

/subsystem=infinispan/cache-container=keycloak/distributed-cache=authenticationSessions:remove
/subsystem=infinispan/cache-container=keycloak/distributed-cache=offlineSessions:remove
/subsystem=infinispan/cache-container=keycloak/distributed-cache=clientSessions:remove
/subsystem=infinispan/cache-container=keycloak/distributed-cache=offlineClientSessions:remove
/subsystem=infinispan/cache-container=keycloak/distributed-cache=loginFailures:remove

/subsystem=infinispan/cache-container=keycloak/distributed-cache=authenticationSessions:add(mode="SYNC",owners="2")
/subsystem=infinispan/cache-container=keycloak/distributed-cache=offlineSessions:add(mode="SYNC",owners="2")
/subsystem=infinispan/cache-container=keycloak/distributed-cache=clientSessions:add(mode="SYNC",owners="2")
/subsystem=infinispan/cache-container=keycloak/distributed-cache=offlineClientSessions:add(mode="SYNC",owners="2")
/subsystem=infinispan/cache-container=keycloak/distributed-cache=loginFailures:add(mode="SYNC",owners="2")

/subsystem=jgroups/stack=tcpping:add()
/subsystem=jgroups/stack=tcpping/transport=TCP:add(socket-binding=jgroups-tcp)
/subsystem=jgroups/stack=tcpping/transport=TCP/property=external_addr:add(value=${jgroups.bind.address:127.0.0.1})
/subsystem=jgroups/stack=tcpping/transport=TCP/property=bind_addr:add(value=${jgroups.bind_addr:SITE_LOCAL})

/subsystem=jgroups/stack=tcpping/protocol=JDBC_PING:add(data-source="KeycloakDS", properties=[initialize_sql="CREATE TABLE IF NOT EXISTS JGROUPSPING (own_addr varchar(200) NOT NULL,bind_addr varchar(200) NOT NULL,created timestamp NOT NULL,cluster_name varchar(200) NOT NULL,ping_data BYTEA,constraint PK_JGROUPSPING PRIMARY KEY (own_addr, cluster_name))",insert_single_sql="INSERT INTO JGROUPSPING (own_addr, bind_addr, created, cluster_name, ping_data) values (?,'${jgroups.bind.address:127.0.0.1}',NOW(), ?, ?)",delete_single_sql="DELETE FROM JGROUPSPING WHERE own_addr=? AND cluster_name=?",select_all_pingdata_sql="SELECT ping_data FROM JGROUPSPING WHERE cluster_name=?;"])

/subsystem=jgroups/stack=tcpping/protocol=MERGE3:add()
/subsystem=jgroups/stack=tcpping:add-protocol(type="FD_SOCK",socket-binding="jgroups-tcp-fd")
/subsystem=jgroups/stack=tcpping/protocol=FD_SOCK/property=external_addr:add(value=${jgroups.bind.address:127.0.0.1})
/subsystem=jgroups/stack=tcpping/protocol=FD:add()
/subsystem=jgroups/stack=tcpping/protocol=VERIFY_SUSPECT:add()
/subsystem=jgroups/stack=tcpping/protocol=pbcast.NAKACK2:add()
/subsystem=jgroups/stack=tcpping/protocol=UNICAST3:add()
/subsystem=jgroups/stack=tcpping/protocol=pbcast.STABLE:add()
/subsystem=jgroups/stack=tcpping/protocol=pbcast.GMS:add()
/subsystem=jgroups/stack=tcpping/protocol=MFC:add()
/subsystem=jgroups/stack=tcpping/protocol=FRAG2:add()

/subsystem=jgroups/channel=ee:remove
/subsystem=jgroups/channel=ee:add(stack=tcpping)
/subsystem=jgroups:write-attribute(name=default-channel, value=ee)

/socket-binding-group=standard-sockets/socket-binding=jgroups-tcp:write-attribute(name="interface",value="private")
/socket-binding-group=standard-sockets/socket-binding=jgroups-tcp-fd:add()
/socket-binding-group=standard-sockets/socket-binding=jgroups-tcp-fd:write-attribute(name="interface",value="private")
/socket-binding-group=standard-sockets/socket-binding=jgroups-tcp-fd:write-attribute(name="port",value="57600")

/subsystem=jgroups/stack=tcp:remove
/subsystem=jgroups/stack=udp:remove

I am not sure all is good in this but it works (in my environment :) ).

(the difficulties I had was that if the cli file is like :
...
	/subsystem=jgroups/stack=tcpping/protocol=JDBC_PING:add(data-source="KeycloakDS")
	/subsystem=jgroups/stack=tcpping/protocol=JDBC_PING/property=datasource_jndi_name:add(value=java:jboss/datasources/KeycloakDS)
/subsystem=jgroups/stack=tcpping/protocol=JDBC_PING/property=otherproperty:add(value=other_value)
...

the configuration in xml is :
  <protocol type="org.apache.jgroups.JDBC_PING" >

Which doesn't work (don't know why). There is not a lot of documentation on this, so I'm listening to all suggestions.

Cheers,
Cédric Couralet