[keycloak-dev] Keycloak Clustering
gambol
gambol99 at gmail.com
Tue Oct 11 11:12:37 EDT 2016
As a short term fix ... removing any references to mping from
standalone-ha.xml fixed the issue, though i'm not sure how to select this
are runtime? ...
Rohith
On Tue, Oct 11, 2016 at 3:20 PM, gambol <gambol99 at gmail.com> wrote:
> Ok ... so the error appears to be related to it's use of multi-casting
> (which would not propagate nodes over flannel overlay network); as when I
> pin the keycloaks instances to the same kubernetes node, suddenly
> everything works when a new pod is added ...
>
> ] (1) [keycloak-2268126783-gjr2g]
> 14:00:57,413 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport]
> (MSC service thread 1-1) ISPN000094: Received new cluster view for channel
> server: [keycloak-2268126783-gjr2g|0] (1) [keycloak-2268126783-gjr2g]
> 14:00:57,414 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport]
> (MSC service thread 1-2) ISPN000094: Received new cluster view for channel
> ejb: [keycloak-2268126783-gjr2g|0] (1) [keycloak-2268126783-gjr2g]
> 14:00:57,415 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport]
> (MSC service thread 1-7) ISPN000094: Received new cluster view for channel
> hibernate: [keycloak-2268126783-gjr2g|0] (1) [keycloak-2268126783-gjr2g]
> 14:00:57,415 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport]
> (MSC service thread 1-5) ISPN000094: Received new cluster view for channel
> keycloak: [keycloak-2268126783-gjr2g|0] (1) [keycloak-2268126783-gjr2g]
> 14:00:57,419 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport]
> (MSC service thread 1-2) ISPN000079: Channel ejb local address is
> keycloak-2268126783-gjr2g, physical addresConnection to 10.70.2.71 closed.
>
>
> Placing on different nodes will result in and unanswered multicasting
>
> 14:14:52.676714 IP 10.10.69.4.32790 > 10.200.227.31.8200: Flags [.], ack
> 1, win 553, options [nop,nop,TS val 600739168 ecr 600901632], length 0
> 14:14:52.676877 IP 10.200.227.31.8200 > 10.10.69.4.32790: Flags [.], ack
> 1, win 235, options [nop,nop,TS val 600911648 ecr 600568905], length 0
> > 14:14:59.557124 IP 10.10.69.4.45700 > 230.0.0.4.45700: UDP, length 92
> > 14:14:59.557157 IP 10.10.69.4.45700 > 230.0.0.4.45700: UDP, length 92
> 14:15:02.628054 IP 10.200.227.31.8200 > 10.10.69.4.32790: Flags [.], ack
> 1, win 235, options [nop,nop,TS val 600921600 ecr 600568905], length 0
> 14:15:02.628073 IP 10.10.69.4.32790 > 10.200.227.31.8200: Flags [.], ack
> 1, win 553, options [nop,nop,TS val 600749119 ecr 600911648], length 0
> 14:15:02.692712 IP 10.10.69.4.32790 > 10.200.227.31.8200: Flags [.], ack
> 1, win 553, options [nop,nop,TS val 600749184 ecr 600911648], length 0
> 14:15:02.692894 IP 10.200.227.31.8200 > 10.10.69.4.32790: Flags [.], ack
> 1, win 235, options [nop,nop,TS val 600921664 ecr 600749119], length 0
> 14:15:07.636711 ARP, Request who-has 10.10.69.4 tell 10.10.69.1, length 28
> 14:15:07.636738 ARP, Reply 10.10.69.4 is-at 02:42:0a:0a:45:04, length 28
>
> Perhaps i've misunderstood tcpgossip? ... Anyone know how to stop using MC
> and where in the config this is set? ...
>
> Rohith
>
>
> On Tue, Oct 11, 2016 at 2:42 PM, gambol <gambol99 at gmail.com> wrote:
>
>>
>> So tcpdumping the in the gossip router I can see traffic from the
>> keycloak instances when it boots
>>
>> 13:23:09.679254 IP 10.10.13.2.57533 > 10.10.69.2.12001: Flags [F.], seq
>> 3916063057, ack 3487985413, win 13442, options [nop,nop,TS val 597445725
>> ecr 597241454], length 0
>> 13:23:09.679956 IP 10.10.69.2.12001 > 10.10.13.2.57533: Flags [F.], seq
>> 1, ack 1, win 210, options [nop,nop,TS val 597636171 ecr 597445725], length
>> 0
>> 13:23:09.680198 IP 10.10.69.2.12001 > 10.10.69.4.37303: Flags [P.], seq
>> 3146214056:3146214091, ack 2740076567, win 210, options [nop,nop,TS val
>> 597636171 ecr 597194069], length 35
>> 13:23:09.680269 IP 10.10.69.4.37303 > 10.10.69.2.12001: Flags [.], ack
>> 35, win 13442, options [nop,nop,TS val 597636171 ecr 597636171], length 0
>> 13:23:09.680550 IP 10.10.13.2.57533 > 10.10.69.2.12001: Flags [.], ack 2,
>> win 13442, options [nop,nop,TS val 597445726 ecr 597636171], length 0
>>
>> Oddly enough I see UDP from KC instance to multicast
>>
>> 13:37:54.224690 IP 10.10.69.4.45700 > 230.0.0.4.45700: UDP, length 92
>> E..x.. at ...Hj
>>
>> ....ee...8..DT.:......)%lp......keycloak-648398853-kfidy..
>>
>> None the less, the instances don't seem to be aware of one another ...
>> logging into one and refreshing will eventually hit the other and error
>> with unknown bearer; also when I had this working before in v1.9.0 bringing
>> up or killing a instance would show membership logs events on the console
>>
>> Rohith
>>
>>
>> On Tue, Oct 11, 2016 at 1:50 PM, gambol <gambol99 at gmail.com> wrote:
>>
>>> Hiya
>>>
>>> I'm running Keycloak inside Kubernetes and attempting to get clustering
>>> working. Multicasting is isn't available so i'm attempting to get the
>>> gossip protocol working.
>>>
>>> Version: 2.2.1-Final
>>>
>>> I've changed the standalone-ha.xml
>>>
>>> <subsystem xmlns="urn:jboss:domain:jgroups:4.0">
>>> <channels default="ee">
>>> <channel name="ee" stack="tcp"/>
>>> </channels>
>>> <stacks>
>>> <stack name="udp">
>>> <transport type="UDP" socket-binding="jgroups-udp"/>
>>> <protocol type="PING"/>
>>> <protocol type="MERGE3"/>
>>> <protocol type="FD_SOCK"
>>> socket-binding="jgroups-udp-fd"/>
>>> <protocol type="FD_ALL"/>
>>> <protocol type="VERIFY_SUSPECT"/>
>>> <protocol type="pbcast.NAKACK2"/>
>>> <protocol type="UNICAST3"/>
>>> <protocol type="pbcast.STABLE"/>
>>> <protocol type="pbcast.GMS"/>
>>> <protocol type="UFC"/>
>>> <protocol type="MFC"/>
>>> <protocol type="FRAG2"/>
>>> </stack>
>>> <stack name="tcp">
>>> <transport type="TCP" socket-binding="jgroups-tcp"/>
>>> <protocol type="TCPGOSSIP">
>>> <property name="initial_hosts">${env.GOS
>>> SIP_ROUTER_HOST:127.0.0.1}[12001]</property>
>>> </protocol>
>>> <protocol type="MPING" socket-binding="jgroups-mping"
>>> />
>>> <protocol type="MERGE3"/>
>>> <protocol type="FD_SOCK"
>>> socket-binding="jgroups-tcp-fd"/>
>>> <protocol type="FD"/>
>>> <protocol type="VERIFY_SUSPECT"/>
>>> <protocol type="pbcast.NAKACK2"/>
>>> <protocol type="UNICAST3"/>
>>> <protocol type="pbcast.STABLE"/>
>>> <protocol type="pbcast.GMS"/>
>>> <protocol type="MFC"/>
>>> <protocol type="FRAG2"/>
>>> </stack>
>>> </stacks>
>>>
>>> The gossip router service were using is jboss/jgroups-gossip
>>>
>>> 12:40:53,114 INFO [org.infinispan.remoting.tran
>>> sport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000078:
>>> Starting JGroups channel ejb
>>> 12:40:53,114 INFO [org.infinispan.remoting.tran
>>> sport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000078:
>>> Starting JGroups channel server
>>> 12:40:53,114 INFO [org.infinispan.remoting.tran
>>> sport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000078:
>>> Starting JGroups channel hibernate
>>> 12:40:53,114 INFO [org.infinispan.remoting.tran
>>> sport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000078:
>>> Starting JGroups channel keycloak
>>> 12:40:53,114 INFO [org.infinispan.remoting.tran
>>> sport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000078:
>>> Starting JGroups channel web
>>> 12:40:53,216 INFO [org.infinispan.remoting.tran
>>> sport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000094:
>>> Received new cluster view for channel ejb: [keycloak-4062290770-sn7jb|0]
>>> (1) [keycloak-4062290770-sn7jb]
>>> 12:40:53,216 INFO [org.infinispan.remoting.tran
>>> sport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000094:
>>> Received new cluster view for channel hibernate:
>>> [keycloak-4062290770-sn7jb|0] (1) [keycloak-4062290770-sn7jb]
>>> 12:40:53,216 INFO [org.infinispan.remoting.tran
>>> sport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000094:
>>> Received new cluster view for channel server: [keycloak-4062290770-sn7jb|0]
>>> (1) [keycloak-4062290770-sn7jb]
>>> 12:40:53,217 INFO [org.infinispan.remoting.tran
>>> sport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000094:
>>> Received new cluster view for channel keycloak:
>>> [keycloak-4062290770-sn7jb|0] (1) [keycloak-4062290770-sn7jb]
>>> 12:40:53,216 INFO [org.infinispan.remoting.tran
>>> sport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000094:
>>> Received new cluster view for channel web: [keycloak-4062290770-sn7jb|0]
>>> (1) [keycloak-4062290770-sn7jb]
>>> 12:40:53,221 INFO [org.infinispan.remoting.tran
>>> sport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000079:
>>> Channel keycloak local address is keycloak-4062290770-sn7jb, physical
>>> addresses are [10.10.69.4:7600]
>>> 12:40:53,221 INFO [org.infinispan.remoting.tran
>>> sport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000079:
>>> Channel server local address is keycloak-4062290770-sn7jb, physical
>>> addresses are [10.10.69.4:7600]
>>> 12:40:53,221 INFO [org.infinispan.remoting.tran
>>> sport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000079:
>>> Channel hibernate local address is keycloak-4062290770-sn7jb, physical
>>> addresses are [10.10.69.4:7600]
>>> 12:40:53,221 INFO [org.infinispan.remoting.tran
>>> sport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000079:
>>> Channel web local address is keycloak-4062290770-sn7jb, physical addresses
>>> are [10.10.69.4:7600]
>>> 12:40:53,221 INFO [org.infinispan.remoting.tran
>>> sport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000079:
>>> Channel ejb local address is keycloak-4062290770-sn7jb, physical addresses
>>> are [10.10.69.4:7600]
>>> 12:40:53,314 INFO [org.infinispan.factories.GlobalComponentRegistry]
>>> (MSC service thread 1-6) ISPN000128: Infinispan version: Infinispan 'Mahou'
>>> 8.1.0.Final
>>> 12:40:53,314 INFO [org.infinispan.factories.GlobalComponentRegistry]
>>> (MSC service thread 1-4) ISPN000128: Infinispan version: Infinispan 'Mahou'
>>> 8.1.0.Final
>>> 12:40:53,314 INFO [org.infinispan.factories.GlobalComponentRegistry]
>>> (MSC service thread 1-8) ISPN000128: Infinispan version: Infinispan 'Mahou'
>>> 8.1.0.Final
>>> 12:40:53,314 INFO [org.infinispan.factories.GlobalComponentRegistry]
>>> (MSC service thread 1-1) ISPN000128: Infinispan version: Infinispan 'Mahou'
>>> 8.1.0.Final
>>> 12:40:55,110 INFO [org.jboss.as.clustering.infinispan] (ServerService
>>> Thread Pool -- 60) WFLYCLINF0002: Started users cache from keycloak
>>> container
>>> 12:40:55,111 INFO [org.jboss.as.clustering.infinispan] (ServerService
>>> Thread Pool -- 52) WFLYCLINF0002: Started realms cache from keycloak
>>> container
>>> 12:40:55,114 INFO [org.jboss.as.clustering.infinispan] (ServerService
>>> Thread Pool -- 56) WFLYCLINF0002: Started sessions cache from keycloak
>>> container
>>> 12:40:55,113 INFO [org.jboss.as.clustering.infinispan] (ServerService
>>> Thread Pool -- 54) WFLYCLINF0002: Started work cache from keycloak container
>>> 12:40:55,117 INFO [org.jboss.as.clustering.infinispan] (ServerService
>>> Thread Pool -- 59) WFLYCLINF0002: Started offlineSessions cache from
>>> keycloak container
>>> 12:40:55,117 INFO [org.jboss.as.clustering.infinispan] (ServerService
>>> Thread Pool -- 58) WFLYCLINF0002: Started authorization cache from keycloak
>>> container
>>> 12:40:55,115 INFO [org.jboss.as.clustering.infinispan] (ServerService
>>> Thread Pool -- 55) WFLYCLINF0002: Started loginFailures cache from keycloak
>>> container
>>> 12:40:58,330 INFO [org.keycloak.services] (ServerService Thread Pool --
>>> 58) KC-SERVICES0001: Loading config from standalone.xml or domain.xml
>>> 12:41:00,911 INFO [org.jboss.as.clustering.infinispan] (ServerService
>>> Thread Pool -- 58) WFLYCLINF0002: Started userRevisions cache from keycloak
>>> container
>>> 12:41:00,921 INFO [org.jboss.as.clustering.infinispan] (ServerService
>>> Thread Pool -- 58) WFLYCLINF0002: Started realmRevisions cache from
>>> keycloak container
>>>
>>> But no matter what i seem to change ... I can't multiple pods to see the
>>> membership ... Note, i don't even see a reference to the gossip route
>>> itself, so i'm not entirely sure it's being used.
>>>
>>> Are there any working examples or perhaps something obvious i'm missing?
>>>
>>> Rohith
>>>
>>>
>>
>
More information about the keycloak-dev
mailing list