[keycloak-user] Keycloak cluster setup on Openshift

[Jon Huang]

Dear Keycloakers

On my local environment, I set up Keycloak cluster with multi-cast and it's
OK.
However, there are some issues which might be related to infinispan when I
migrate to Openshift with KUBE_PING.
(btw, I tested with Keycloak version: 4.8.1 docker version)

I put detail log in attachment and hope it helps. (log below is abstract
version)
It seems that infinispan timeout and not working correctly (which works in
my local environment though)
Does anyone have same experience on Openshift?
Thanks

*Firstly, node 1 detected node2*
[org.infinispan.CLUSTER] (thread-15,ejb,kc-22-qzws9) ISPN000094: Received
new cluster view for channel ejb: [kc-22-qzws9|5] (2) [kc-22-qzws9,
kc-22-wf2pf]
[org.infinispan.CLUSTER] (thread-15,ejb,kc-22-qzws9) ISPN100000: Node
kc-22-wf2pf joined the cluster
[org.infinispan.CLUSTER] (remote-thread--p13-t6) [Context=loginFailures]
ISPN100002: Starting rebalance with members [kc-22-qzws9, kc-22-wf2pf],
phase READ_OLD_WRITE_ALL, topology id 2
...
*Then some error happened*
*[log from node1:]*
[org.infinispan.topology.ClusterTopologyManagerImpl]
(transport-thread--p24-t3) ISPN000197: Error updating cluster member list:
org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out
waiting for responses for request 2 from kc-22-wf2pf
at
org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
at
org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
at
org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 1 more
[org.infinispan.statetransfer.StateConsumerImpl] (transport-thread--p16-t9)
ISPN000208: No live owners found for segments {0-255} of cache
clientSessions. Excluded owners: []
*[log from node2:]*
[org.jboss.msc.service.fail] (ServerService Thread Pool -- 58) MSC000001:
Failed to start service
org.wildfly.clustering.infinispan.cache.keycloak.offlineClientSessions:
org.jboss.msc.service.StartException in service
org.wildfly.clustering.infinispan.cache.keycloak.offlineClientSessions:
org.infinispan.commons.CacheException: Unable to invoke method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
at
org.wildfly.clustering.service.FunctionalService.start(FunctionalService.java:70)
at
org.wildfly.clustering.service.AsyncServiceConfigurator$AsyncService.lambda$start$0(AsyncServiceConfigurator.java:117)
...
Caused by: org.infinispan.commons.CacheException: Unable to invoke method
public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
at
org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83)
at
org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
at
org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
at
org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
... 7 more
Caused by: org.infinispan.commons.CacheException: Initial state transfer
timed out for cache offlineClientSessions on kc-22-wf2pf
at
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:233)
... 30 more
[org.jboss.as.controller.management-operation] (Controller Boot Thread)
WFLYCTL0013: Operation ("add") failed - address: ([
    ("subsystem" => "infinispan"),
    ("cache-container" => "keycloak"),
    ("replicated-cache" => "work")
]) - failure description: {"WFLYCTL0080: Failed services" => {"
org.wildfly.clustering.infinispan.cache.keycloak.work" =>
"org.infinispan.commons.CacheException: Unable to invoke method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
    Caused by: org.infinispan.commons.CacheException: Unable to invoke
method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
    Caused by: org.infinispan.commons.CacheException: Initial state
transfer timed out for cache work on kc-22-wf2pf"}}
-------------- next part --------------
A non-text attachment was scrubbed...
Name: keycloakError.log
Type: application/octet-stream
Size: 10950 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20190430/fb6f2daa/attachment-0001.obj