[keycloak-user] NoRouteToHostException - Using external Infinispan with Keycloak on OpenShift platform

[Vikrant Singh]

Hi,

I am running Keycloak(3.1.0.Final) on Openshift platform. I am using
external infinispan(9.0.1-Final) for sessions, work and offlineSessions
cache to achieve multi datacenter failover.

Below is configuration for infinispan remote-store in Keycloak

> <local-cache name="sessions">
>     <remote-store passivation="false" fetch-state="false" purge="false"
> preload="false" shared="true" cache="sessions"
> remote-servers="remote-cache">
> <property name="rawValues">true</property>
> <property
> name="marshaller">org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory</property>
>     </remote-store>
> </local-cache>


 <outbound-socket-binding name="remote-cache">
>    <remote-destination host="${env.INFINISPAN_HOST}"
> port="${env.INFINISPAN_PORT:11222}"/>
> </outbound-socket-binding>


External Infinispan cluster is front ended by a load-balancer(kubernetes
service) which provides a static hostname for infinispan and this hostname
is configured in keycloak for keycloak to infinispan communication.

The setup work fine but if all instances(pods) in external infinispan goes
down and we bring up the cluster again, keycloak is not able to get to new
infinispan instance and it keeps trying on old ip address with below error.

The issue seems to be keycloak trying to use ip address instead of load
balancer hostname provided in configuration. As we are running on
openshift, infinispan instances will get new ip address each time it is
restarted.

 ERROR
[org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation]
(persistence-thread--p8-t108) ISPN004007: Exception encountered. Retry 10
out of 10: org.infinispan.client.hotrod.exceptions.TransportException::
Could not fetch transport
at
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:405)
at
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:244)
at
org.infinispan.client.hotrod.impl.operations.BulkGetKeysOperation.getTransport(BulkGetKeysOperation.java:29)
at
org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:53)
at
org.infinispan.client.hotrod.impl.RemoteCacheImpl.keySet(RemoteCacheImpl.java:670)
at
org.infinispan.persistence.remote.RemoteStore.process(RemoteStore.java:135)
at
org.infinispan.persistence.manager.PersistenceManagerImpl.processOnAllStores(PersistenceManagerImpl.java:447)
at
org.infinispan.persistence.manager.PersistenceManagerImpl.processOnAllStores(PersistenceManagerImpl.java:432)
at
org.infinispan.persistence.util.PersistenceManagerCloseableSupplier.lambda$get$261(PersistenceManagerCloseableSupplier.java:115)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.infinispan.client.hotrod.exceptions.TransportException::
Could not connect to server: /10.0.34.100:11222
at
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:78)
at
org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:37)
at
org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:16)
at
org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
at
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:400)
... 11 more
Caused by: java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at sun.nio.ch.SocketAdaptor.connect(Unknown Source)
at
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:68)
... 15 more

Is there anyway we can force keycloak to use hostname instead of resolving
to ip address?

Thanks,
Vikrant