[keycloak-dev] HttpClientBuilder and timeouts

Hi! We have had a couple of strange issues where the HTTP connection pool for the HttpClient used by ServerRequest has been exhausted. Strange thing is, the thread dump on the JVM reports many threads waiting for a connection, but none actually using them. "http-apr-8080-exec-1" #33 daemon prio=5 os_prio=0 tid=0x00007f5c3400b000 nid=0x723a waiting on condition [0x00007f5bff7f6000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000eb3edac0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at org.apache.http.impl.conn.tsccm.WaitingThread.await(WaitingThread.java:159) at org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:398) at org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:298) at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:238) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:423) at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57) at org.keycloak.adapters.ServerRequest.invokeRefresh(ServerRequest.java:149) at org.keycloak.adapters.RefreshableKeycloakSecurityContext.refreshExpiredToken(RefreshableKeycloakSecurityContext.java:113) I'm still trying to figure out what's going on here, but while digging around I noticed that the HttpClientBuilder[1] does not set a timeout for obtaining a connection from the pool. Wouldn't it be a good idea to set one? Maybe 30s or so, just to avoid waiting indefinitely. The same applies to socket timeout as well. [1] https://github.com/keycloak/keycloak/blob/1.9.4.Final/services/src/main/j... Best regards, Thomas