[keycloak-dev] HttpClientBuilder and timeouts

[Thomas Raehalme]

Hi!

We have had a couple of strange issues where the HTTP connection pool for
the HttpClient used by ServerRequest has been exhausted. Strange thing is,
the thread dump on the JVM reports many threads waiting for a connection,
but none actually using them.

"http-apr-8080-exec-1" #33 daemon prio=5 os_prio=0 tid=0x00007f5c3400b000
nid=0x723a waiting on condition [0x00007f5bff7f6000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000eb3edac0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
org.apache.http.impl.conn.tsccm.WaitingThread.await(WaitingThread.java:159)
        at
org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:398)
        at
org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:298)
        at
org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:238)
        at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:423)
        at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
        at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
        at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
        at
org.keycloak.adapters.ServerRequest.invokeRefresh(ServerRequest.java:149)
        at
org.keycloak.adapters.RefreshableKeycloakSecurityContext.refreshExpiredToken(RefreshableKeycloakSecurityContext.java:113)

I'm still trying to figure out what's going on here, but while digging
around I noticed that the HttpClientBuilder[1] does not set a timeout for
obtaining a connection from the pool. Wouldn't it be a good idea to set
one? Maybe 30s or so, just to avoid waiting indefinitely. The same applies
to socket timeout as well.

[1]
https://github.com/keycloak/keycloak/blob/1.9.4.Final/services/src/main/java/org/keycloak/connections/httpclient/HttpClientBuilder.java

Best regards,
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-dev/attachments/20160517/5232bb86/attachment.html