Hi! We have had a couple of strange issues where the HTTP connection pool for the HttpClient used by ServerRequest has been exhausted. Strange thing is, the thread dump on the JVM reports many threads waiting for a connection, but none actually using them. "http-apr-8080-exec-1" #33 daemon prio=5 os_prio=0 tid=0x00007f5c3400b000 nid=0x723a waiting on condition [0x00007f5bff7f6000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000eb3edac0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at org.apache.http.impl.conn.tsccm.WaitingThread.await(WaitingThread.java:159) at org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:398) at org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:298) at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:238) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:423) at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57) at org.keycloak.adapters.ServerRequest.invokeRefresh(ServerRequest.java:149) at org.keycloak.adapters.RefreshableKeycloakSecurityContext.refreshExpiredToken(RefreshableKeycloakSecurityContext.java:113) I'm still trying to figure out what's going on here, but while digging around I noticed that the HttpClientBuilder[1] does not set a timeout for obtaining a connection from the pool. Wouldn't it be a good idea to set one? Maybe 30s or so, just to avoid waiting indefinitely. The same applies to socket timeout as well. [1] https://github.com/keycloak/keycloak/blob/1.9.4.Final/services/src/main/j... Best regards, Thomas _______________________________________________ keycloak-dev mailing list keycloak-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-dev

Hi! In my opinion they are two separate issues, but related to each other. KEYCLOAK-3016 describes a situation where this behavior can be reproduced. As a safety measure I'd still define reasonable timeouts instead of waiting indefinitely after the issue has been fixed. I'm examining the testsuite to find the proper place for testing BasicAuthRequestAuthenticator error handling... Best regards, Thomas On Wed, May 18, 2016 at 9:47 AM, Stian Thorgersen <sthorger(a)redhat.com&gt; wrote: > I see you created https://issues.jboss.org/browse/KEYCLOAK-3016. Is that > the cause of this issue or are there potentially more issues here? > > On 17 May 2016 at 15:21, Thomas Raehalme < > thomas.raehalme(a)aitiofinland.com&gt; wrote: > >> Hi! >> >> We have had a couple of strange issues where the HTTP connection pool >> for the HttpClient used by ServerRequest has been exhausted. Strange thing >> is, the thread dump on the JVM reports many threads waiting for a >> connection, but none actually using them. >> >> "http-apr-8080-exec-1" #33 daemon prio=5 os_prio=0 >> tid=0x00007f5c3400b000 nid=0x723a waiting on condition [0x00007f5bff7f6000] >> java.lang.Thread.State: WAITING (parking) >> at sun.misc.Unsafe.park(Native Method) >> - parking to wait for <0x00000000eb3edac0> (a >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) >> at >> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) >> at >> org.apache.http.impl.conn.tsccm.WaitingThread.await(WaitingThread.java:159) >> at >> org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:398) >> at >> org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:298) >> at >> org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:238) >> at >> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:423) >> at >> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863) >> at >> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) >> at >> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106) >> at >> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57) >> at >> org.keycloak.adapters.ServerRequest.invokeRefresh(ServerRequest.java:149) >> at >> org.keycloak.adapters.RefreshableKeycloakSecurityContext.refreshExpiredToken(RefreshableKeycloakSecurityContext.java:113) >> >> I'm still trying to figure out what's going on here, but while digging >> around I noticed that the HttpClientBuilder[1] does not set a timeout for >> obtaining a connection from the pool. Wouldn't it be a good idea to set >> one? Maybe 30s or so, just to avoid waiting indefinitely. The same applies >> to socket timeout as well. >> >> [1] >> https://github.com/keycloak/keycloak/blob/1.9.4.Final/services/src/main/j... >> >> Best regards, >> Thomas >> >> _______________________________________________ >> keycloak-dev mailing list >> keycloak-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-dev >> > >