[keycloak-user] Wildfly adapter: failed to turn code into token: java.net.SocketException

[Marek Posolda]

Does this happen just if more users tries to login concurrently? It 
seems there is some limit somewhere for the number of maximum 
connections. Maybe it will help to increase maximum number of threads at 
the undertow subsystem on Keycloak side, or maybe at the AWS ELB load 
balancer side? Or maybe there is even some limit at the docker level?

AFAIK The Apache HTTP Client (which is used by our adapters under the 
hood for exchange code to token) has some retry strategy enabled by 
default. AFAIK we don't allow to tweak this, but it's likely 1 or 2 
retries for each HTTP failed request AFAIK. However this should be 
rather tweaked by increasing the pool of connections for your server. 
Relying on Apache HTTP client retryies doesn't look like good solution IMO.

Marek

On 25/11/16 01:48, Jesse Chahal wrote:
> We very recently started working on moving our web/app servers to AWS ECS
> (docker). We have been using docker locally for a very long time without
> issues.
> During this transition we started seeing intermittent login failures
> through the normal confidential browser redirect flow (not using implicit).
> It inconsistently fails during the process of turning an authorization code
> into a bearer+refresh_token. The exception is always the same as well. I
> have done
> tcpdumps packet analysis for things such as RST packets but have not seen
> anything that would result in this issue. I have confirmed that the adapter
> does
> have the authorization code (although I don't know how to validate it). I
> have confirmed that the the date+times are synchronized in both the auth
> server
> and the app server (therefore no weird expiration issues).
> I have posted a stacktrace from the adapter incase anyone else has seen
> this issue before. I'm hoping at the very least that maybe we could add a
> retry
> mechanism to the adapter if a Connection reset issues occurs if no other
> solutions present themselves.
>
> Current network setup:
> Keycloak.2.1.0.Final on Centos6 EC2/VM --> AWS ELB load balancer -->
> internet -->
>
>       ↓
>   <---------------------------------------------------------------------
> -----------
> ↓
> AWS NAT Gateway --> AWS ELB load balancer --> Amazon Linux ECS EC2/VM -->
> Wildfly10 docker container + keycloak 2.1.0-wildfly-adapter
>
>
> ERROR [org.keycloak.adapters.OAuthRequestAuthenticator] (default task-13) [
>   ] failed to turn code into token: java.net.SocketException: Connection
> reset
>          at java.net.SocketInputStream.read(SocketInputStream.java:209)
> [rt.jar:1.8.0_111]
>          at java.net.SocketInputStream.read(SocketInputStream.java:141)
> [rt.jar:1.8.0_111]
>          at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
> [jsse.jar:1.8.0_111]
>          at sun.security.ssl.InputRecord.read(InputRecord.java:503)
> [jsse.jar:1.8.0_111]
>          at
> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
> [jsse.jar:1.8.0_111]
>          at
> sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
> [jsse.jar:1.8.0_111]
>          at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
> [jsse.jar:1.8.0_111]
>          at
> org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
>          at
> org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
>          at
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
>          at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
>          at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
>          at
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
>          at
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
>          at
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
>          at
> org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:223)
>          at
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
>          at
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
>          at
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
>          at
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
>          at
> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
>          at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
>          at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
>          at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
>          at
> org.keycloak.adapters.ServerRequest.invokeAccessCodeToToken(ServerRequest.java:107)
>          at
> org.keycloak.adapters.OAuthRequestAuthenticator.resolveCode(OAuthRequestAuthenticator.java:327)
>          at
> org.keycloak.adapters.OAuthRequestAuthenticator.authenticate(OAuthRequestAuthenticator.java:273)
>          at
> org.keycloak.adapters.RequestAuthenticator.authenticate(RequestAuthenticator.java:130)
>          at
> org.keycloak.adapters.undertow.AbstractUndertowKeycloakAuthMech.keycloakAuthenticate(AbstractUndertowKeycloakAuthMech.java:110)
>          at
> org.keycloak.adapters.undertow.ServletKeycloakAuthMech.authenticate(ServletKeycloakAuthMech.java:92)
>          at
> io.undertow.security.impl.SecurityContextImpl$AuthAttempter.transition(SecurityContextImpl.java:233)
>          at
> io.undertow.security.impl.SecurityContextImpl$AuthAttempter.transition(SecurityContextImpl.java:250)
>          at
> io.undertow.security.impl.SecurityContextImpl$AuthAttempter.access$100(SecurityContextImpl.java:219)
>          at
> io.undertow.security.impl.SecurityContextImpl.attemptAuthentication(SecurityContextImpl.java:121)
>          at
> io.undertow.security.impl.SecurityContextImpl.authTransition(SecurityContextImpl.java:96)
>          at
> io.undertow.security.impl.SecurityContextImpl.authenticate(SecurityContextImpl.java:89)
>          at
> io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:55)
>          at
> io.undertow.server.handlers.DisableCacheHandler.handleRequest(DisableCacheHandler.java:33)
>          at
> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>          at
> io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:51)
>          at
> io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
>          at
> io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
>          at
> io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:56)
>          at
> io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
>          at
> io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
>          at
> io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
>          at
> io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
>          at
> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>          at
> org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
>          at
> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>          at
> org.keycloak.adapters.undertow.ServletPreAuthActionsHandler.handleRequest(ServletPreAuthActionsHandler.java:69)
>          at
> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>          at
> io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)
>          at
> io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
>          at
> io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
>          at
> io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
>          at
> io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
>          at
> io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [rt.jar:1.8.0_111]
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [rt.jar:1.8.0_111]
>          at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_111]
>
> 2016-11-24 22:37:14,255 WARN
>   [com.indicee.service.authorization.EjbTimerInterceptor] (default task-13)
> [  ] UserIdentityServiceImpl:findByEmail took 484 ms
> 2016-11-24 22:37:14,256 WARN
>   [com.indicee.service.authorization.EjbTimerInterceptor] (default task-13)
> [  ] AccountSrvImpl:isCurrentAccountUnknown took 499 ms
> 2016-11-24 22:37:14,257 WARN
>   [com.indicee.service.authorization.EjbTimerInterceptor] (default task-13)
> [  ] BrandedVariableLookupSrvImpl:cacheKey took 500 ms
> 2016-11-24 22:37:14,959 WARN
>   [com.indicee.service.authorization.EjbTimerInterceptor] (default task-13)
> [  ] UserIdentityServiceImpl:findByEmail took 697 ms
> 2016-11-24 22:37:14,960 WARN
>   [com.indicee.service.authorization.EjbTimerInterceptor] (default task-13)
> [  ] AccountSrvImpl:isCurrentAccountUnknown took 701 ms
> 2016-11-24 22:37:14,962 WARN
>   [com.indicee.service.authorization.EjbTimerInterceptor] (default task-13)
> [  ] BrandedVariableLookupSrvImpl:getVariables took 703 ms
> 2016-11-24 22:37:15,095 INFO
>   [com.indicee.service.servlet.filter.ClientRequestIdFilter] (default
> task-14) [  ] Setting clientId [gwt_43cpg2jpm5oklfjkuue204jsmh] for session
> [qAYdgQ6PytsZ8kG5ah6mxEdmGeNPQpgT_2ExqU6A] created [1480026525644]
> 2016-11-24 22:37:15,096 INFO
>   [com.indicee.service.servlet.filter.ClientRequestIdFilter] (default
> task-14) [  ] clientRequestId via atomic integer =
> [gwt_43cpg2jpm5oklfjkuue204jsmh_0]
> 2016-11-24 22:37:15,138 INFO
>   [com.indicee.service.servlet.filter.ClientRequestIdFilter] (default
> task-15) [  ] clientRequestId via atomic integer =
> [gwt_43cpg2jpm5oklfjkuue204jsmh_1]
> 2016-11-24 22:37:15,224 INFO
>   [com.indicee.service.servlet.filter.ClientRequestIdFilter] (default
> task-12) [  ] clientRequestId via atomic integer =
> [gwt_43cpg2jpm5oklfjkuue204jsmh_2]
> 2016-11-24 22:37:15,635 WARN
>   [com.indicee.service.authorization.EjbTimerInterceptor]
> (dInitialHandler.java:284)
>          at
> io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletIni
> tialHandler.java:263)
>          at
> io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialH
> andler.java:81)
>          at
> io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletIni
> tialHandler.java:174)
>          at
> io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
>          at
> io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:114
> 2) [rt.jar:1.8.0_111]
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:61
> 7) [rt.jar:1.8.0_111]
>          at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_111]
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user