[keycloak-user] db deadlock in concurrence environment

Stian Thorgersen sthorger at redhat.com
Tue Mar 8 01:34:31 EST 2016 

As it times out after 30 seconds to obtain a connection it looks more like
connections/transactions are not being closed. Is your federation provider
using the same datasource? Any chance it's not closing the connections
properly?

On 7 March 2016 at 22:37, Thomas Darimont <thomas.darimont at googlemail.com>
wrote:

> Hello Alex,
>
> looks like your database connection pool is exhausted - how is your
> connection pool configured in wildfly?
>
> Cheers,
> Thomas
>
> 2016-03-07 22:32 GMT+01:00 alex orl <alex_orl1079 at yahoo.it>:
>
>> Hi,
>> i wrote my custom federation provider.
>> All the synchronize operations (roles and user-roles) are done inside the
>> getUserByUsername and the isValid method.
>> This way every time a user or a new-user logins into my system, all the
>> roles and role-mapping are kept up to date.
>> All seems to work well with one user, but now i'm experiencing a lot of
>> exceptions in a concurrence environment.
>> These are the tests i'm launching:
>> 1) 20 concurrent threads, each of which tries to login, to send a REST
>> request (to backend) and finally to logout. All 20 threads login with the
>> same test username.
>> 2) 200 concurrent threads, doing the same as at the previous point, with
>> 20 differents username.
>>
>> Everytime each federation provider instance tries to synchronize all
>> realm roles...adding or removing roles in concurrence, i catch this error:
>>
>>
>> 16:10:08,228 ERROR [io.undertow.request] (default task-96) UT005023:
>> Exception handling request to
>> /auth/realms/MyRealm/protocol/openid-connect/token:
>> java.lang.RuntimeException: reques
>> t path: /auth/realms/MyRealm/protocol/openid-connect/token
>>         at
>> org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:75)
>>         at
>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
>>         at
>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
>>         at
>> io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:85)
>>         at
>> io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
>>         at
>> io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
>>         at
>> org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
>>         at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>>         at
>> io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
>>         at
>> io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
>>         at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>>         at
>> io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
>>         at
>> io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
>>         at
>> io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58)
>>         at
>> io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:72)
>>         at
>> io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
>>         at
>> io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76)
>>         at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>>         at
>> org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
>>         at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>>         at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>>         at
>> io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:282)
>>         at
>> io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:261)
>>         at
>> io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80)
>>         at
>> io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172)
>>         at
>> io.undertow.server.Connectors.executeRootHandler(Connectors.java:199)
>>         at
>> io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:774)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>         at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.jboss.resteasy.spi.UnhandledException:
>> javax.persistence.PersistenceException:
>> org.hibernate.exception.GenericJDBCException: Could not open connection
>>         at
>> org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:76)
>>         at
>> org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:212)
>>         at
>> org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:149)
>>         at
>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:372)
>>         at
>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
>>         at
>> org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
>>         at
>> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
>>         at
>> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
>>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>>         at
>> io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86)
>>         at
>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:130)
>>         at
>> org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:61)
>>         ... 29 more
>> Caused by: javax.persistence.PersistenceException:
>> org.hibernate.exception.GenericJDBCException: Could not open connection
>>         at
>> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1763)
>>         at
>> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1677)
>>         at
>> org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1771)
>>         at
>> org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:64)
>>         at
>> org.keycloak.connections.jpa.JpaKeycloakTransaction.begin(JpaKeycloakTransaction.java:22)
>>         at
>> org.keycloak.services.DefaultKeycloakTransactionManager.enlist(DefaultKeycloakTransactionManager.java:25)
>>         at
>> org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.create(DefaultJpaConnectionProviderFactory.java:46)
>>         at
>> org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.create(DefaultJpaConnectionProviderFactory.java:30)
>>         at
>> org.keycloak.services.DefaultKeycloakSession.getProvider(DefaultKeycloakSession.java:103)
>>         at
>> org.keycloak.models.jpa.JpaUserProviderFactory.create(JpaUserProviderFactory.java:34)
>>         at
>> org.keycloak.models.jpa.JpaUserProviderFactory.create(JpaUserProviderFactory.java:16)
>>         at
>> org.keycloak.services.DefaultKeycloakSession.getProvider(DefaultKeycloakSession.java:103)
>>         at
>> org.keycloak.models.cache.infinispan.DefaultCacheUserProvider.getDelegate(DefaultCacheUserProvider.java:50)
>>         at
>> org.keycloak.models.cache.infinispan.DefaultCacheUserProvider.getUserByUsername(DefaultCacheUserProvider.java:147)
>>         at
>> org.keycloak.models.UserFederationManager.getUserByUsername(UserFederationManager.java:180)
>>         at
>> org.keycloak.models.utils.KeycloakModelUtils.findUserByNameOrEmail(KeycloakModelUtils.java:246)
>>         at
>> org.keycloak.authentication.authenticators.directgrant.ValidateUsername.authenticate(ValidateUsername.java:47)
>>         at
>> org.keycloak.authentication.DefaultAuthenticationFlow.processFlow(DefaultAuthenticationFlow.java:155)
>>         at
>> org.keycloak.authentication.AuthenticationProcessor.authenticateOnly(AuthenticationProcessor.java:776)
>>         at
>> org.keycloak.protocol.oidc.endpoints.TokenEndpoint.buildResourceOwnerPasswordCredentialsGrant(TokenEndpoint.java:369)
>>         at
>> org.keycloak.protocol.oidc.endpoints.TokenEndpoint.build(TokenEndpoint.java:110)
>>         at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source)
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>         at java.lang.reflect.Method.invoke(Method.java:497)
>>         at
>> org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137)
>>         at
>> org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:296)
>>         at
>> org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:250)
>>         at
>> org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:140)
>>         at
>> org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:109)
>>         at
>> org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:135)
>>         at
>> org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:103)
>>         at
>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356)
>>         ... 37 more
>> Caused by: org.hibernate.exception.GenericJDBCException: Could not open
>> connection
>>         at
>> org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54)
>>         at
>> org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:126)
>>         at
>> org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:112)
>>         at
>> org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:235)
>>         at
>> org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:171)
>>         at
>> org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:67)
>>         at
>> org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:162)
>>         at
>> org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1471)
>>         at
>> org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:61)
>>         ... 65 more
>> Caused by: java.sql.SQLException: javax.resource.ResourceException:
>> IJ000453: Unable to get managed connection for
>> java:jboss/datasources/KeycloakOracleDS
>>         at
>> org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:146)
>>         at
>> org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:67)
>>         at
>> org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:139)
>>         at
>> org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:380)
>>         at
>> org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:228)
>>         ... 70 more
>> Caused by: javax.resource.ResourceException: IJ000453: Unable to get
>> managed connection for java:jboss/datasources/KeycloakOracleDS
>>         at
>> org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:646)
>>         at
>> org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:430)
>>         at
>> org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:737)
>>         at
>> org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:138)
>>         ... 74 more
>> Caused by: javax.resource.ResourceException: IJ000655: No managed
>> connections available within configured blocking timeout (30000 [ms])
>>         at
>> org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:569)
>>         at
>> org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:627)
>>         at
>> org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:599)
>>         at
>> org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:579)
>>         ... 77 more
>>
>> 16:10:08,260 ERROR [stderr] (default task-41)
>> javax.persistence.PersistenceException:
>> org.hibernate.exception.GenericJDBCException: Could not open connection
>> 16:10:08,333 ERROR [stderr] (default task-41)   at
>> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1763)
>> 16:10:08,333 ERROR [stderr] (default task-41)   at
>> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1677)
>>
>> What's the solution to correctly handle a concurrence environment?
>> What am doing wrong? Is there a way to make synchronized keycloak jpa
>> transactions?
>> Thanks
>>
>>
>> _______________________________________________
>> keycloak-user mailing list
>> keycloak-user at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/keycloak-user
>>
>
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-user/attachments/20160308/d5376ea7/attachment-0001.html

More information about the keycloak-user mailing list