[keycloak-user] db deadlock in concurrence environment

Thomas Darimont thomas.darimont at googlemail.com
Mon Mar 7 16:37:44 EST 2016


Hello Alex,

looks like your database connection pool is exhausted - how is your
connection pool configured in wildfly?

Cheers,
Thomas

2016-03-07 22:32 GMT+01:00 alex orl <alex_orl1079 at yahoo.it>:

> Hi,
> i wrote my custom federation provider.
> All the synchronize operations (roles and user-roles) are done inside the
> getUserByUsername and the isValid method.
> This way every time a user or a new-user logins into my system, all the
> roles and role-mapping are kept up to date.
> All seems to work well with one user, but now i'm experiencing a lot of
> exceptions in a concurrence environment.
> These are the tests i'm launching:
> 1) 20 concurrent threads, each of which tries to login, to send a REST
> request (to backend) and finally to logout. All 20 threads login with the
> same test username.
> 2) 200 concurrent threads, doing the same as at the previous point, with
> 20 differents username.
>
> Everytime each federation provider instance tries to synchronize all realm
> roles...adding or removing roles in concurrence, i catch this error:
>
>
> 16:10:08,228 ERROR [io.undertow.request] (default task-96) UT005023:
> Exception handling request to
> /auth/realms/MyRealm/protocol/openid-connect/token:
> java.lang.RuntimeException: reques
> t path: /auth/realms/MyRealm/protocol/openid-connect/token
>         at
> org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:75)
>         at
> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
>         at
> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
>         at
> io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:85)
>         at
> io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
>         at
> io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
>         at
> org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
>         at
> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>         at
> io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
>         at
> io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
>         at
> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>         at
> io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
>         at
> io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
>         at
> io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58)
>         at
> io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:72)
>         at
> io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
>         at
> io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76)
>         at
> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>         at
> org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
>         at
> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>         at
> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>         at
> io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:282)
>         at
> io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:261)
>         at
> io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80)
>         at
> io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172)
>         at
> io.undertow.server.Connectors.executeRootHandler(Connectors.java:199)
>         at
> io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:774)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: org.jboss.resteasy.spi.UnhandledException:
> javax.persistence.PersistenceException:
> org.hibernate.exception.GenericJDBCException: Could not open connection
>         at
> org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:76)
>         at
> org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:212)
>         at
> org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:149)
>         at
> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:372)
>         at
> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
>         at
> org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
>         at
> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
>         at
> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>         at
> io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86)
>         at
> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:130)
>         at
> org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:61)
>         ... 29 more
> Caused by: javax.persistence.PersistenceException:
> org.hibernate.exception.GenericJDBCException: Could not open connection
>         at
> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1763)
>         at
> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1677)
>         at
> org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1771)
>         at
> org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:64)
>         at
> org.keycloak.connections.jpa.JpaKeycloakTransaction.begin(JpaKeycloakTransaction.java:22)
>         at
> org.keycloak.services.DefaultKeycloakTransactionManager.enlist(DefaultKeycloakTransactionManager.java:25)
>         at
> org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.create(DefaultJpaConnectionProviderFactory.java:46)
>         at
> org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.create(DefaultJpaConnectionProviderFactory.java:30)
>         at
> org.keycloak.services.DefaultKeycloakSession.getProvider(DefaultKeycloakSession.java:103)
>         at
> org.keycloak.models.jpa.JpaUserProviderFactory.create(JpaUserProviderFactory.java:34)
>         at
> org.keycloak.models.jpa.JpaUserProviderFactory.create(JpaUserProviderFactory.java:16)
>         at
> org.keycloak.services.DefaultKeycloakSession.getProvider(DefaultKeycloakSession.java:103)
>         at
> org.keycloak.models.cache.infinispan.DefaultCacheUserProvider.getDelegate(DefaultCacheUserProvider.java:50)
>         at
> org.keycloak.models.cache.infinispan.DefaultCacheUserProvider.getUserByUsername(DefaultCacheUserProvider.java:147)
>         at
> org.keycloak.models.UserFederationManager.getUserByUsername(UserFederationManager.java:180)
>         at
> org.keycloak.models.utils.KeycloakModelUtils.findUserByNameOrEmail(KeycloakModelUtils.java:246)
>         at
> org.keycloak.authentication.authenticators.directgrant.ValidateUsername.authenticate(ValidateUsername.java:47)
>         at
> org.keycloak.authentication.DefaultAuthenticationFlow.processFlow(DefaultAuthenticationFlow.java:155)
>         at
> org.keycloak.authentication.AuthenticationProcessor.authenticateOnly(AuthenticationProcessor.java:776)
>         at
> org.keycloak.protocol.oidc.endpoints.TokenEndpoint.buildResourceOwnerPasswordCredentialsGrant(TokenEndpoint.java:369)
>         at
> org.keycloak.protocol.oidc.endpoints.TokenEndpoint.build(TokenEndpoint.java:110)
>         at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:497)
>         at
> org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137)
>         at
> org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:296)
>         at
> org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:250)
>         at
> org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:140)
>         at
> org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:109)
>         at
> org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:135)
>         at
> org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:103)
>         at
> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356)
>         ... 37 more
> Caused by: org.hibernate.exception.GenericJDBCException: Could not open
> connection
>         at
> org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54)
>         at
> org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:126)
>         at
> org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:112)
>         at
> org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:235)
>         at
> org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:171)
>         at
> org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:67)
>         at
> org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:162)
>         at
> org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1471)
>         at
> org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:61)
>         ... 65 more
> Caused by: java.sql.SQLException: javax.resource.ResourceException:
> IJ000453: Unable to get managed connection for
> java:jboss/datasources/KeycloakOracleDS
>         at
> org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:146)
>         at
> org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:67)
>         at
> org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:139)
>         at
> org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:380)
>         at
> org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:228)
>         ... 70 more
> Caused by: javax.resource.ResourceException: IJ000453: Unable to get
> managed connection for java:jboss/datasources/KeycloakOracleDS
>         at
> org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:646)
>         at
> org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:430)
>         at
> org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:737)
>         at
> org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:138)
>         ... 74 more
> Caused by: javax.resource.ResourceException: IJ000655: No managed
> connections available within configured blocking timeout (30000 [ms])
>         at
> org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:569)
>         at
> org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:627)
>         at
> org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:599)
>         at
> org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:579)
>         ... 77 more
>
> 16:10:08,260 ERROR [stderr] (default task-41)
> javax.persistence.PersistenceException:
> org.hibernate.exception.GenericJDBCException: Could not open connection
> 16:10:08,333 ERROR [stderr] (default task-41)   at
> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1763)
> 16:10:08,333 ERROR [stderr] (default task-41)   at
> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1677)
>
> What's the solution to correctly handle a concurrence environment?
> What am doing wrong? Is there a way to make synchronized keycloak jpa
> transactions?
> Thanks
>
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-user/attachments/20160307/84beaa5c/attachment-0001.html 


More information about the keycloak-user mailing list