[keycloak-user] db deadlock in concurrence environment

Mon Mar 7 16:32:40 EST 2016

Hi,i wrote my custom federation provider.All the synchronize operations (roles and user-roles) are done inside the getUserByUsername and the isValid method.This way every time a user or a new-user logins into my system, all the roles and role-mapping are kept up to date.All seems to work well with one user, but now i'm experiencing a lot of exceptions in a concurrence environment.These are the tests i'm launching:1) 20 concurrent threads, each of which tries to login, to send a REST request (to backend) and finally to logout. All 20 threads login with the same test username.2) 200 concurrent threads, doing the same as at the previous point, with 20 differents username.
Everytime each federation provider instance tries to synchronize all realm roles...adding or removing roles in concurrence, i catch this error:

16:10:08,228 ERROR [io.undertow.request] (default task-96) UT005023: Exception handling request to /auth/realms/MyRealm/protocol/openid-connect/token: java.lang.RuntimeException: request path: /auth/realms/MyRealm/protocol/openid-connect/token        at org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:75)        at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)        at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)        at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:85)        at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)        at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)        at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)        at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)        at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)        at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)        at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)        at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58)        at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:72)        at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)        at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76)        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)        at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)        at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:282)        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:261)        at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80)        at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172)        at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199)        at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:774)        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)        at java.lang.Thread.run(Thread.java:745)Caused by: org.jboss.resteasy.spi.UnhandledException: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection        at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:76)        at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:212)        at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:149)        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:372)        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)        at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)        at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86)        at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:130)        at org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:61)        ... 29 moreCaused by: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection        at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1763)        at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1677)        at org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1771)        at org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:64)        at org.keycloak.connections.jpa.JpaKeycloakTransaction.begin(JpaKeycloakTransaction.java:22)        at org.keycloak.services.DefaultKeycloakTransactionManager.enlist(DefaultKeycloakTransactionManager.java:25)        at org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.create(DefaultJpaConnectionProviderFactory.java:46)        at org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.create(DefaultJpaConnectionProviderFactory.java:30)        at org.keycloak.services.DefaultKeycloakSession.getProvider(DefaultKeycloakSession.java:103)        at org.keycloak.models.jpa.JpaUserProviderFactory.create(JpaUserProviderFactory.java:34)        at org.keycloak.models.jpa.JpaUserProviderFactory.create(JpaUserProviderFactory.java:16)        at org.keycloak.services.DefaultKeycloakSession.getProvider(DefaultKeycloakSession.java:103)        at org.keycloak.models.cache.infinispan.DefaultCacheUserProvider.getDelegate(DefaultCacheUserProvider.java:50)        at org.keycloak.models.cache.infinispan.DefaultCacheUserProvider.getUserByUsername(DefaultCacheUserProvider.java:147)        at org.keycloak.models.UserFederationManager.getUserByUsername(UserFederationManager.java:180)        at org.keycloak.models.utils.KeycloakModelUtils.findUserByNameOrEmail(KeycloakModelUtils.java:246)        at org.keycloak.authentication.authenticators.directgrant.ValidateUsername.authenticate(ValidateUsername.java:47)        at org.keycloak.authentication.DefaultAuthenticationFlow.processFlow(DefaultAuthenticationFlow.java:155)        at org.keycloak.authentication.AuthenticationProcessor.authenticateOnly(AuthenticationProcessor.java:776)        at org.keycloak.protocol.oidc.endpoints.TokenEndpoint.buildResourceOwnerPasswordCredentialsGrant(TokenEndpoint.java:369)        at org.keycloak.protocol.oidc.endpoints.TokenEndpoint.build(TokenEndpoint.java:110)        at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source)        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)        at java.lang.reflect.Method.invoke(Method.java:497)        at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137)        at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:296)        at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:250)        at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:140)        at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:109)        at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:135)        at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:103)        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356)        ... 37 moreCaused by: org.hibernate.exception.GenericJDBCException: Could not open connection        at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54)        at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:126)        at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:112)        at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:235)        at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:171)        at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:67)        at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:162)        at org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1471)        at org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:61)        ... 65 moreCaused by: java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/KeycloakOracleDS        at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:146)        at org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:67)        at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:139)        at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:380)        at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:228)        ... 70 moreCaused by: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/KeycloakOracleDS        at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:646)        at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:430)        at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:737)        at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:138)        ... 74 moreCaused by: javax.resource.ResourceException: IJ000655: No managed connections available within configured blocking timeout (30000 [ms])        at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:569)        at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:627)        at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:599)        at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:579)        ... 77 more
16:10:08,260 ERROR [stderr] (default task-41) javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection16:10:08,333 ERROR [stderr] (default task-41)   at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1763)16:10:08,333 ERROR [stderr] (default task-41)   at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1677)
What's the solution to correctly handle a concurrence environment?What am doing wrong? Is there a way to make synchronized keycloak jpa transactions?Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-user/attachments/20160307/b05eb60a/attachment-0001.html 