Re: [keycloak-user] db deadlock in concurrence environment

Hi, i wrote my custom federation provider. All the synchronize operations (roles and user-roles) are done inside the getUserByUsername and the isValid method. This way every time a user or a new-user logins into my system, all the roles and role-mapping are kept up to date. All seems to work well with one user, but now i'm experiencing a lot of exceptions in a concurrence environment. These are the tests i'm launching: 1) 20 concurrent threads, each of which tries to login, to send a REST request (to backend) and finally to logout. All 20 threads login with the same test username. 2) 200 concurrent threads, doing the same as at the previous point, with 20 differents username. Everytime each federation provider instance tries to synchronize all realm roles...adding or removing roles in concurrence, i catch this error: 16:10:08,228 ERROR [io.undertow.request] (default task-96) UT005023: Exception handling request to /auth/realms/MyRealm/protocol/openid-connect/token: java.lang.RuntimeException: reques t path: /auth/realms/MyRealm/protocol/openid-connect/token at org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:75) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:85) at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58) at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:72) at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:282) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:261) at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80) at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:774) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.jboss.resteasy.spi.UnhandledException: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:76) at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:212) at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:149) at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:372) at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179) at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220) at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:130) at org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:61) ... 29 more Caused by: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1763) at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1677) at org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1771) at org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:64) at org.keycloak.connections.jpa.JpaKeycloakTransaction.begin(JpaKeycloakTransaction.java:22) at org.keycloak.services.DefaultKeycloakTransactionManager.enlist(DefaultKeycloakTransactionManager.java:25) at org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.create(DefaultJpaConnectionProviderFactory.java:46) at org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.create(DefaultJpaConnectionProviderFactory.java:30) at org.keycloak.services.DefaultKeycloakSession.getProvider(DefaultKeycloakSession.java:103) at org.keycloak.models.jpa.JpaUserProviderFactory.create(JpaUserProviderFactory.java:34) at org.keycloak.models.jpa.JpaUserProviderFactory.create(JpaUserProviderFactory.java:16) at org.keycloak.services.DefaultKeycloakSession.getProvider(DefaultKeycloakSession.java:103) at org.keycloak.models.cache.infinispan.DefaultCacheUserProvider.getDelegate(DefaultCacheUserProvider.java:50) at org.keycloak.models.cache.infinispan.DefaultCacheUserProvider.getUserByUsername(DefaultCacheUserProvider.java:147) at org.keycloak.models.UserFederationManager.getUserByUsername(UserFederationManager.java:180) at org.keycloak.models.utils.KeycloakModelUtils.findUserByNameOrEmail(KeycloakModelUtils.java:246) at org.keycloak.authentication.authenticators.directgrant.ValidateUsername.authenticate(ValidateUsername.java:47) at org.keycloak.authentication.DefaultAuthenticationFlow.processFlow(DefaultAuthenticationFlow.java:155) at org.keycloak.authentication.AuthenticationProcessor.authenticateOnly(AuthenticationProcessor.java:776) at org.keycloak.protocol.oidc.endpoints.TokenEndpoint.buildResourceOwnerPasswordCredentialsGrant(TokenEndpoint.java:369) at org.keycloak.protocol.oidc.endpoints.TokenEndpoint.build(TokenEndpoint.java:110) at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137) at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:296) at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:250) at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:140) at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:109) at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:135) at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:103) at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356) ... 37 more Caused by: org.hibernate.exception.GenericJDBCException: Could not open connection at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:126) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:112) at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:235) at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:171) at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:67) at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:162) at org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1471) at org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:61) ... 65 more Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/KeycloakOracleDS at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:146) at org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:67) at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:139) at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:380) at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:228) ... 70 more Caused by: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/KeycloakOracleDS at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:646) at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:430) at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:737) at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:138) ... 74 more Caused by: javax.resource.ResourceException: IJ000655: No managed connections available within configured blocking timeout (30000 [ms]) at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:569) at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:627) at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:599) at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:579) ... 77 more 16:10:08,260 ERROR [stderr] (default task-41) javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection 16:10:08,333 ERROR [stderr] (default task-41) at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1763) 16:10:08,333 ERROR [stderr] (default task-41) at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1677) What's the solution to correctly handle a concurrence environment? What am doing wrong? Is there a way to make synchronized keycloak jpa transactions? Thanks _______________________________________________ keycloak-user mailing list keycloak-user(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-user