9:59 a.m.
Hi ,
I am using keycloak 4.5.0.Final in one on my projects and i have fairly large number of
tenants (> 500).Off late i see frequently lock acquisation related errors and
timeout.
I am not able to figure out where and how this is origniating? can you please help?
My suspecion is is this related to events logging? could this be because of farily large
number of entrys in the audit/ events table?
Note the thread id default task-19354 in the for event REFRESH_TOKEN_ERROR and
corresponding thread throwing LockAcquisitionException
Regards,Madhu
2019-03-17 17:14:47,010 WARN [org.keycloak.events] (default task-19354)
type=REFRESH_TOKEN_ERROR, realmId=*********, clientId=null, userId=null, ipAddress=
xx.yy.zz.aaa, error=invalid_client_credentials, grant_type=refresh_token2019-03-17
17:15:13,183 WARN [org.keycloak.events] (default task-19354) type=REFRESH_TOKEN_ERROR,
realmId=*********, clientId=null, userId=null, ipAddress=xx.yy.zz.aaa,
error=invalid_client_credentials, grant_type=refresh_token2019-03-17 17:24:31,128 WARN
[org.keycloak.events] (default task-19353) type=REFRESH_TOKEN_ERROR, realmId=*********,
clientId=null, userId=null, ipAddress=xx.yy.zz.aaa, error=invalid_client_credentials,
grant_type=refresh_token2019-03-17 17:46:17,677 WARN [org.keycloak.events] (default
task-19353) type=REFRESH_TOKEN_ERROR, realmId=*********, clientId=null, userId=null,
ipAddress=xx.yy.zz.aaa, error=invalid_client_credentials,
grant_type=refresh_token2019-03-17 17:47:00,850 WARN [org.keycloak.events] (default
task-19353) type=REFRESH_TOKEN_ERROR, realmId=*********, clientId=null, userId=null,
ipAddress=xx.yy.zz.aaa, error=invalid_client_credentials,
grant_type=refresh_token2019-03-17 18:46:48,058 WARN
[org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (default task-19354) SQL Error: 1205,
SQLState: 400012019-03-17 18:46:48,059 ERROR
[org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (default task-19354) Lock wait timeout
exceeded; try restarting transaction2019-03-17 18:46:48,059 INFO
[org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl] (default task-19354)
HHH000010: On release of batch it still contained JDBC statements2019-03-17 18:46:48,077
WARN [com.arjuna.ats.arjuna] (default task-19354) ARJUNA012125:
TwoPhaseCoordinator.beforeCompletion - failed for SynchronizationImple<
0:ffffc0a803b1:-a9285f2:5bf97526:bb36de,
org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization@5ad70e16
>: javax.persistence.PersistenceException:
org.hibernate.exception.LockAcquisitionException: could not execute statement at
org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
at
org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
at
org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1608)
at
org.hibernate.jpa.internal.EntityManagerImpl$CallbackExceptionMapperImpl.mapManagedFlushFailure(EntityManagerImpl.java:235)
at
org.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3163)
at org.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:2352)
at
org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion(JdbcCoordinatorImpl.java:491)
at
org.hibernate.resource.transaction.backend.jta.internal.JtaTransactionCoordinatorImpl.beforeCompletion(JtaTransactionCoordinatorImpl.java:316)
at
org.hibernate.resource.transaction.backend.jta.internal.synchronization.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:47)
at
org.hibernate.resource.transaction.backend.jta.internal.synchronization.RegisteredSynchronization.beforeCompletion(RegisteredSynchronization.java:37)
at
org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:236)
at
org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:247)
at
org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:292)
at
com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
at
com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162) at
com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
at
com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at
com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at
org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:77)
at
org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
at org.keycloak.transaction.JtaTransactionWrapper.commit(JtaTransactionWrapper.java:92) at
org.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:136)
at
org.keycloak.services.filters.KeycloakTransactionCommitter.filter(KeycloakTransactionCommitter.java:43)
at
org.jboss.resteasy.core.interception.ContainerResponseContextImpl.filter(ContainerResponseContextImpl.java:353)
at
org.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:207)
at
org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:85)
at
org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:59)
at
org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:530)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:461) at
org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:231)
at
org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:137)
at
org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:361)
at
org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:140)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:217) at
org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at
io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) at
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at
org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:90)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) at
io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) at
io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at
org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at
io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
at
io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at
io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at
io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at
io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at
io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at
io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
at
io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at
org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at
org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at
io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
at
io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
at
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
at
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
at
io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at
io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at
org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
at
io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
at
io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at
io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) at
io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) at
org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
at java.lang.Thread.run(Thread.java:748)Caused by:
org.hibernate.exception.LockAcquisitionException: could not execute statement at
org.hibernate.dialect.MySQLDialect$3.convert(MySQLDialect.java:511) at
org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97) at
org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:207)
at
org.hibernate.engine.jdbc.batch.internal.NonBatchingBatch.addToBatch(NonBatchingBatch.java:45)
at
org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3013)
at
org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3513)
at org.hibernate.action.internal.EntityInsertAction.execute(EntityInsertAction.java:89) at
org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:589) at
org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:463) at
org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:337)
at
org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:39)
at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1295) at
org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:468) at
org.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3159)
... 81 moreCaused by: com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Lock
wait timeout exceeded; try restarting transaction at
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:121) at
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:95) at
com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:960)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1116)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1066)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1396)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:1051)
at
org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:537)
at
org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:204)
... 92 more
2019-03-17 18:46:48,558 ERROR [org.keycloak.services.error.KeycloakErrorHandler] (default
task-19354) Uncaught server error: org.keycloak.models.ModelException:
org.hibernate.exception.LockAcquisitionException: could not execute statement at
org.keycloak.connections.jpa.PersistenceExceptionConverter.convert(PersistenceExceptionConverter.java:61)
at
org.keycloak.connections.jpa.JpaExceptionConverter.convert(JpaExceptionConverter.java:31)
at
org.keycloak.transaction.JtaTransactionWrapper.handleException(JtaTransactionWrapper.java:65)
at org.keycloak.transaction.JtaTransactionWrapper.commit(JtaTransactionWrapper.java:94) at
org.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:136)
at
org.keycloak.services.filters.KeycloakTransactionCommitter.filter(KeycloakTransactionCommitter.java:43)
at
org.jboss.resteasy.core.interception.ContainerResponseContextImpl.filter(ContainerResponseContextImpl.java:353)
at
org.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:207)
at
org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:85)
at
org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:59)
at
org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:530)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:461) at
org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:231)
at
org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:137)
at
org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:361)
at
org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:140)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:217) at
org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at
io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) at
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at
org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:90)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) at
io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) at
io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at
org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at
io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
at
io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at
io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at
io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at
io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at
io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at
io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
at
io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at
org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at
org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at
io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
at
io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
at
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
at
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
at
io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at
io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at
org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
at
io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
at
io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at
io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) at
io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) at
org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
at java.lang.Thread.run(Thread.java:748)Caused by:
org.hibernate.exception.LockAcquisitionException: could not execute statement at
org.hibernate.dialect.MySQLDialect$3.convert(MySQLDialect.java:511) at
org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97) at
org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:207)
at
org.hibernate.engine.jdbc.batch.internal.NonBatchingBatch.addToBatch(NonBatchingBatch.java:45)
at
org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3013)
at
org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3513)
at org.hibernate.action.internal.EntityInsertAction.execute(EntityInsertAction.java:89) at
org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:589) at
org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:463) at
org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:337)
at
org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:39)
at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1295) at
org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:468) at
org.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3159)
at org.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:2352)
at
org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion(JdbcCoordinatorImpl.java:491)
at
org.hibernate.resource.transaction.backend.jta.internal.JtaTransactionCoordinatorImpl.beforeCompletion(JtaTransactionCoordinatorImpl.java:316)
at
org.hibernate.resource.transaction.backend.jta.internal.synchronization.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:47)
at
org.hibernate.resource.transaction.backend.jta.internal.synchronization.RegisteredSynchronization.beforeCompletion(RegisteredSynchronization.java:37)
at
org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:236)
at
org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:247)
at
org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:292)
at
com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
at
com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162) at
com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
at
com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at
com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at
org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:77)
at
org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
at org.keycloak.transaction.JtaTransactionWrapper.commit(JtaTransactionWrapper.java:92)
... 63 moreCaused by: com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Lock
wait timeout exceeded; try restarting transaction at
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:121) at
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:95) at
com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:960)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1116)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1066)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1396)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:1051)
at
org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:537)
at
org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:204)
... 92 more
3:51 p.m.
Hi,
what exactly is "tenant" in your case? Is it client or realm?
We know that there are some issues with big number of those entities, so
you will probably see issues with 500 or more realms/clients. Maybe it
helps to increase count of max DB connections - both at the datasource
level in standalone(-ha).xml and in the settings of your MySQL DB. But
not really sure...
Marek
On 18/03/2019 15:59, Madhu wrote:
Hi ,
I am using keycloak 4.5.0.Final in one on my projects and i have
fairly large number of tenants (> 500).
Off late i see frequently lock acquisation related errors and timeout.
I am not able to figure out where and how this is origniating? can you
please help?
My suspecion is is this related to events logging? could this be
because of farily large number of entrys in the audit/ events table?
Note the thread id default task-19354 in the for event
REFRESH_TOKEN_ERROR and corresponding thread throwing
LockAcquisitionException
Regards,
Madhu
2019-03-17 17:14:47,010 WARN [org.keycloak.events] (default
task-19354) type=REFRESH_TOKEN_ERROR, realmId=*********,
clientId=null, userId=null, ipAddress= xx.yy.zz.aaa,
error=invalid_client_credentials, grant_type=refresh_token
2019-03-17 17:15:13,183 WARN [org.keycloak.events] (default
task-19354) type=REFRESH_TOKEN_ERROR, realmId=*********,
clientId=null, userId=null, ipAddress=xx.yy.zz.aaa,
error=invalid_client_credentials, grant_type=refresh_token
2019-03-17 17:24:31,128 WARN [org.keycloak.events] (default
task-19353) type=REFRESH_TOKEN_ERROR, realmId=*********,
clientId=null, userId=null, ipAddress=xx.yy.zz.aaa,
error=invalid_client_credentials, grant_type=refresh_token
2019-03-17 17:46:17,677 WARN [org.keycloak.events] (default
task-19353) type=REFRESH_TOKEN_ERROR, realmId=*********,
clientId=null, userId=null, ipAddress=xx.yy.zz.aaa,
error=invalid_client_credentials, grant_type=refresh_token
2019-03-17 17:47:00,850 WARN [org.keycloak.events] (default
task-19353) type=REFRESH_TOKEN_ERROR, realmId=*********,
clientId=null, userId=null, ipAddress=xx.yy.zz.aaa,
error=invalid_client_credentials, grant_type=refresh_token
2019-03-17 18:46:48,058 WARN
[org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (default
task-19354) SQL Error: 1205, SQLState: 40001
2019-03-17 18:46:48,059 ERROR
[org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (default
task-19354) Lock wait timeout exceeded; try restarting transaction
2019-03-17 18:46:48,059 INFO
[org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl] (default
task-19354) HHH000010: On release of batch it still contained JDBC
statements
2019-03-17 18:46:48,077 WARN [com.arjuna.ats.arjuna] (default
task-19354) ARJUNA012125: TwoPhaseCoordinator.beforeCompletion -
failed for SynchronizationImple<
0:ffffc0a803b1:-a9285f2:5bf97526:bb36de,
org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization@5ad70e16
>: javax.persistence.PersistenceException:
org.hibernate.exception.LockAcquisitionException: could not execute
statement
at
org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
at
org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
at
org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1608)
at
org.hibernate.jpa.internal.EntityManagerImpl$CallbackExceptionMapperImpl.mapManagedFlushFailure(EntityManagerImpl.java:235)
at
org.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3163)
at
org.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:2352)
at
org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion(JdbcCoordinatorImpl.java:491)
at
org.hibernate.resource.transaction.backend.jta.internal.JtaTransactionCoordinatorImpl.beforeCompletion(JtaTransactionCoordinatorImpl.java:316)
at
org.hibernate.resource.transaction.backend.jta.internal.synchronization.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:47)
at
org.hibernate.resource.transaction.backend.jta.internal.synchronization.RegisteredSynchronization.beforeCompletion(RegisteredSynchronization.java:37)
at
org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:236)
at
org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:247)
at
org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:292)
at
com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
at
com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
at
com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at
com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
at
com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at
com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at
org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:77)
at
org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
at
org.keycloak.transaction.JtaTransactionWrapper.commit(JtaTransactionWrapper.java:92)
at
org.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:136)
at
org.keycloak.services.filters.KeycloakTransactionCommitter.filter(KeycloakTransactionCommitter.java:43)
at
org.jboss.resteasy.core.interception.ContainerResponseContextImpl.filter(ContainerResponseContextImpl.java:353)
at
org.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:207)
at
org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:85)
at
org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:59)
at
org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:530)
at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:461)
at
org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:231)
at
org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:137)
at
org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:361)
at
org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:140)
at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:217)
at
org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at
io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
at
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at
org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:90)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at
io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at
io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at
io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
at
io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at
org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
at
io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at
io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at
io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at
io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at
io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
at
io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
at
io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
at
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
at
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
at
io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at
io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at
org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
at
io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
at
io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at
io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
at
io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
at
org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at
org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.hibernate.exception.LockAcquisitionException: could not
execute statement
at org.hibernate.dialect.MySQLDialect$3.convert(MySQLDialect.java:511)
at
org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
at
org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)
at
org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97)
at
org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:207)
at
org.hibernate.engine.jdbc.batch.internal.NonBatchingBatch.addToBatch(NonBatchingBatch.java:45)
at
org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3013)
at
org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3513)
at
org.hibernate.action.internal.EntityInsertAction.execute(EntityInsertAction.java:89)
at
org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:589)
at
org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:463)
at
org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:337)
at
org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:39)
at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1295)
at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:468)
at
org.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3159)
... 81 more
Caused by:
com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Lock
wait timeout exceeded; try restarting transaction
at
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:121)
at
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:95)
at
com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:960)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1116)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1066)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1396)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:1051)
at
org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:537)
at
org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:204)
... 92 more
2019-03-17 18:46:48,558 ERROR
[org.keycloak.services.error.KeycloakErrorHandler] (default
task-19354) Uncaught server error: org.keycloak.models.ModelException:
org.hibernate.exception.LockAcquisitionException: could not execute
statement
at
org.keycloak.connections.jpa.PersistenceExceptionConverter.convert(PersistenceExceptionConverter.java:61)
at
org.keycloak.connections.jpa.JpaExceptionConverter.convert(JpaExceptionConverter.java:31)
at
org.keycloak.transaction.JtaTransactionWrapper.handleException(JtaTransactionWrapper.java:65)
at
org.keycloak.transaction.JtaTransactionWrapper.commit(JtaTransactionWrapper.java:94)
at
org.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:136)
at
org.keycloak.services.filters.KeycloakTransactionCommitter.filter(KeycloakTransactionCommitter.java:43)
at
org.jboss.resteasy.core.interception.ContainerResponseContextImpl.filter(ContainerResponseContextImpl.java:353)
at
org.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:207)
at
org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:85)
at
org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:59)
at
org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:530)
at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:461)
at
org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:231)
at
org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:137)
at
org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:361)
at
org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:140)
at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:217)
at
org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at
io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
at
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at
org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:90)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at
io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at
io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at
io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
at
io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at
org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
at
io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at
io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at
io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at
io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at
io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
at
io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
at
io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
at
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
at
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
at
io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at
io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at
org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
at
io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
at
io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at
io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
at
io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
at
org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at
org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.hibernate.exception.LockAcquisitionException: could not
execute statement
at org.hibernate.dialect.MySQLDialect$3.convert(MySQLDialect.java:511)
at
org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
at
org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)
at
org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97)
at
org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:207)
at
org.hibernate.engine.jdbc.batch.internal.NonBatchingBatch.addToBatch(NonBatchingBatch.java:45)
at
org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3013)
at
org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3513)
at
org.hibernate.action.internal.EntityInsertAction.execute(EntityInsertAction.java:89)
at
org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:589)
at
org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:463)
at
org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:337)
at
org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:39)
at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1295)
at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:468)
at
org.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3159)
at
org.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:2352)
at
org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion(JdbcCoordinatorImpl.java:491)
at
org.hibernate.resource.transaction.backend.jta.internal.JtaTransactionCoordinatorImpl.beforeCompletion(JtaTransactionCoordinatorImpl.java:316)
at
org.hibernate.resource.transaction.backend.jta.internal.synchronization.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:47)
at
org.hibernate.resource.transaction.backend.jta.internal.synchronization.RegisteredSynchronization.beforeCompletion(RegisteredSynchronization.java:37)
at
org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:236)
at
org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:247)
at
org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:292)
at
com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
at
com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
at
com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at
com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
at
com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at
com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at
org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:77)
at
org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
at
org.keycloak.transaction.JtaTransactionWrapper.commit(JtaTransactionWrapper.java:92)
... 63 more
Caused by:
com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Lock
wait timeout exceeded; try restarting transaction
at
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:121)
at
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:95)
at
com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:960)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1116)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1066)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1396)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:1051)
at
org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:537)
at
org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:204)
... 92 more
4:21 p.m.
Realm is tenant in my case. I have more than 600 realms and each realm has about 6 clients
(excluding what gets shipped by default). There are 2 realm roles and 2 to 3 client roles
per client, I have a scriptmapper and 2 groups and about 10 users in each realm.My Max db
connection is 30. Let me check that again try increasing it..
Madhu
Sent from Yahoo Mail on Android
On Tue, 19 Mar 2019 at 2:21 AM, Marek Posolda<mposolda(a)redhat.com> wrote: Hi,
what exactly is "tenant" in your case? Is it client or realm?
We know that there are some issues with big number of those entities, so you will
probably see issues with 500 or more realms/clients. Maybe it helps to increase count of
max DB connections - both at the datasource level in standalone(-ha).xml and in the
settings of your MySQL DB. But not really sure...
Marek
On 18/03/2019 15:59, Madhu wrote:
Hi ,
I am using keycloak 4.5.0.Final in one on my projects and i have fairly large number of
tenants (> 500). Off late i see frequently lock acquisation related errors and
timeout.
I am not able to figure out where and how this is origniating? can you please help?
My suspecion is is this related to events logging? could this be because of farily
large number of entrys in the audit/ events table?
Note the thread id default task-19354 in the for event REFRESH_TOKEN_ERROR and
corresponding thread throwing LockAcquisitionException
Regards, Madhu
2019-03-17 17:14:47,010 WARN [org.keycloak.events] (default task-19354)
type=REFRESH_TOKEN_ERROR, realmId=*********, clientId=null, userId=null, ipAddress=
xx.yy.zz.aaa, error=invalid_client_credentials, grant_type=refresh_token 2019-03-17
17:15:13,183 WARN [org.keycloak.events] (default task-19354) type=REFRESH_TOKEN_ERROR,
realmId=*********, clientId=null, userId=null, ipAddress=xx.yy.zz.aaa,
error=invalid_client_credentials, grant_type=refresh_token 2019-03-17 17:24:31,128 WARN
[org.keycloak.events] (default task-19353) type=REFRESH_TOKEN_ERROR, realmId=*********,
clientId=null, userId=null, ipAddress=xx.yy.zz.aaa, error=invalid_client_credentials,
grant_type=refresh_token 2019-03-17 17:46:17,677 WARN [org.keycloak.events] (default
task-19353) type=REFRESH_TOKEN_ERROR, realmId=*********, clientId=null, userId=null,
ipAddress=xx.yy.zz.aaa, error=invalid_client_credentials, grant_type=refresh_token
2019-03-17 17:47:00,850 WARN [org.keycloak.events] (default task-19353)
type=REFRESH_TOKEN_ERROR, realmId=*********, clientId=null, userId=null,
ipAddress=xx.yy.zz.aaa, error=invalid_client_credentials, grant_type=refresh_token
2019-03-17 18:46:48,058 WARN [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (default
task-19354) SQL Error: 1205, SQLState: 40001 2019-03-17 18:46:48,059 ERROR
[org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (default task-19354) Lock wait timeout
exceeded; try restarting transaction 2019-03-17 18:46:48,059 INFO
[org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl] (default task-19354)
HHH000010: On release of batch it still contained JDBC statements 2019-03-17 18:46:48,077
WARN [com.arjuna.ats.arjuna] (default task-19354) ARJUNA012125:
TwoPhaseCoordinator.beforeCompletion - failed for SynchronizationImple<
0:ffffc0a803b1:-a9285f2:5bf97526:bb36de,org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization@5ad70e16
>: javax.persistence.PersistenceException:
org.hibernate.exception.LockAcquisitionException: could not execute statement
atorg.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
atorg.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
atorg.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1608)
atorg.hibernate.jpa.internal.EntityManagerImpl$CallbackExceptionMapperImpl.mapManagedFlushFailure(EntityManagerImpl.java:235)
atorg.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3163)
atorg.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:2352)
atorg.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion(JdbcCoordinatorImpl.java:491)
atorg.hibernate.resource.transaction.backend.jta.internal.JtaTransactionCoordinatorImpl.beforeCompletion(JtaTransactionCoordinatorImpl.java:316)
atorg.hibernate.resource.transaction.backend.jta.internal.synchronization.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:47)
atorg.hibernate.resource.transaction.backend.jta.internal.synchronization.RegisteredSynchronization.beforeCompletion(RegisteredSynchronization.java:37)
atorg.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:236)
atorg.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:247)
atorg.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:292)
atcom.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
atcom.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
atcom.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
atcom.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
atcom.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
atcom.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
atorg.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:77)
atorg.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
atorg.keycloak.transaction.JtaTransactionWrapper.commit(JtaTransactionWrapper.java:92)
atorg.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:136)
atorg.keycloak.services.filters.KeycloakTransactionCommitter.filter(KeycloakTransactionCommitter.java:43)
atorg.jboss.resteasy.core.interception.ContainerResponseContextImpl.filter(ContainerResponseContextImpl.java:353)
atorg.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:207)
atorg.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:85)
atorg.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:59)
atorg.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:530)
atorg.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:461)
atorg.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:231)
atorg.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:137)
atorg.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:361)
atorg.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:140)
atorg.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:217)
atorg.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227)
atorg.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
atorg.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
atio.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
atio.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
atorg.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:90)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
atio.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
atio.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
atio.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
atio.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
atio.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
atorg.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atio.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
atio.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atio.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
atio.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
atio.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
atio.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
atio.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
atio.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atorg.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atorg.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atio.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
atio.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
atio.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
atio.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
atio.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
atio.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
atorg.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
atorg.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
atorg.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
atorg.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
atorg.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
atio.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
atio.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
atio.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) at
io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
atorg.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
atorg.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
atorg.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
atorg.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
at java.lang.Thread.run(Thread.java:748) Caused by:
org.hibernate.exception.LockAcquisitionException: could not execute statement at
org.hibernate.dialect.MySQLDialect$3.convert(MySQLDialect.java:511)
atorg.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
atorg.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)
atorg.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97)
atorg.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:207)
atorg.hibernate.engine.jdbc.batch.internal.NonBatchingBatch.addToBatch(NonBatchingBatch.java:45)
atorg.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3013)
atorg.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3513)
atorg.hibernate.action.internal.EntityInsertAction.execute(EntityInsertAction.java:89)
atorg.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:589)
atorg.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:463)
atorg.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:337)
atorg.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:39)
at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1295) at
org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:468)
atorg.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3159)
... 81 more Caused by: com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException:
Lock wait timeout exceeded; try restarting transaction
atcom.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:121)
atcom.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:95)
atcom.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:960)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1116)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1066)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1396)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:1051)
atorg.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:537)
atorg.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:204)
... 92 more
2019-03-17 18:46:48,558 ERROR [org.keycloak.services.error.KeycloakErrorHandler]
(default task-19354) Uncaught server error: org.keycloak.models.ModelException:
org.hibernate.exception.LockAcquisitionException: could not execute statement
atorg.keycloak.connections.jpa.PersistenceExceptionConverter.convert(PersistenceExceptionConverter.java:61)
atorg.keycloak.connections.jpa.JpaExceptionConverter.convert(JpaExceptionConverter.java:31)
atorg.keycloak.transaction.JtaTransactionWrapper.handleException(JtaTransactionWrapper.java:65)
atorg.keycloak.transaction.JtaTransactionWrapper.commit(JtaTransactionWrapper.java:94)
atorg.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:136)
atorg.keycloak.services.filters.KeycloakTransactionCommitter.filter(KeycloakTransactionCommitter.java:43)
atorg.jboss.resteasy.core.interception.ContainerResponseContextImpl.filter(ContainerResponseContextImpl.java:353)
atorg.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:207)
atorg.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:85)
atorg.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:59)
atorg.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:530)
atorg.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:461)
atorg.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:231)
atorg.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:137)
atorg.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:361)
atorg.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:140)
atorg.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:217)
atorg.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227)
atorg.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
atorg.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
atio.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
atio.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
atorg.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:90)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
atio.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
atio.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
atio.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
atio.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
atio.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
atorg.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atio.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
atio.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atio.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
atio.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
atio.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
atio.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
atio.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
atio.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atorg.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atorg.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atio.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
atio.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
atio.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
atio.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
atio.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
atio.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
atorg.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
atorg.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
atorg.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
atorg.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
atorg.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
atio.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
atio.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
atio.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) at
io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
atorg.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
atorg.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
atorg.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
atorg.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
at java.lang.Thread.run(Thread.java:748) Caused by:
org.hibernate.exception.LockAcquisitionException: could not execute statement at
org.hibernate.dialect.MySQLDialect$3.convert(MySQLDialect.java:511)
atorg.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
atorg.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)
atorg.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97)
atorg.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:207)
atorg.hibernate.engine.jdbc.batch.internal.NonBatchingBatch.addToBatch(NonBatchingBatch.java:45)
atorg.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3013)
atorg.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3513)
atorg.hibernate.action.internal.EntityInsertAction.execute(EntityInsertAction.java:89)
atorg.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:589)
atorg.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:463)
atorg.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:337)
atorg.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:39)
at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1295) at
org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:468)
atorg.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3159)
atorg.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:2352)
atorg.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion(JdbcCoordinatorImpl.java:491)
atorg.hibernate.resource.transaction.backend.jta.internal.JtaTransactionCoordinatorImpl.beforeCompletion(JtaTransactionCoordinatorImpl.java:316)
atorg.hibernate.resource.transaction.backend.jta.internal.synchronization.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:47)
atorg.hibernate.resource.transaction.backend.jta.internal.synchronization.RegisteredSynchronization.beforeCompletion(RegisteredSynchronization.java:37)
atorg.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:236)
atorg.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:247)
atorg.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:292)
atcom.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
atcom.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
atcom.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
atcom.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
atcom.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
atcom.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
atorg.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:77)
atorg.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
atorg.keycloak.transaction.JtaTransactionWrapper.commit(JtaTransactionWrapper.java:92)
... 63 more Caused by: com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException:
Lock wait timeout exceeded; try restarting transaction
atcom.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:121)
atcom.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:95)
atcom.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:960)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1116)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1066)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1396)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:1051)
atorg.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:537)
atorg.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:204)
... 92 more
12:23 a.m.
Hi Marek,
Thanks for quick response.
I double checked my connection pool settings, this is what i have configuredMin Pool
Size:50Max Pool Size:100Flush Strategy:IdleConnectionsPool Fair:truePool Prefill:falsePool
Use Strict Min:falseUse Fast Fail:false
Do you think i need to change this, i donth think the system is starving for
connections.. will dig more deep here and get back.
Madhu On Tuesday, 19 March, 2019, 2:51:39 am IST, Madhu <kkcmadhu(a)yahoo.com>
wrote:
Realm is tenant in my case. I have more than 600 realms and each realm has about 6
clients (excluding what gets shipped by default). There are 2 realm roles and 2 to 3
client roles per client, I have a scriptmapper and 2 groups and about 10 users in each
realm.My Max db connection is 30. Let me check that again try increasing it..
Madhu
Sent from Yahoo Mail on Android
On Tue, 19 Mar 2019 at 2:21 AM, Marek Posolda<mposolda(a)redhat.com> wrote: Hi,
what exactly is "tenant" in your case? Is it client or realm?
We know that there are some issues with big number of those entities, so you will
probably see issues with 500 or more realms/clients. Maybe it helps to increase count of
max DB connections - both at the datasource level in standalone(-ha).xml and in the
settings of your MySQL DB. But not really sure...
Marek
On 18/03/2019 15:59, Madhu wrote:
Hi ,
I am using keycloak 4.5.0.Final in one on my projects and i have fairly large number of
tenants (> 500). Off late i see frequently lock acquisation related errors and
timeout.
I am not able to figure out where and how this is origniating? can you please help?
My suspecion is is this related to events logging? could this be because of farily
large number of entrys in the audit/ events table?
Note the thread id default task-19354 in the for event REFRESH_TOKEN_ERROR and
corresponding thread throwing LockAcquisitionException
Regards, Madhu
2019-03-17 17:14:47,010 WARN [org.keycloak.events] (default task-19354)
type=REFRESH_TOKEN_ERROR, realmId=*********, clientId=null, userId=null, ipAddress=
xx.yy.zz.aaa, error=invalid_client_credentials, grant_type=refresh_token 2019-03-17
17:15:13,183 WARN [org.keycloak.events] (default task-19354) type=REFRESH_TOKEN_ERROR,
realmId=*********, clientId=null, userId=null, ipAddress=xx.yy.zz.aaa,
error=invalid_client_credentials, grant_type=refresh_token 2019-03-17 17:24:31,128 WARN
[org.keycloak.events] (default task-19353) type=REFRESH_TOKEN_ERROR, realmId=*********,
clientId=null, userId=null, ipAddress=xx.yy.zz.aaa, error=invalid_client_credentials,
grant_type=refresh_token 2019-03-17 17:46:17,677 WARN [org.keycloak.events] (default
task-19353) type=REFRESH_TOKEN_ERROR, realmId=*********, clientId=null, userId=null,
ipAddress=xx.yy.zz.aaa, error=invalid_client_credentials, grant_type=refresh_token
2019-03-17 17:47:00,850 WARN [org.keycloak.events] (default task-19353)
type=REFRESH_TOKEN_ERROR, realmId=*********, clientId=null, userId=null,
ipAddress=xx.yy.zz.aaa, error=invalid_client_credentials, grant_type=refresh_token
2019-03-17 18:46:48,058 WARN [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (default
task-19354) SQL Error: 1205, SQLState: 40001 2019-03-17 18:46:48,059 ERROR
[org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (default task-19354) Lock wait timeout
exceeded; try restarting transaction 2019-03-17 18:46:48,059 INFO
[org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl] (default task-19354)
HHH000010: On release of batch it still contained JDBC statements 2019-03-17 18:46:48,077
WARN [com.arjuna.ats.arjuna] (default task-19354) ARJUNA012125:
TwoPhaseCoordinator.beforeCompletion - failed for SynchronizationImple<
0:ffffc0a803b1:-a9285f2:5bf97526:bb36de,org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization@5ad70e16
>: javax.persistence.PersistenceException:
org.hibernate.exception.LockAcquisitionException: could not execute statement
atorg.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
atorg.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
atorg.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1608)
atorg.hibernate.jpa.internal.EntityManagerImpl$CallbackExceptionMapperImpl.mapManagedFlushFailure(EntityManagerImpl.java:235)
atorg.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3163)
atorg.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:2352)
atorg.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion(JdbcCoordinatorImpl.java:491)
atorg.hibernate.resource.transaction.backend.jta.internal.JtaTransactionCoordinatorImpl.beforeCompletion(JtaTransactionCoordinatorImpl.java:316)
atorg.hibernate.resource.transaction.backend.jta.internal.synchronization.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:47)
atorg.hibernate.resource.transaction.backend.jta.internal.synchronization.RegisteredSynchronization.beforeCompletion(RegisteredSynchronization.java:37)
atorg.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:236)
atorg.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:247)
atorg.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:292)
atcom.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
atcom.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
atcom.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
atcom.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
atcom.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
atcom.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
atorg.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:77)
atorg.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
atorg.keycloak.transaction.JtaTransactionWrapper.commit(JtaTransactionWrapper.java:92)
atorg.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:136)
atorg.keycloak.services.filters.KeycloakTransactionCommitter.filter(KeycloakTransactionCommitter.java:43)
atorg.jboss.resteasy.core.interception.ContainerResponseContextImpl.filter(ContainerResponseContextImpl.java:353)
atorg.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:207)
atorg.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:85)
atorg.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:59)
atorg.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:530)
atorg.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:461)
atorg.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:231)
atorg.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:137)
atorg.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:361)
atorg.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:140)
atorg.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:217)
atorg.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227)
atorg.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
atorg.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
atio.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
atio.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
atorg.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:90)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
atio.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
atio.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
atio.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
atio.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
atio.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
atorg.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atio.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
atio.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atio.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
atio.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
atio.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
atio.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
atio.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
atio.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atorg.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atorg.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atio.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
atio.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
atio.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
atio.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
atio.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
atio.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
atorg.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
atorg.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
atorg.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
atorg.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
atorg.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
atio.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
atio.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
atio.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) at
io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
atorg.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
atorg.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
atorg.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
atorg.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
at java.lang.Thread.run(Thread.java:748) Caused by:
org.hibernate.exception.LockAcquisitionException: could not execute statement at
org.hibernate.dialect.MySQLDialect$3.convert(MySQLDialect.java:511)
atorg.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
atorg.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)
atorg.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97)
atorg.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:207)
atorg.hibernate.engine.jdbc.batch.internal.NonBatchingBatch.addToBatch(NonBatchingBatch.java:45)
atorg.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3013)
atorg.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3513)
atorg.hibernate.action.internal.EntityInsertAction.execute(EntityInsertAction.java:89)
atorg.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:589)
atorg.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:463)
atorg.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:337)
atorg.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:39)
at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1295) at
org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:468)
atorg.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3159)
... 81 more Caused by: com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException:
Lock wait timeout exceeded; try restarting transaction
atcom.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:121)
atcom.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:95)
atcom.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:960)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1116)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1066)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1396)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:1051)
atorg.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:537)
atorg.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:204)
... 92 more
2019-03-17 18:46:48,558 ERROR [org.keycloak.services.error.KeycloakErrorHandler]
(default task-19354) Uncaught server error: org.keycloak.models.ModelException:
org.hibernate.exception.LockAcquisitionException: could not execute statement
atorg.keycloak.connections.jpa.PersistenceExceptionConverter.convert(PersistenceExceptionConverter.java:61)
atorg.keycloak.connections.jpa.JpaExceptionConverter.convert(JpaExceptionConverter.java:31)
atorg.keycloak.transaction.JtaTransactionWrapper.handleException(JtaTransactionWrapper.java:65)
atorg.keycloak.transaction.JtaTransactionWrapper.commit(JtaTransactionWrapper.java:94)
atorg.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:136)
atorg.keycloak.services.filters.KeycloakTransactionCommitter.filter(KeycloakTransactionCommitter.java:43)
atorg.jboss.resteasy.core.interception.ContainerResponseContextImpl.filter(ContainerResponseContextImpl.java:353)
atorg.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:207)
atorg.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:85)
atorg.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:59)
atorg.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:530)
atorg.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:461)
atorg.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:231)
atorg.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:137)
atorg.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:361)
atorg.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:140)
atorg.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:217)
atorg.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227)
atorg.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
atorg.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
atio.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
atio.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
atorg.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:90)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
atio.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
atio.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
atio.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
atio.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
atio.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
atorg.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atio.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
atio.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atio.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
atio.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
atio.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
atio.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
atio.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
atio.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atorg.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atorg.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atio.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
atio.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
atio.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
atio.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
atio.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
atio.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
atorg.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
atorg.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
atorg.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
atorg.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
atorg.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
atio.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
atio.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
atio.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) at
io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
atorg.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
atorg.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
atorg.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
atorg.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
at java.lang.Thread.run(Thread.java:748) Caused by:
org.hibernate.exception.LockAcquisitionException: could not execute statement at
org.hibernate.dialect.MySQLDialect$3.convert(MySQLDialect.java:511)
atorg.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
atorg.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)
atorg.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97)
atorg.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:207)
atorg.hibernate.engine.jdbc.batch.internal.NonBatchingBatch.addToBatch(NonBatchingBatch.java:45)
atorg.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3013)
atorg.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3513)
atorg.hibernate.action.internal.EntityInsertAction.execute(EntityInsertAction.java:89)
atorg.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:589)
atorg.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:463)
atorg.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:337)
atorg.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:39)
at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1295) at
org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:468)
atorg.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3159)
atorg.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:2352)
atorg.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion(JdbcCoordinatorImpl.java:491)
atorg.hibernate.resource.transaction.backend.jta.internal.JtaTransactionCoordinatorImpl.beforeCompletion(JtaTransactionCoordinatorImpl.java:316)
atorg.hibernate.resource.transaction.backend.jta.internal.synchronization.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:47)
atorg.hibernate.resource.transaction.backend.jta.internal.synchronization.RegisteredSynchronization.beforeCompletion(RegisteredSynchronization.java:37)
atorg.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:236)
atorg.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:247)
atorg.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:292)
atcom.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
atcom.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
atcom.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
atcom.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
atcom.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
atcom.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
atorg.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:77)
atorg.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
atorg.keycloak.transaction.JtaTransactionWrapper.commit(JtaTransactionWrapper.java:92)
... 63 more Caused by: com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException:
Lock wait timeout exceeded; try restarting transaction
atcom.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:121)
atcom.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:95)
atcom.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:960)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1116)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1066)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1396)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:1051)
atorg.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:537)
atorg.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:204)
... 92 more
3:33 a.m.
Cool, it can help maybe. You will see... Another thing is, that we know
that big number of realms is causing issues. There are some improvements
planned, hopefully for this year.
Marek
On 19/03/2019 06:23, Madhu wrote:
Hi Marek,
Thanks for quick response.
I double checked my connection pool settings, this is what i have
configured
Min Pool Size:50
Max Pool Size:100
Flush Strategy:IdleConnections
Pool Fair:true
Pool Prefill:false
Pool Use Strict Min:false
Use Fast Fail:false
Do you think i need to change this, i donth think the system is
starving for connections.. will dig more deep here and get back.
Madhu
On Tuesday, 19 March, 2019, 2:51:39 am IST, Madhu <kkcmadhu(a)yahoo.com>
wrote:
Realm is tenant in my case. I have more than 600 realms and each realm
has about 6 clients (excluding what gets shipped by default). There
are 2 realm roles and 2 to 3 client roles per client, I have a
scriptmapper and 2 groups and about 10 users in each realm.
My Max db connection is 30. Let me check that again try increasing it..
Madhu
Sent from Yahoo Mail on Android
<https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrow...
On Tue, 19 Mar 2019 at 2:21 AM, Marek Posolda
<mposolda(a)redhat.com> wrote:
Hi,
what exactly is "tenant" in your case? Is it client or realm?
We know that there are some issues with big number of those
entities, so you will probably see issues with 500 or more
realms/clients. Maybe it helps to increase count of max DB
connections - both at the datasource level in standalone(-ha).xml
and in the settings of your MySQL DB. But not really sure...
Marek
On 18/03/2019 15:59, Madhu wrote:
Hi ,
I am using keycloak 4.5.0.Final in one on my projects and i have
fairly large number of tenants (> 500).
Off late i see frequently lock acquisation related errors and
timeout.
I am not able to figure out where and how this is origniating? can
you please help?
My suspecion is is this related to events logging? could this be
because of farily large number of entrys in the audit/ events table?
Note the thread id default task-19354 in the for event
REFRESH_TOKEN_ERROR and corresponding thread throwing
LockAcquisitionException
Regards,
Madhu
2019-03-17 17:14:47,010 WARN [org.keycloak.events] (default
task-19354) type=REFRESH_TOKEN_ERROR, realmId=*********,
clientId=null, userId=null, ipAddress= xx.yy.zz.aaa,
error=invalid_client_credentials, grant_type=refresh_token
2019-03-17 17:15:13,183 WARN [org.keycloak.events] (default
task-19354) type=REFRESH_TOKEN_ERROR, realmId=*********,
clientId=null, userId=null, ipAddress=xx.yy.zz.aaa,
error=invalid_client_credentials, grant_type=refresh_token
2019-03-17 17:24:31,128 WARN [org.keycloak.events] (default
task-19353) type=REFRESH_TOKEN_ERROR, realmId=*********,
clientId=null, userId=null, ipAddress=xx.yy.zz.aaa,
error=invalid_client_credentials, grant_type=refresh_token
2019-03-17 17:46:17,677 WARN [org.keycloak.events] (default
task-19353) type=REFRESH_TOKEN_ERROR, realmId=*********,
clientId=null, userId=null, ipAddress=xx.yy.zz.aaa,
error=invalid_client_credentials, grant_type=refresh_token
2019-03-17 17:47:00,850 WARN [org.keycloak.events] (default
task-19353) type=REFRESH_TOKEN_ERROR, realmId=*********,
clientId=null, userId=null, ipAddress=xx.yy.zz.aaa,
error=invalid_client_credentials, grant_type=refresh_token
2019-03-17 18:46:48,058 WARN
[org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (default
task-19354) SQL Error: 1205, SQLState: 40001
2019-03-17 18:46:48,059 ERROR
[org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (default
task-19354) Lock wait timeout exceeded; try restarting transaction
2019-03-17 18:46:48,059 INFO
[org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl]
(default task-19354) HHH000010: On release of batch it still
contained JDBC statements
2019-03-17 18:46:48,077 WARN [com.arjuna.ats.arjuna] (default
task-19354) ARJUNA012125: TwoPhaseCoordinator.beforeCompletion -
failed for SynchronizationImple<
0:ffffc0a803b1:-a9285f2:5bf97526:bb36de,
org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization@5ad70e16
>: javax.persistence.PersistenceException:
org.hibernate.exception.LockAcquisitionException: could not
execute statement
at
org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
at
org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
at
org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1608)
at
org.hibernate.jpa.internal.EntityManagerImpl$CallbackExceptionMapperImpl.mapManagedFlushFailure(EntityManagerImpl.java:235)
at
org.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3163)
at
org.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:2352)
at
org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion(JdbcCoordinatorImpl.java:491)
at
org.hibernate.resource.transaction.backend.jta.internal.JtaTransactionCoordinatorImpl.beforeCompletion(JtaTransactionCoordinatorImpl.java:316)
at
org.hibernate.resource.transaction.backend.jta.internal.synchronization.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:47)
at
org.hibernate.resource.transaction.backend.jta.internal.synchronization.RegisteredSynchronization.beforeCompletion(RegisteredSynchronization.java:37)
at
org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:236)
at
org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:247)
at
org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:292)
at
com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
at
com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
at
com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at
com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
at
com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at
com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at
org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:77)
at
org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
at
org.keycloak.transaction.JtaTransactionWrapper.commit(JtaTransactionWrapper.java:92)
at
org.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:136)
at
org.keycloak.services.filters.KeycloakTransactionCommitter.filter(KeycloakTransactionCommitter.java:43)
at
org.jboss.resteasy.core.interception.ContainerResponseContextImpl.filter(ContainerResponseContextImpl.java:353)
at
org.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:207)
at
org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:85)
at
org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:59)
at
org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:530)
at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:461)
at
org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:231)
at
org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:137)
at
org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:361)
at
org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:140)
at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:217)
at
org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at
io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
at
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at
org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:90)
at
io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at
io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at
io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at
io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
at
io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at
org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
at
io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at
io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at
io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at
io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at
io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
at
io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
at
io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
at
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
at
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
at
io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at
io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at
org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
at
io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
at
io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at
io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
at
io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
at
io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
at
org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at
org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.hibernate.exception.LockAcquisitionException: could
not execute statement
at org.hibernate.dialect.MySQLDialect$3.convert(MySQLDialect.java:511)
at
org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
at
org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)
at
org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97)
at
org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:207)
at
org.hibernate.engine.jdbc.batch.internal.NonBatchingBatch.addToBatch(NonBatchingBatch.java:45)
at
org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3013)
at
org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3513)
at
org.hibernate.action.internal.EntityInsertAction.execute(EntityInsertAction.java:89)
at
org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:589)
at
org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:463)
at
org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:337)
at
org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:39)
at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1295)
at
org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:468)
at
org.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3159)
... 81 more
Caused by:
com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException:
Lock wait timeout exceeded; try restarting transaction
at
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:121)
at
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:95)
at
com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:960)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1116)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1066)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1396)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:1051)
at
org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:537)
at
org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:204)
... 92 more
2019-03-17 18:46:48,558 ERROR
[org.keycloak.services.error.KeycloakErrorHandler] (default
task-19354) Uncaught server error:
org.keycloak.models.ModelException:
org.hibernate.exception.LockAcquisitionException: could not
execute statement
at
org.keycloak.connections.jpa.PersistenceExceptionConverter.convert(PersistenceExceptionConverter.java:61)
at
org.keycloak.connections.jpa.JpaExceptionConverter.convert(JpaExceptionConverter.java:31)
at
org.keycloak.transaction.JtaTransactionWrapper.handleException(JtaTransactionWrapper.java:65)
at
org.keycloak.transaction.JtaTransactionWrapper.commit(JtaTransactionWrapper.java:94)
at
org.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:136)
at
org.keycloak.services.filters.KeycloakTransactionCommitter.filter(KeycloakTransactionCommitter.java:43)
at
org.jboss.resteasy.core.interception.ContainerResponseContextImpl.filter(ContainerResponseContextImpl.java:353)
at
org.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:207)
at
org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:85)
at
org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:59)
at
org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:530)
at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:461)
at
org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:231)
at
org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:137)
at
org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:361)
at
org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:140)
at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:217)
at
org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at
io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
at
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at
org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:90)
at
io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at
io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at
io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at
io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
at
io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at
org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
at
io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at
io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at
io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at
io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at
io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
at
io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at
io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
at
io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
at
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
at
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
at
io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at
io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at
org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
at
io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
at
io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at
io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
at
io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
at
io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
at
org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at
org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.hibernate.exception.LockAcquisitionException: could
not execute statement
at org.hibernate.dialect.MySQLDialect$3.convert(MySQLDialect.java:511)
at
org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
at
org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)
at
org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97)
at
org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:207)
at
org.hibernate.engine.jdbc.batch.internal.NonBatchingBatch.addToBatch(NonBatchingBatch.java:45)
at
org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3013)
at
org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3513)
at
org.hibernate.action.internal.EntityInsertAction.execute(EntityInsertAction.java:89)
at
org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:589)
at
org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:463)
at
org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:337)
at
org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:39)
at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1295)
at
org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:468)
at
org.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3159)
at
org.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:2352)
at
org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion(JdbcCoordinatorImpl.java:491)
at
org.hibernate.resource.transaction.backend.jta.internal.JtaTransactionCoordinatorImpl.beforeCompletion(JtaTransactionCoordinatorImpl.java:316)
at
org.hibernate.resource.transaction.backend.jta.internal.synchronization.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:47)
at
org.hibernate.resource.transaction.backend.jta.internal.synchronization.RegisteredSynchronization.beforeCompletion(RegisteredSynchronization.java:37)
at
org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:236)
at
org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:247)
at
org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:292)
at
com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
at
com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
at
com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at
com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
at
com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at
com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at
org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:77)
at
org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
at
org.keycloak.transaction.JtaTransactionWrapper.commit(JtaTransactionWrapper.java:92)
... 63 more
Caused by:
com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException:
Lock wait timeout exceeded; try restarting transaction
at
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:121)
at
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:95)
at
com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:960)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1116)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1066)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1396)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:1051)
at
org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:537)
at
org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:204)
... 92 more
4:59 a.m.
Nope, i Meant, i already have these settingMin Pool Size:50Max Pool Size:100Flush
Strategy:IdleConnectionsPool Fair:truePool Prefill:falsePool Use Strict Min:falseUse Fast
Fail:false
and it did not help :(
On Tuesday, 19 March, 2019, 2:03:48 pm IST, Marek Posolda <mposolda(a)redhat.com>
wrote:
Cool, it can help maybe. You will see... Another thing is, that we know that big number
of realms is causing issues. There are some improvements planned, hopefully for this year.
Marek
On 19/03/2019 06:23, Madhu wrote:
Hi Marek,
Thanks for quick response.
I double checked my connection pool settings, this is what i have configured Min Pool
Size:50 Max Pool Size:100 Flush Strategy:IdleConnections Pool Fair:true Pool Prefill:false
Pool Use Strict Min:false Use Fast Fail:false
Do you think i need to change this, i donth think the system is starving for
connections.. will dig more deep here and get back.
Madhu On Tuesday, 19 March, 2019, 2:51:39 am IST, Madhu <kkcmadhu(a)yahoo.com>
wrote:
Realm is tenant in my case. I have more than 600 realms and each realm has about 6
clients (excluding what gets shipped by default). There are 2 realm roles and 2 to 3
client roles per client, I have a scriptmapper and 2 groups and about 10 users in each
realm. My Max db connection is 30. Let me check that again try increasing it..
Madhu
Sent from Yahoo Mail on Android
On Tue, 19 Mar 2019 at 2:21 AM, Marek Posolda <mposolda(a)redhat.com> wrote: Hi,
what exactly is "tenant" in your case? Is it client or realm?
We know that there are some issues with big number of those entities, so you will
probably see issues with 500 or more realms/clients. Maybe it helps to increase count of
max DB connections - both at the datasource level in standalone(-ha).xml and in the
settings of your MySQL DB. But not really sure...
Marek
On 18/03/2019 15:59, Madhu wrote:
Hi ,
I am using keycloak 4.5.0.Final in one on my projects and i have fairly large number of
tenants (> 500). Off late i see frequently lock acquisation related errors and
timeout.
I am not able to figure out where and how this is origniating? can you please help?
My suspecion is is this related to events logging? could this be because of farily
large number of entrys in the audit/ events table?
Note the thread id default task-19354 in the for event REFRESH_TOKEN_ERROR and
corresponding thread throwing LockAcquisitionException
Regards, Madhu
2019-03-17 17:14:47,010 WARN [org.keycloak.events] (default task-19354)
type=REFRESH_TOKEN_ERROR, realmId=*********, clientId=null, userId=null, ipAddress=
xx.yy.zz.aaa, error=invalid_client_credentials, grant_type=refresh_token 2019-03-17
17:15:13,183 WARN [org.keycloak.events] (default task-19354) type=REFRESH_TOKEN_ERROR,
realmId=*********, clientId=null, userId=null, ipAddress=xx.yy.zz.aaa,
error=invalid_client_credentials, grant_type=refresh_token 2019-03-17 17:24:31,128 WARN
[org.keycloak.events] (default task-19353) type=REFRESH_TOKEN_ERROR, realmId=*********,
clientId=null, userId=null, ipAddress=xx.yy.zz.aaa, error=invalid_client_credentials,
grant_type=refresh_token 2019-03-17 17:46:17,677 WARN [org.keycloak.events] (default
task-19353) type=REFRESH_TOKEN_ERROR, realmId=*********, clientId=null, userId=null,
ipAddress=xx.yy.zz.aaa, error=invalid_client_credentials, grant_type=refresh_token
2019-03-17 17:47:00,850 WARN [org.keycloak.events] (default task-19353)
type=REFRESH_TOKEN_ERROR, realmId=*********, clientId=null, userId=null,
ipAddress=xx.yy.zz.aaa, error=invalid_client_credentials, grant_type=refresh_token
2019-03-17 18:46:48,058 WARN [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (default
task-19354) SQL Error: 1205, SQLState: 40001 2019-03-17 18:46:48,059 ERROR
[org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (default task-19354) Lock wait timeout
exceeded; try restarting transaction 2019-03-17 18:46:48,059 INFO
[org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl] (default task-19354)
HHH000010: On release of batch it still contained JDBC statements 2019-03-17 18:46:48,077
WARN [com.arjuna.ats.arjuna] (default task-19354) ARJUNA012125:
TwoPhaseCoordinator.beforeCompletion - failed for SynchronizationImple<
0:ffffc0a803b1:-a9285f2:5bf97526:bb36de,org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization@5ad70e16
>:
javax.persistence.PersistenceException:org.hibernate.exception.LockAcquisitionException:
could not execute statement
atorg.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
atorg.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
atorg.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1608)
atorg.hibernate.jpa.internal.EntityManagerImpl$CallbackExceptionMapperImpl.mapManagedFlushFailure(EntityManagerImpl.java:235)
atorg.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3163)
atorg.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:2352)
atorg.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion(JdbcCoordinatorImpl.java:491)
atorg.hibernate.resource.transaction.backend.jta.internal.JtaTransactionCoordinatorImpl.beforeCompletion(JtaTransactionCoordinatorImpl.java:316)
atorg.hibernate.resource.transaction.backend.jta.internal.synchronization.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:47)
atorg.hibernate.resource.transaction.backend.jta.internal.synchronization.RegisteredSynchronization.beforeCompletion(RegisteredSynchronization.java:37)
atorg.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:236)
atorg.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:247)
atorg.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:292)
atcom.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
atcom.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
atcom.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
atcom.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
atcom.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
atcom.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
atcom.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
atorg.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:77)
atorg.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
atorg.keycloak.transaction.JtaTransactionWrapper.commit(JtaTransactionWrapper.java:92)
atorg.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:136)
atorg.keycloak.services.filters.KeycloakTransactionCommitter.filter(KeycloakTransactionCommitter.java:43)
atorg.jboss.resteasy.core.interception.ContainerResponseContextImpl.filter(ContainerResponseContextImpl.java:353)
atorg.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:207)
atorg.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:85)
atorg.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:59)
atorg.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:530)
atorg.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:461)
atorg.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:231)
atorg.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:137)
atorg.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:361)
atorg.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:140)
atorg.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:217)
atorg.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227)
atorg.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
atorg.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
atjavax.servlet.http.HttpServlet.service(HttpServlet.java:790)
atio.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
atio.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
atorg.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:90)
atio.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
atio.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
atio.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
atio.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
atio.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
atio.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
atorg.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atio.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
atio.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atio.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
atio.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
atio.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
atio.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
atio.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
atio.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atorg.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atorg.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atio.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
atio.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
atio.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
atio.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
atio.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
atio.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
atorg.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
atorg.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
atorg.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
atorg.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
atorg.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
atio.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
atio.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
atio.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
atio.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
atio.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
atorg.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
atorg.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
atorg.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
atorg.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
atjava.lang.Thread.run(Thread.java:748) Caused by:
org.hibernate.exception.LockAcquisitionException: could not execute statement
atorg.hibernate.dialect.MySQLDialect$3.convert(MySQLDialect.java:511)
atorg.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
atorg.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)
atorg.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97)
atorg.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:207)
atorg.hibernate.engine.jdbc.batch.internal.NonBatchingBatch.addToBatch(NonBatchingBatch.java:45)
atorg.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3013)
atorg.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3513)
atorg.hibernate.action.internal.EntityInsertAction.execute(EntityInsertAction.java:89)
atorg.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:589)
atorg.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:463)
atorg.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:337)
atorg.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:39)
atorg.hibernate.internal.SessionImpl.flush(SessionImpl.java:1295)
atorg.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:468)
atorg.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3159)
... 81 more Caused by: com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException:
Lock wait timeout exceeded; try restarting transaction
atcom.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:121)
atcom.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:95)
atcom.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:960)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1116)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1066)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1396)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:1051)
atorg.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:537)
atorg.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:204)
... 92 more
2019-03-17 18:46:48,558 ERROR [org.keycloak.services.error.KeycloakErrorHandler]
(default task-19354) Uncaught server error:
org.keycloak.models.ModelException:org.hibernate.exception.LockAcquisitionException: could
not execute statement
atorg.keycloak.connections.jpa.PersistenceExceptionConverter.convert(PersistenceExceptionConverter.java:61)
atorg.keycloak.connections.jpa.JpaExceptionConverter.convert(JpaExceptionConverter.java:31)
atorg.keycloak.transaction.JtaTransactionWrapper.handleException(JtaTransactionWrapper.java:65)
atorg.keycloak.transaction.JtaTransactionWrapper.commit(JtaTransactionWrapper.java:94)
atorg.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:136)
atorg.keycloak.services.filters.KeycloakTransactionCommitter.filter(KeycloakTransactionCommitter.java:43)
atorg.jboss.resteasy.core.interception.ContainerResponseContextImpl.filter(ContainerResponseContextImpl.java:353)
atorg.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:207)
atorg.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:85)
atorg.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:59)
atorg.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:530)
atorg.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:461)
atorg.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:231)
atorg.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:137)
atorg.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:361)
atorg.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:140)
atorg.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:217)
atorg.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227)
atorg.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
atorg.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
atjavax.servlet.http.HttpServlet.service(HttpServlet.java:790)
atio.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
atio.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
atorg.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:90)
atio.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
atio.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
atio.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
atio.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
atio.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
atio.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
atorg.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atio.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
atio.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atio.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
atio.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
atio.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
atio.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
atio.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
atio.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atorg.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atorg.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
atio.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
atio.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
atio.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
atio.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
atio.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
atio.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
atio.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
atorg.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
atorg.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
atorg.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
atorg.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
atorg.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
atio.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
atio.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
atio.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
atio.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
atio.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
atorg.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
atorg.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
atorg.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
atorg.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
atjava.lang.Thread.run(Thread.java:748) Caused by:
org.hibernate.exception.LockAcquisitionException: could not execute statement
atorg.hibernate.dialect.MySQLDialect$3.convert(MySQLDialect.java:511)
atorg.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
atorg.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)
atorg.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97)
atorg.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:207)
atorg.hibernate.engine.jdbc.batch.internal.NonBatchingBatch.addToBatch(NonBatchingBatch.java:45)
atorg.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3013)
atorg.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3513)
atorg.hibernate.action.internal.EntityInsertAction.execute(EntityInsertAction.java:89)
atorg.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:589)
atorg.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:463)
atorg.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:337)
atorg.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:39)
atorg.hibernate.internal.SessionImpl.flush(SessionImpl.java:1295)
atorg.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:468)
atorg.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3159)
atorg.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:2352)
atorg.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion(JdbcCoordinatorImpl.java:491)
atorg.hibernate.resource.transaction.backend.jta.internal.JtaTransactionCoordinatorImpl.beforeCompletion(JtaTransactionCoordinatorImpl.java:316)
atorg.hibernate.resource.transaction.backend.jta.internal.synchronization.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:47)
atorg.hibernate.resource.transaction.backend.jta.internal.synchronization.RegisteredSynchronization.beforeCompletion(RegisteredSynchronization.java:37)
atorg.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:236)
atorg.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:247)
atorg.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:292)
atcom.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
atcom.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)
atcom.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)
atcom.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
atcom.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
atcom.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
atcom.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
atorg.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:77)
atorg.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
atorg.keycloak.transaction.JtaTransactionWrapper.commit(JtaTransactionWrapper.java:92)
... 63 more Caused by: com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException:
Lock wait timeout exceeded; try restarting transaction
atcom.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:121)
atcom.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:95)
atcom.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:960)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1116)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1066)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1396)
atcom.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:1051)
atorg.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:537)
atorg.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:204)
... 92 more
2587
days inactive
2588
days old
5 comments
2 participants
participants (2)
-
Madhu -
Marek Posolda