Deadlock in cleanup following DatabaseCertLoginModuleTestCase ------------------------------------------------------------- Key: WFLY-11706 URL: https://issues.jboss.org/browse/WFLY-11706 Project: WildFly Issue Type: Bug Components: Test Suite, Transactions Reporter: Brian Stansberry Assignee: Tom Jenkinson Priority: Critical Attachments: hang.txt A test hung due to a server-side deadlock. https://ci.wildfly.org/viewLog.html?buildId=139407&buildTypeId=WFPR&a... Build log output stopped at this point: [18:27:07][Step 2/3] [INFO] Running org.jboss.as.test.manualmode.security.OutboundLdapConnectionTestCase [18:27:42][Step 2/3] [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.744 s - in org.jboss.as.test.manualmode.security.OutboundLdapConnectionTestCase [18:27:43][Step 2/3] [INFO] Running org.jboss.as.test.manualmode.web.ssl.DatabaseCertLoginModuleTestCase The 'main' thread of the test shows the test was shutting down the server: {code} "main" #1 prio=5 os_prio=0 tid=0xf6507c00 nid=0x205c in Object.wait() [0xf667d000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0xc807a168> (a java.lang.UNIXProcess) at java.lang.Object.wait(Object.java:502) at java.lang.UNIXProcess.waitFor(UNIXProcess.java:395) - locked <0xc807a168> (a java.lang.UNIXProcess) at org.wildfly.core.launcher.ProcessHelper.destroyProcess(ProcessHelper.java:60) at org.jboss.as.arquillian.container.managed.ManagedDeployableContainer.stopInternal(ManagedDeployableContainer.java:294) at org.jboss.as.arquillian.container.CommonDeployableContainer.stop(CommonDeployableContainer.java:135) at org.jboss.arquillian.container.impl.ContainerImpl.stop(ContainerImpl.java:217) at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController$9.perform(ContainerLifecycleController.java:178) at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController$9.perform(ContainerLifecycleController.java:172) at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController.forContainer(ContainerLifecycleController.java:255) at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController.stopContainer(ContainerLifecycleController.java:171) {code} The server-side thread dump shows the hang: {code} Found one Java-level deadlock: ============================= "Periodic Recovery": waiting to lock monitor 0xbcc6dbc0 (object 0xc8f80918, a java.util.concurrent.atomic.AtomicInteger), which is held by "ServerService Thread Pool -- 8" "ServerService Thread Pool -- 8": waiting to lock monitor 0xc1a6ffc8 (object 0xc8f64470, a com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule), which is held by "Periodic Recovery" Java stack information for the threads listed above: =================================================== "Periodic Recovery": at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.setScanState(XARecoveryModule.java:1088) - waiting to lock <0xc8f80918> (a java.util.concurrent.atomic.AtomicInteger) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:240) - locked <0xc8f64470> (a com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule) at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:816) at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:382) "ServerService Thread Pool -- 8": at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.removeXAResourceRecoveryHelper(XARecoveryModule.java:119) - waiting to lock <0xc8f64470> (a com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule) - locked <0xc8f80918> (a java.util.concurrent.atomic.AtomicInteger) at com.arjuna.ats.jbossatx.jta.RecoveryManagerService.removeXAResourceRecovery(RecoveryManagerService.java:129) at org.jboss.jca.core.tx.jbossts.XAResourceRecoveryRegistryImpl.removeXAResourceRecovery(XAResourceRecoveryRegistryImpl.java:63) at org.jboss.as.connector.subsystems.datasources.XaDataSourceService.stopService(XaDataSourceService.java:66) - locked <0xc8f95a38> (a org.jboss.as.connector.subsystems.datasources.XaDataSourceService) at org.jboss.as.connector.subsystems.datasources.AbstractDataSourceService$1.run(AbstractDataSourceService.java:188) at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377) at java.lang.Thread.run(Thread.java:748) at org.jboss.threads.JBossThread.run(JBossThread.java:485) Found 1 deadlock. {code} I'll attach the full thread dump.