[jboss-jira] [JBoss JIRA] (WFCORE-4047) MSC Deadlock in undertow subsystem test

MSC Deadlock in undertow subsystem test --------------------------------------- Key: WFCORE-4047 URL: https://issues.jboss.org/browse/WFCORE-4047 Project: WildFly Core Issue Type: Bug Components: Management Reporter: Brian Stansberry Assignee: Richard Opalka Priority: Critical Fix For: 6.0.0.CR3 Testsuite hang in https://ci.wildfly.org/viewLog.html?buildId=118052&buildTypeId=WF_Pul... {code} [11:30:50][Step 2/3] 2018-08-20 11:30:49 Full thread dump Java HotSpot(TM) Server VM (25.152-b16 mixed mode): "Attach Listener" #182 daemon prio=9 os_prio=0 tid=0xa3950000 nid=0x3e28 waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE "XNIO-1 Accept" #181 prio=5 os_prio=0 tid=0xa1ec2000 nid=0x56d4 runnable [0xa090b000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0xe55f89f0> (a sun.nio.ch.Util$3) - locked <0xe55f89e0> (a java.util.Collections$UnmodifiableSet) - locked <0xe55f88e8> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.xnio.nio.WorkerThread.run(WorkerThread.java:532) "XNIO-1 I/O-2" #180 prio=5 os_prio=0 tid=0xa1ec1800 nid=0x56d3 runnable [0xa11ad000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0xe55f7d60> (a sun.nio.ch.Util$3) - locked <0xe55f7d50> (a java.util.Collections$UnmodifiableSet) - locked <0xe55f7c58> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.xnio.nio.WorkerThread.run(WorkerThread.java:532) "XNIO-1 I/O-1" #179 prio=5 os_prio=0 tid=0x9f4bc400 nid=0x56d2 runnable [0xa18ad000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0xe55f83a8> (a sun.nio.ch.Util$3) - locked <0xe55f8398> (a java.util.Collections$UnmodifiableSet) - locked <0xe55f82a0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.xnio.nio.WorkerThread.run(WorkerThread.java:532) "MSC service thread 11-8" #178 prio=5 os_prio=0 tid=0x09d34c00 nid=0x56d0 waiting for monitor entry [0xa08ba000] java.lang.Thread.State: BLOCKED (on object monitor) at org.jboss.msc.service.ServiceControllerImpl.dependencyAvailable(ServiceControllerImpl.java:850) - waiting to lock <0xe561fa38> (a org.jboss.msc.service.ServiceControllerImpl) at org.jboss.msc.service.ServiceControllerImpl$DependencyAvailableTask.inform(ServiceControllerImpl.java:1665) at org.jboss.msc.service.ServiceControllerImpl$DependentsControllerTask.execute(ServiceControllerImpl.java:1612) at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1556) at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) at java.lang.Thread.run(Thread.java:748) "Controller Boot Thread" #177 prio=5 os_prio=0 tid=0x9fc9c400 nid=0x56cf waiting for monitor entry [0xa0efe000] java.lang.Thread.State: BLOCKED (on object monitor) at org.jboss.msc.service.ServiceControllerImpl.getImmediateUnavailableDependencies(ServiceControllerImpl.java:1287) - waiting to lock <0xe5718bf0> (a org.jboss.msc.service.ServiceRegistrationImpl) - locked <0xe561fa38> (a org.jboss.msc.service.ServiceControllerImpl) at org.jboss.msc.service.ServiceControllerImpl.getUnavailableDependencies(ServiceControllerImpl.java:1279) at org.jboss.as.controller.ServiceVerificationHelper.execute(ServiceVerificationHelper.java:93) - locked <0xe575c008> (a org.jboss.as.controller.ServiceVerificationHelper) at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999) at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743) at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467) at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1411) at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:495) at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:470) at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:432) at org.jboss.as.model.test.ModelTestModelControllerService.boot(ModelTestModelControllerService.java:276) at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:419) at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:372) at java.lang.Thread.run(Thread.java:748) "MSC service thread 11-7" #176 prio=5 os_prio=0 tid=0xa3954800 nid=0x56ce waiting on condition [0x9fa5c000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0xe570b240> (a org.jboss.threads.EnhancedQueueExecutor) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1410) at java.lang.Thread.run(Thread.java:748) "MSC service thread 11-6" #175 prio=5 os_prio=0 tid=0xa0a57c00 nid=0x56cd waiting for monitor entry [0xa095c000] java.lang.Thread.State: BLOCKED (on object monitor) at org.jboss.msc.service.ServiceControllerImpl.dependencyAvailable(ServiceControllerImpl.java:850) - waiting to lock <0xe561fa38> (a org.jboss.msc.service.ServiceControllerImpl) at org.jboss.msc.service.ServiceControllerImpl$DependencyAvailableTask.inform(ServiceControllerImpl.java:1665) at org.jboss.msc.service.ServiceControllerImpl$DependentsControllerTask.execute(ServiceControllerImpl.java:1612) at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1556) at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) at java.lang.Thread.run(Thread.java:748) "MSC service thread 11-5" #174 prio=5 os_prio=0 tid=0xa12a8000 nid=0x56cc waiting on condition [0xa0686000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0xe570b240> (a org.jboss.threads.EnhancedQueueExecutor) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1410) at java.lang.Thread.run(Thread.java:748) "MSC service thread 11-4" #173 prio=5 os_prio=0 tid=0xa0a59400 nid=0x56cb waiting on condition [0x9fe5c000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0xe570b240> (a org.jboss.threads.EnhancedQueueExecutor) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1410) at java.lang.Thread.run(Thread.java:748) "MSC service thread 11-3" #172 prio=5 os_prio=0 tid=0xa0a56800 nid=0x56ca waiting on condition [0x9fead000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0xe570b240> (a org.jboss.threads.EnhancedQueueExecutor) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1410) at java.lang.Thread.run(Thread.java:748) "MSC service thread 11-2" #171 prio=5 os_prio=0 tid=0xa129ac00 nid=0x56c9 waiting for monitor entry [0xa0bac000] java.lang.Thread.State: BLOCKED (on object monitor) at org.jboss.msc.service.ServiceControllerImpl.dependencyAvailable(ServiceControllerImpl.java:850) - waiting to lock <0xe561fa38> (a org.jboss.msc.service.ServiceControllerImpl) at org.jboss.msc.service.ServiceControllerImpl$DependencyAvailableTask.inform(ServiceControllerImpl.java:1665) at org.jboss.msc.service.ServiceControllerImpl$DependentsControllerTask.execute(ServiceControllerImpl.java:1612) at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1556) at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) at java.lang.Thread.run(Thread.java:748) "MSC service thread 11-1" #170 prio=5 os_prio=0 tid=0x9fc9d000 nid=0x56c8 waiting on condition [0x9fbfe000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0xe570b240> (a org.jboss.threads.EnhancedQueueExecutor) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1410) at java.lang.Thread.run(Thread.java:748) "Reference Reaper" #14 daemon prio=5 os_prio=0 tid=0xa16e1000 nid=0x5628 in Object.wait() [0xa18fe000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0xa749ccf8> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) - locked <0xa749ccf8> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) at org.jboss.modules.ref.References$ReaperThread.run(References.java:64) "process reaper" #11 daemon prio=10 os_prio=0 tid=0x09c3a800 nid=0x5626 waiting on condition [0xa288c000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0xa73305f8> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "surefire-forkedjvm-ping-30s" #10 daemon prio=5 os_prio=0 tid=0xa3164c00 nid=0x5624 waiting on condition [0xa3271000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0xa71c1c00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "surefire-forkedjvm-command-thread" #9 daemon prio=5 os_prio=0 tid=0xa3154c00 nid=0x5623 runnable [0xa32c2000] java.lang.Thread.State: RUNNABLE at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:255) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read(BufferedInputStream.java:265) - locked <0xa71f9040> (a java.io.BufferedInputStream) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.maven.surefire.booter.MasterProcessCommand.decode(MasterProcessCommand.java:115) at org.apache.maven.surefire.booter.CommandReader$CommandRunnable.run(CommandReader.java:391) at java.lang.Thread.run(Thread.java:748) "Service Thread" #8 daemon prio=9 os_prio=0 tid=0xa3aab400 nid=0x5621 runnable [0x00000000] java.lang.Thread.State: RUNNABLE "C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0xa3aa7c00 nid=0x5620 waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0xa3aa6800 nid=0x561f waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0xa3aa4800 nid=0x561e waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0xa3aa3000 nid=0x561d runnable [0x00000000] java.lang.Thread.State: RUNNABLE "Finalizer" #3 daemon prio=8 os_prio=0 tid=0xa3a88c00 nid=0x561c in Object.wait() [0xa3b57000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) - locked <0xa71f7ee8> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0xa3a86000 nid=0x561b in Object.wait() [0xa677d000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference.tryHandlePending(Reference.java:191) - locked <0xa71ec3e8> (a java.lang.ref.Reference$Lock) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) "main" #1 prio=5 os_prio=0 tid=0xf6507400 nid=0x5615 waiting for monitor entry [0xf666f000] java.lang.Thread.State: BLOCKED (on object monitor) at org.jboss.msc.service.ServiceControllerImpl.getState(ServiceControllerImpl.java:1101) - waiting to lock <0xe561fa38> (a org.jboss.msc.service.ServiceControllerImpl) at org.jboss.msc.service.ServiceContainerImpl.detectCircularity(ServiceContainerImpl.java:806) at org.jboss.msc.service.ServiceContainerImpl.detectCircularity(ServiceContainerImpl.java:784) - locked <0xe5718bf0> (a org.jboss.msc.service.ServiceRegistrationImpl) at org.jboss.msc.service.ServiceContainerImpl.install(ServiceContainerImpl.java:761) at org.jboss.msc.service.ServiceTargetImpl.install(ServiceTargetImpl.java:260) at org.jboss.msc.service.ObsoleteServiceBuilderImpl.install(ObsoleteServiceBuilderImpl.java:293) at org.wildfly.extension.undertow.AbstractUndertowSubsystemTestCase$RuntimeInitialization.addExtraServices(AbstractUndertowSubsystemTestCase.java:240) at org.jboss.as.subsystem.test.AbstractKernelServicesImpl.create(AbstractKernelServicesImpl.java:127) at org.jboss.as.subsystem.test.SubsystemTestDelegate$KernelServicesBuilderImpl.build(SubsystemTestDelegate.java:568) at org.wildfly.extension.undertow.UndertowSubsystem11TestCase.testRuntime(UndertowSubsystem11TestCase.java:72) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:27) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75) at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:383) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:344) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:417) "VM Thread" os_prio=0 tid=0xa3a81800 nid=0x561a runnable "GC task thread#0 (ParallelGC)" os_prio=0 tid=0xf6510c00 nid=0x5616 runnable "GC task thread#1 (ParallelGC)" os_prio=0 tid=0xf6512000 nid=0x5617 runnable "GC task thread#2 (ParallelGC)" os_prio=0 tid=0xf6513800 nid=0x5618 runnable "GC task thread#3 (ParallelGC)" os_prio=0 tid=0xf6514c00 nid=0x5619 runnable "VM Periodic Task Thread" os_prio=0 tid=0xa3ab7400 nid=0x5622 waiting on condition JNI global references: 706 Found one Java-level deadlock: ============================= "MSC service thread 11-8": waiting to lock monitor 0xa221b37c (object 0xe561fa38, a org.jboss.msc.service.ServiceControllerImpl), which is held by "Controller Boot Thread" "Controller Boot Thread": waiting to lock monitor 0xa20b3f64 (object 0xe5718bf0, a org.jboss.msc.service.ServiceRegistrationImpl), which is held by "main" "main": waiting to lock monitor 0xa221b37c (object 0xe561fa38, a org.jboss.msc.service.ServiceControllerImpl), which is held by "Controller Boot Thread" Java stack information for the threads listed above: =================================================== "MSC service thread 11-8": at org.jboss.msc.service.ServiceControllerImpl.dependencyAvailable(ServiceControllerImpl.java:850) - waiting to lock <0xe561fa38> (a org.jboss.msc.service.ServiceControllerImpl) at org.jboss.msc.service.ServiceControllerImpl$DependencyAvailableTask.inform(ServiceControllerImpl.java:1665) at org.jboss.msc.service.ServiceControllerImpl$DependentsControllerTask.execute(ServiceControllerImpl.java:1612) at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1556) at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) at java.lang.Thread.run(Thread.java:748) "Controller Boot Thread": at org.jboss.msc.service.ServiceControllerImpl.getImmediateUnavailableDependencies(ServiceControllerImpl.java:1287) - waiting to lock <0xe5718bf0> (a org.jboss.msc.service.ServiceRegistrationImpl) - locked <0xe561fa38> (a org.jboss.msc.service.ServiceControllerImpl) at org.jboss.msc.service.ServiceControllerImpl.getUnavailableDependencies(ServiceControllerImpl.java:1279) at org.jboss.as.controller.ServiceVerificationHelper.execute(ServiceVerificationHelper.java:93) - locked <0xe575c008> (a org.jboss.as.controller.ServiceVerificationHelper) at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999) at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743) at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467) at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1411) at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:495) at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:470) at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:432) at org.jboss.as.model.test.ModelTestModelControllerService.boot(ModelTestModelControllerService.java:276) at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:419) at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:372) at java.lang.Thread.run(Thread.java:748) "main": at org.jboss.msc.service.ServiceControllerImpl.getState(ServiceControllerImpl.java:1101) - waiting to lock <0xe561fa38> (a org.jboss.msc.service.ServiceControllerImpl) at org.jboss.msc.service.ServiceContainerImpl.detectCircularity(ServiceContainerImpl.java:806) at org.jboss.msc.service.ServiceContainerImpl.detectCircularity(ServiceContainerImpl.java:784) - locked <0xe5718bf0> (a org.jboss.msc.service.ServiceRegistrationImpl) at org.jboss.msc.service.ServiceContainerImpl.install(ServiceContainerImpl.java:761) at org.jboss.msc.service.ServiceTargetImpl.install(ServiceTargetImpl.java:260) at org.jboss.msc.service.ObsoleteServiceBuilderImpl.install(ObsoleteServiceBuilderImpl.java:293) at org.wildfly.extension.undertow.AbstractUndertowSubsystemTestCase$RuntimeInitialization.addExtraServices(AbstractUndertowSubsystemTestCase.java:240) at org.jboss.as.subsystem.test.AbstractKernelServicesImpl.create(AbstractKernelServicesImpl.java:127) at org.jboss.as.subsystem.test.SubsystemTestDelegate$KernelServicesBuilderImpl.build(SubsystemTestDelegate.java:568) at org.wildfly.extension.undertow.UndertowSubsystem11TestCase.testRuntime(UndertowSubsystem11TestCase.java:72) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:27) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75) at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:383) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:344) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:417) Found 1 deadlock. {code} What's going on here is the main thread in subsystem test fixture AbstractKernelServicesImpl at L115 installs a Service<ModelController>. Like a normal, non-test-fixture boot that spawns the ControllerBootThread to do parsing and run the boot management ops. Then main continues on to L127 where it starts adding any additional services the test wants. AbstractUndertowSubsystemTestCase.RuntimeInitialization installs a bunch. So, we have two threads, ControllerBootThread and main, installing services. And then of course MSC threads can also install services. What the subsystem test fixtures does where main is adding services at the same time the boot thread is running is unorthodox in terms of how a real server would operate. But, it's been this way for years and I've never seen a deadlock like this before. [~ropalka] I'm assigning this to you because this seems more like an MSC problem.