shutdown command stops the Infinispan Server but leaves the JVM process running ------------------------------------------------------------------------------- Key: ISPN-12374 URL: https://issues.redhat.com/browse/ISPN-12374 Project: Infinispan Issue Type: Bug Affects Versions: 11.0.3.Final, 12.0.0.Dev04 Reporter: Radoslav Husar Priority: Major Simple steps to reproduce: [rhusar@ribera infinispan-server-11.0.3.Final]$ ./bin/server.sh ... [rhusar@ribera infinispan-server-11.0.3.Final]$ ./bin/cli.sh [disconnected]> connect --username=admin --password=secret [ribera-54749@cluster//containers/default]> shutdown server ribera-54749 Ensuring it's shutdown: [ribera-54749@cluster//containers/default]> shutdown cluster Could not connect to server: java.net.ConnectException: Failed to connect to localhost/0:0:0:0:0:0:0:1:11222 Check the log: 19:36:45,922 INFO (pool-6-thread-1) [org.infinispan.SERVER] ISPN080002: Infinispan Server stopping 19:36:46,050 INFO (pool-6-thread-1) [org.infinispan.manager.DefaultCacheManager] Stopping cache manager on ribera-54749 19:36:46,115 INFO (pool-6-thread-1) [org.infinispan.CLUSTER] ISPN000080: Disconnecting JGroups channel cluster 19:36:46,124 INFO (pool-6-thread-1) [org.infinispan.CONTAINER] ISPN000390: Persisted state, version=11.0.3.Final timestamp=2020-09-30T17:36:46.122Z 19:36:46,131 INFO (pool-6-thread-1) [org.infinispan.SERVER] ISPN080003: Infinispan Server stopped Check ps: {noformat} 43335 ttys001 0:13.50 /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/bin/java -server -Xms64m -Xmx512m -XX:MetaspaceSize=64M -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Dvisualvm.display.name=infinispan-server -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dinfinispan.server.home.path=/Users/rhusar/git/wildfly/testsuite/integration/clustering/target/infinispan-server-11.0.3.Final -classpath :/Users/rhusar/git/wildfly/testsuite/integration/clustering/target/infinispan-server-11.0.3.Final/boot/infinispan-server-runtime-11.0.3.Final-loader.jar org.infinispan.server.loader.Loader org.infinispan.server.Bootstrap {noformat} check jstack: {noformat} [rhusar@ribera infinispan-server-11.0.3.Final]$ jstack 43335 2020-09-30 19:46:57 Full thread dump OpenJDK 64-Bit Server VM (25.265-b01 mixed mode): "Attach Listener" #107 daemon prio=9 os_prio=31 tid=0x00007f984e988800 nid=0xbb07 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "DestroyJavaVM" #106 prio=5 os_prio=31 tid=0x00007f984e910000 nid=0x2903 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "pool-5-thread-1" #76 prio=5 os_prio=31 tid=0x00007f984cb10800 nid=0xc003 waiting on condition [0x0000700013dc0000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007bed3c5d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "RxCachedWorkerPoolEvictor-1" #15 daemon prio=5 os_prio=31 tid=0x00007f9845a36000 nid=0xa503 waiting on condition [0x000070001010c000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007a1820548> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "RxSchedulerPurge-1" #14 daemon prio=5 os_prio=31 tid=0x00007f9845a35000 nid=0x5703 waiting on condition [0x0000700010009000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007a18c3f08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "Log4j2-TF-6-Scheduled-1" #12 daemon prio=5 os_prio=31 tid=0x00007f9849052000 nid=0xa803 waiting on condition [0x000070000ff06000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007a103fb30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "Service Thread" #9 daemon prio=9 os_prio=31 tid=0x00007f984580d800 nid=0x4003 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C1 CompilerThread3" #8 daemon prio=9 os_prio=31 tid=0x00007f9844024000 nid=0x4203 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread2" #7 daemon prio=9 os_prio=31 tid=0x00007f984382b800 nid=0x4303 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread1" #6 daemon prio=9 os_prio=31 tid=0x00007f984382a800 nid=0x3d03 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread0" #5 daemon prio=9 os_prio=31 tid=0x00007f9844842000 nid=0x3b03 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" #4 daemon prio=9 os_prio=31 tid=0x00007f984484e000 nid=0x4403 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007f9843015000 nid=0x3403 in Object.wait() [0x000070000f5e8000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000007a08693d8> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) - locked <0x00000007a08693d8> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216) "Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007f9843012000 nid=0x3303 in Object.wait() [0x000070000f4e5000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000007a08b7420> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference.tryHandlePending(Reference.java:191) - locked <0x00000007a08b7420> (a java.lang.ref.Reference$Lock) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) "VM Thread" os_prio=31 tid=0x00007f9844838000 nid=0x3103 runnable "GC task thread#0 (ParallelGC)" os_prio=31 tid=0x00007f984400f000 nid=0x1d07 runnable "GC task thread#1 (ParallelGC)" os_prio=31 tid=0x00007f9844809000 nid=0x1f03 runnable "GC task thread#2 (ParallelGC)" os_prio=31 tid=0x00007f9844809800 nid=0x5403 runnable "GC task thread#3 (ParallelGC)" os_prio=31 tid=0x00007f9845808800 nid=0x2b03 runnable "GC task thread#4 (ParallelGC)" os_prio=31 tid=0x00007f9845809000 nid=0x5103 runnable "GC task thread#5 (ParallelGC)" os_prio=31 tid=0x00007f984300b000 nid=0x2c03 runnable "GC task thread#6 (ParallelGC)" os_prio=31 tid=0x00007f984580a000 nid=0x4f03 runnable "GC task thread#7 (ParallelGC)" os_prio=31 tid=0x00007f9843809800 nid=0x2f03 runnable "VM Periodic Task Thread" os_prio=31 tid=0x00007f984580e800 nid=0x5603 waiting on condition JNI global references: 2019 {noformat} This logic is use by the forked test driver, if its for some reason meant to keep running, this becomes a driver issue.