[JBoss JIRA] (ISPN-12374) shutdown command stops the Infinispan Server but leaves the JVM process running
by Radoslav Husar (Jira)
[ https://issues.redhat.com/browse/ISPN-12374?page=com.atlassian.jira.plugi... ]
Radoslav Husar updated ISPN-12374:
----------------------------------
Affects Version/s: 12.0.0.Dev04
> shutdown command stops the Infinispan Server but leaves the JVM process running
> -------------------------------------------------------------------------------
>
> Key: ISPN-12374
> URL: https://issues.redhat.com/browse/ISPN-12374
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 11.0.3.Final, 12.0.0.Dev04
> Reporter: Radoslav Husar
> Priority: Major
>
> Simple steps to reproduce:
> [rhusar@ribera infinispan-server-11.0.3.Final]$ ./bin/server.sh
> ...
> [rhusar@ribera infinispan-server-11.0.3.Final]$ ./bin/cli.sh
> [disconnected]> connect --username=admin --password=secret
> [ribera-54749@cluster//containers/default]> shutdown server ribera-54749
> Ensuring it's shutdown:
> [ribera-54749@cluster//containers/default]> shutdown cluster
> Could not connect to server: java.net.ConnectException: Failed to connect to localhost/0:0:0:0:0:0:0:1:11222
> Check the log:
> 19:36:45,922 INFO (pool-6-thread-1) [org.infinispan.SERVER] ISPN080002: Infinispan Server stopping
> 19:36:46,050 INFO (pool-6-thread-1) [org.infinispan.manager.DefaultCacheManager] Stopping cache manager on ribera-54749
> 19:36:46,115 INFO (pool-6-thread-1) [org.infinispan.CLUSTER] ISPN000080: Disconnecting JGroups channel cluster
> 19:36:46,124 INFO (pool-6-thread-1) [org.infinispan.CONTAINER] ISPN000390: Persisted state, version=11.0.3.Final timestamp=2020-09-30T17:36:46.122Z
> 19:36:46,131 INFO (pool-6-thread-1) [org.infinispan.SERVER] ISPN080003: Infinispan Server stopped
> Check ps:
> {noformat}
> 43335 ttys001 0:13.50 /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/bin/java -server -Xms64m -Xmx512m -XX:MetaspaceSize=64M -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Dvisualvm.display.name=infinispan-server -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dinfinispan.server.home.path=/Users/rhusar/git/wildfly/testsuite/integration/clustering/target/infinispan-server-11.0.3.Final -classpath :/Users/rhusar/git/wildfly/testsuite/integration/clustering/target/infinispan-server-11.0.3.Final/boot/infinispan-server-runtime-11.0.3.Final-loader.jar org.infinispan.server.loader.Loader org.infinispan.server.Bootstrap
> {noformat}
> check jstack:
> {noformat}
> [rhusar@ribera infinispan-server-11.0.3.Final]$ jstack 43335
> 2020-09-30 19:46:57
> Full thread dump OpenJDK 64-Bit Server VM (25.265-b01 mixed mode):
> "Attach Listener" #107 daemon prio=9 os_prio=31 tid=0x00007f984e988800 nid=0xbb07 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "DestroyJavaVM" #106 prio=5 os_prio=31 tid=0x00007f984e910000 nid=0x2903 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "pool-5-thread-1" #76 prio=5 os_prio=31 tid=0x00007f984cb10800 nid=0xc003 waiting on condition [0x0000700013dc0000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000007bed3c5d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> "RxCachedWorkerPoolEvictor-1" #15 daemon prio=5 os_prio=31 tid=0x00007f9845a36000 nid=0xa503 waiting on condition [0x000070001010c000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000007a1820548> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> "RxSchedulerPurge-1" #14 daemon prio=5 os_prio=31 tid=0x00007f9845a35000 nid=0x5703 waiting on condition [0x0000700010009000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000007a18c3f08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> "Log4j2-TF-6-Scheduled-1" #12 daemon prio=5 os_prio=31 tid=0x00007f9849052000 nid=0xa803 waiting on condition [0x000070000ff06000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000007a103fb30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> "Service Thread" #9 daemon prio=9 os_prio=31 tid=0x00007f984580d800 nid=0x4003 runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "C1 CompilerThread3" #8 daemon prio=9 os_prio=31 tid=0x00007f9844024000 nid=0x4203 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread2" #7 daemon prio=9 os_prio=31 tid=0x00007f984382b800 nid=0x4303 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread1" #6 daemon prio=9 os_prio=31 tid=0x00007f984382a800 nid=0x3d03 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread0" #5 daemon prio=9 os_prio=31 tid=0x00007f9844842000 nid=0x3b03 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "Signal Dispatcher" #4 daemon prio=9 os_prio=31 tid=0x00007f984484e000 nid=0x4403 runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007f9843015000 nid=0x3403 in Object.wait() [0x000070000f5e8000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000007a08693d8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
> - locked <0x00000007a08693d8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
> "Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007f9843012000 nid=0x3303 in Object.wait() [0x000070000f4e5000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000007a08b7420> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
> - locked <0x00000007a08b7420> (a java.lang.ref.Reference$Lock)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
> "VM Thread" os_prio=31 tid=0x00007f9844838000 nid=0x3103 runnable
> "GC task thread#0 (ParallelGC)" os_prio=31 tid=0x00007f984400f000 nid=0x1d07 runnable
> "GC task thread#1 (ParallelGC)" os_prio=31 tid=0x00007f9844809000 nid=0x1f03 runnable
> "GC task thread#2 (ParallelGC)" os_prio=31 tid=0x00007f9844809800 nid=0x5403 runnable
> "GC task thread#3 (ParallelGC)" os_prio=31 tid=0x00007f9845808800 nid=0x2b03 runnable
> "GC task thread#4 (ParallelGC)" os_prio=31 tid=0x00007f9845809000 nid=0x5103 runnable
> "GC task thread#5 (ParallelGC)" os_prio=31 tid=0x00007f984300b000 nid=0x2c03 runnable
> "GC task thread#6 (ParallelGC)" os_prio=31 tid=0x00007f984580a000 nid=0x4f03 runnable
> "GC task thread#7 (ParallelGC)" os_prio=31 tid=0x00007f9843809800 nid=0x2f03 runnable
> "VM Periodic Task Thread" os_prio=31 tid=0x00007f984580e800 nid=0x5603 waiting on condition
> JNI global references: 2019
> {noformat}
> This logic is use by the forked test driver, if its for some reason meant to keep running, this becomes a driver issue.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
3 years, 6 months
[JBoss JIRA] (ISPN-12368) ServerRunMode.FORKED always leaks running server processes when cluster shutdown fails
by Radoslav Husar (Jira)
[ https://issues.redhat.com/browse/ISPN-12368?page=com.atlassian.jira.plugi... ]
Radoslav Husar updated ISPN-12368:
----------------------------------
Fix Version/s: 11.0.4.Final
12.0.0.Dev05
> ServerRunMode.FORKED always leaks running server processes when cluster shutdown fails
> --------------------------------------------------------------------------------------
>
> Key: ISPN-12368
> URL: https://issues.redhat.com/browse/ISPN-12368
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Affects Versions: 11.0.3.Final, 12.0.0.Dev04
> Reporter: Radoslav Husar
> Assignee: Radoslav Husar
> Priority: Major
> Fix For: 11.0.4.Final, 12.0.0.Dev05
>
>
> See WFLY-13897 for the 'reproducer' in WildFly.
> This is due to a couple of things:
> 1. Obtaining PID in JDK8 is not implemented.
> 2. Obtaining PID is broken in higher JDKs due to ISPN-12366.
> 3. The obtained PID is wrong anyway, because its the PID of the parent shell process, not the child server process.
> 4. The kill command usage is incorrect and fails anyway.
> 5. Even if killed, the monitoring process leaks anyway due to ISPN-12367.
> 6. The stop command doesn't check for client and server errors on the REST call.
> The fix is to assign random UUID to a server process and filter properly; use process API correctly.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
3 years, 6 months