[
https://issues.redhat.com/browse/WFLY-13897?page=com.atlassian.jira.plugi...
]
Radoslav Husar edited comment on WFLY-13897 at 9/29/20 6:21 AM:
----------------------------------------------------------------
I can reproduce the problem locally with e.g.
{{./integration-tests.sh clean install -Dts.noSmoke -Dts.clustering
-P="-ts.clustering.cluster.ha.profile,-ts.clustering.cluster.fullha.profile,-ts.clustering.byteman.profile,-ts.clustering.single.profile"}}
even with driver version {{12.0.0.Dev04}}.
{noformat}
Every 2.0s: jps -l | sort
ribera.local: Tue Sep 29 12:20:18 2020
1722
51882 org.infinispan.server.loader.Loader
52289 org.infinispan.server.loader.Loader
52995 org.infinispan.server.loader.Loader
53322 org.infinispan.server.loader.Loader
53966 org.infinispan.server.loader.Loader
54582 org.infinispan.server.loader.Loader
54979 org.infinispan.server.loader.Loader
55290 org.infinispan.server.loader.Loader
55916 org.infinispan.server.loader.Loader
58013 sun.tools.jps.Jps
{noformat}
jstack form the server looks like
{noformat}
[rhusar@ribera wildfly]$ jstack 54582
2020-09-29 12:21:04
Full thread dump OpenJDK 64-Bit Server VM (25.265-b01 mixed mode):
"Attach Listener" #181 daemon prio=9 os_prio=31 tid=0x00007fae92008800
nid=0x17507 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"DestroyJavaVM" #180 prio=5 os_prio=31 tid=0x00007fae8c80c800 nid=0x2903 waiting
on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"SINGLE_PORT-ServerMaster-2-1" #173 prio=5 os_prio=31 tid=0x00007fae922ab000
nid=0x17603 runnable [0x000070000f418000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000007bedc3a70> (a io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000007bedc3a88> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007bedc3a20> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:803)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
"RxCachedWorkerPoolEvictor-1" #15 daemon prio=5 os_prio=31
tid=0x00007fae964ba000 nid=0x5903 waiting on condition [0x0000700005541000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007a163dce0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"RxSchedulerPurge-1" #14 daemon prio=5 os_prio=31 tid=0x00007fae8c7cf000
nid=0x5703 waiting on condition [0x000070000543e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007a163e350> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Log4j2-TF-6-Scheduled-1" #12 daemon prio=5 os_prio=31 tid=0x00007fae8c220800
nid=0xa803 waiting on condition [0x000070000533b000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007a10bdb08> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Service Thread" #9 daemon prio=9 os_prio=31 tid=0x00007fae8c06e800 nid=0x4103
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread3" #8 daemon prio=9 os_prio=31 tid=0x00007fae8c053800
nid=0x3f03 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread2" #7 daemon prio=9 os_prio=31 tid=0x00007fae8f00e800
nid=0x4203 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #6 daemon prio=9 os_prio=31 tid=0x00007fae8c04a800
nid=0x3d03 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=31 tid=0x00007fae8f84b800
nid=0x3c03 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=31 tid=0x00007fae8f80d000
nid=0x4503 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007fae8c82b000 nid=0x3203 in
Object.wait() [0x0000700004a1d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007a089d328> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
- locked <0x00000007a089d328> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
"Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007fae8c828000
nid=0x4c03 in Object.wait() [0x000070000491a000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007a089d368> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000007a089d368> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"VM Thread" os_prio=31 tid=0x00007fae8c821800 nid=0x4e03 runnable
"GC task thread#0 (ParallelGC)" os_prio=31 tid=0x00007fae8c810800 nid=0x1f07
runnable
"GC task thread#1 (ParallelGC)" os_prio=31 tid=0x00007fae8c81c000 nid=0x1e03
runnable
"GC task thread#2 (ParallelGC)" os_prio=31 tid=0x00007fae8c81c800 nid=0x5403
runnable
"GC task thread#3 (ParallelGC)" os_prio=31 tid=0x00007fae8c81d800 nid=0x5203
runnable
"GC task thread#4 (ParallelGC)" os_prio=31 tid=0x00007fae8c81e000 nid=0x2b03
runnable
"GC task thread#5 (ParallelGC)" os_prio=31 tid=0x00007fae8c00d000 nid=0x2d03
runnable
"GC task thread#6 (ParallelGC)" os_prio=31 tid=0x00007fae8c00e000 nid=0x5003
runnable
"GC task thread#7 (ParallelGC)" os_prio=31 tid=0x00007fae8c00e800 nid=0x3003
runnable
"VM Periodic Task Thread" os_prio=31 tid=0x00007fae8c06f000 nid=0x5603 waiting
on condition
JNI global references: 2019
{noformat}
but the logs are overwritten.
was (Author: rhusar):
I can reproduce the problem locally with e.g.
{{./integration-tests.sh clean install -Dts.noSmoke -Dts.clustering
-P="-ts.clustering.cluster.ha.profile,-ts.clustering.cluster.fullha.profile,-ts.clustering.byteman.profile,-ts.clustering.single.profile"}}
even with driver version {{12.0.0.Dev04}}.
infinispan-server instances provisioned by testsuite never shutdown
-------------------------------------------------------------------
Key: WFLY-13897
URL:
https://issues.redhat.com/browse/WFLY-13897
Project: WildFly
Issue Type: Bug
Components: Clustering, Test Suite
Affects Versions: 21.0.0.Beta1
Reporter: Paul Ferraro
Assignee: Radoslav Husar
Priority: Critical
Running the clustering testsuite locally leaves 9 instances of infinispan-server
running. This wreaks havoc on the CI, which will remain running until the VM shuts down.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)