[JBoss JIRA] (ISPN-10180) Automatically take a site offline if the Cross-Site is in ASYNC mode
by Pedro Ruivo (Jira)
[ https://issues.jboss.org/browse/ISPN-10180?page=com.atlassian.jira.plugin... ]
Work on ISPN-10180 started by Pedro Ruivo.
------------------------------------------
> Automatically take a site offline if the Cross-Site is in ASYNC mode
> --------------------------------------------------------------------
>
> Key: ISPN-10180
> URL: https://issues.jboss.org/browse/ISPN-10180
> Project: Infinispan
> Issue Type: Feature Request
> Reporter: Pedro Ruivo
> Assignee: Pedro Ruivo
> Priority: Major
>
> With a SYNC XSite replication, it is possible to automatically mark a site offline with the following configuration
> {code:xml}
> <distributed-cache name="XSiteCache" ....>
> ...
> <backups>
> <backup site="site02" failure-policy="IGNORE" strategy="SYNC">
> <take-offline after-failures="10" min-wait="60000"/>
> </backup>
> </backups>
> </distributed-cache>
> {code}
> If the strategy is ASYNC (i.e. for performance reason) the element take-offline will have no effect as the site will 'fire and forget' the updates and don't care about failure.
> It should be possible to use the take-offline configuration in ASYNC mode as well
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (ISPN-10180) Automatically take a site offline if the Cross-Site is in ASYNC mode
by Pedro Ruivo (Jira)
Pedro Ruivo created ISPN-10180:
----------------------------------
Summary: Automatically take a site offline if the Cross-Site is in ASYNC mode
Key: ISPN-10180
URL: https://issues.jboss.org/browse/ISPN-10180
Project: Infinispan
Issue Type: Feature Request
Reporter: Pedro Ruivo
Assignee: Pedro Ruivo
With a SYNC XSite replication, it is possible to automatically mark a site offline with the following configuration
{code:xml}
<distributed-cache name="XSiteCache" ....>
...
<backups>
<backup site="site02" failure-policy="IGNORE" strategy="SYNC">
<take-offline after-failures="10" min-wait="60000"/>
</backup>
</backups>
</distributed-cache>
{code}
If the strategy is ASYNC (i.e. for performance reason) the element take-offline will have no effect as the site will 'fire and forget' the updates and don't care about failure.
It should be possible to use the take-offline configuration in ASYNC mode as well
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (ISPN-10180) Automatically take a site offline if the Cross-Site is in ASYNC mode
by Pedro Ruivo (Jira)
[ https://issues.jboss.org/browse/ISPN-10180?page=com.atlassian.jira.plugin... ]
Pedro Ruivo updated ISPN-10180:
-------------------------------
Status: Open (was: New)
> Automatically take a site offline if the Cross-Site is in ASYNC mode
> --------------------------------------------------------------------
>
> Key: ISPN-10180
> URL: https://issues.jboss.org/browse/ISPN-10180
> Project: Infinispan
> Issue Type: Feature Request
> Reporter: Pedro Ruivo
> Assignee: Pedro Ruivo
> Priority: Major
>
> With a SYNC XSite replication, it is possible to automatically mark a site offline with the following configuration
> {code:xml}
> <distributed-cache name="XSiteCache" ....>
> ...
> <backups>
> <backup site="site02" failure-policy="IGNORE" strategy="SYNC">
> <take-offline after-failures="10" min-wait="60000"/>
> </backup>
> </backups>
> </distributed-cache>
> {code}
> If the strategy is ASYNC (i.e. for performance reason) the element take-offline will have no effect as the site will 'fire and forget' the updates and don't care about failure.
> It should be possible to use the take-offline configuration in ASYNC mode as well
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (ISPN-10179) Long test killer doesn't kill the JVM on Windows
by Tristan Tarrant (Jira)
Tristan Tarrant created ISPN-10179:
--------------------------------------
Summary: Long test killer doesn't kill the JVM on Windows
Key: ISPN-10179
URL: https://issues.jboss.org/browse/ISPN-10179
Project: Infinispan
Issue Type: Bug
Components: Test Suite - Core
Affects Versions: 10.0.0.Beta3, 9.4.13.Final
Reporter: Dan Berindei
Assignee: Dan Berindei
Neither {{jstack}} nor {{taskkill}} work:
{noformat}
[ERROR] Test org.infinispan.client.hotrod.impl.iteration.ProtobufRemoteIteratorIndexingTest.testFilteredIterationWithQuery has been running for more than 300 seconds. Interrupting the test thread and dumping threads of the test suite process and its children.
Cannot find jstack in T:\opt\windows\x86_64\openjdk-1.8.0\jre, programmatically dumping thread stacks of testsuite process to C:\home\jenkins\workspace\jdg-7.3.x-jdg-func-ispn-testsuite-win-openjdk\b521375b\infinispan\client\hotrod-client\threaddump-org_infinispan_client_hotrod_impl_iteration_ProtobufRemoteIteratorIndexingTest_testFilteredIterationWithQuery-2019-05-10.log
Interrupted thread testng-ProtobufRemoteIteratorIndexingTest (57).
Killed processes 9904
[ERROR] Test org.infinispan.client.hotrod.impl.iteration.ReplFailOverRemoteIteratorTest.testFailOver has been running for more than 300 seconds. Interrupting the test thread and dumping threads of the test suite process and its children.
Cannot find jstack in T:\opt\windows\x86_64\openjdk-1.8.0\jre, programmatically dumping thread stacks of testsuite process to C:\home\jenkins\workspace\jdg-7.3.x-jdg-func-ispn-testsuite-win-openjdk\b521375b\infinispan\client\hotrod-client\threaddump-org_infinispan_client_hotrod_impl_iteration_ReplFailOverRemoteIteratorTest_testFailOver-2019-05-10.log
Interrupted thread testng-ReplFailOverRemoteIteratorTest (62).
Killed processes 9904
{noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (ISPN-10178) TCP_NIO2 deadlock during stop on Windows
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-10178?page=com.atlassian.jira.plugin... ]
Dan Berindei updated ISPN-10178:
--------------------------------
Attachment: CustomFailurePolicyTxTest.txt
> TCP_NIO2 deadlock during stop on Windows
> ----------------------------------------
>
> Key: ISPN-10178
> URL: https://issues.jboss.org/browse/ISPN-10178
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 9.4.13.Final
> Environment: Windows
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Fix For: 10.0.0.Beta4, 9.4.14.Final
>
> Attachments: CustomFailurePolicyTxTest.txt
>
>
> {noformat}
> "testng-CustomFailurePolicyTxTest" #60 prio=0 tid=0x3c nid=NA blocked
> java.lang.Thread.State: BLOCKED
> at java.nio.channels.spi.AbstractSelectableChannel.removeKey(AbstractSelectableChannel.java:130)
> - waiting to lock <0x7ae7c881> (a java.lang.Object)
> at java.nio.channels.spi.AbstractSelector.deregister(AbstractSelector.java:188)
> at sun.nio.ch.WindowsSelectorImpl.implClose(WindowsSelectorImpl.java:514)
> - locked <0x6cc2bf11> (a java.lang.Object)
> at sun.nio.ch.SelectorImpl.implCloseSelector(SelectorImpl.java:113)
> - locked <0x2ab0aa2d> (a sun.nio.ch.Util$3)
> - locked <0x2f9dc993> (a java.util.Collections$UnmodifiableSet)
> - locked <0xfeca69a> (a sun.nio.ch.WindowsSelectorImpl)
> at java.nio.channels.spi.AbstractSelector.close(AbstractSelector.java:111)
> at org.jgroups.util.Util.close(Util.java:428)
> at org.jgroups.util.Util.close(Util.java:439)
> at org.jgroups.blocks.cs.NioServer.stop(NioServer.java:127)
> - locked <0x21bca9ea> (a org.jgroups.blocks.cs.NioServer)
> at org.jgroups.protocols.TCP_NIO2.handleDisconnect(TCP_NIO2.java:136)
> at org.jgroups.protocols.TP.handleDownEvent(TP.java:1607)
> at org.jgroups.protocols.BasicTCP.handleDownEvent(BasicTCP.java:129)
> at org.jgroups.protocols.TP.down(TP.java:1082)
> at org.jgroups.protocols.Discovery.down(Discovery.java:425)
> at org.infinispan.test.fwk.TEST_PING.down(TEST_PING.java:208)
> at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:379)
> at org.jgroups.protocols.FD_ALL.down(FD_ALL.java:236)
> at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:101)
> at org.jgroups.protocols.pbcast.NAKACK2.down(NAKACK2.java:553)
> at org.jgroups.protocols.UNICAST3.down(UNICAST3.java:570)
> at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:346)
> at org.jgroups.protocols.pbcast.GMS.down(GMS.java:1076)
> at org.jgroups.protocols.tom.TOA.down(TOA.java:80)
> at org.jgroups.protocols.FlowControl.down(FlowControl.java:296)
> at org.jgroups.protocols.RSVP.down(RSVP.java:102)
> at org.jgroups.protocols.relay.RELAY2.down(RELAY2.java:380)
> at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:908)
> at org.jgroups.JChannel.down(JChannel.java:668)
> at org.jgroups.JChannel.disconnect(JChannel.java:469)
> - locked <0xef29e32> (a org.jgroups.JChannel)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.stop(JGroupsTransport.java:712)
> at sun.reflect.GeneratedMethodAccessor148.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> at org.infinispan.commons.util.SecurityActions$$Lambda$154/1125443972.run(Unknown Source)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:181)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.performStop(BasicComponentRegistryImpl.java:601)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.stopWrapper(BasicComponentRegistryImpl.java:590)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.stop(BasicComponentRegistryImpl.java:461)
> at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:441)
> at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:376)
> at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:775)
> at org.infinispan.xsite.AbstractXSiteTest$$Lambda$4294/1094472832.accept(Unknown Source)
> at java.util.ArrayList.forEach(ArrayList.java:1257)
> at org.infinispan.xsite.AbstractXSiteTest.killSite(AbstractXSiteTest.java:89)
> at org.infinispan.xsite.AbstractXSiteTest.killSites(AbstractXSiteTest.java:82)
> at org.infinispan.xsite.AbstractXSiteTest.destroy(AbstractXSiteTest.java:76)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
> at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:59)
> at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:458)
> at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:222)
> at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:142)
> at org.testng.internal.TestMethodWorker.invokeAfterClassMethods(TestMethodWorker.java:214)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:648)
> at org.testng.TestRunner.run(TestRunner.java:505)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:40)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:489)
> at org.testng.internal.thread.ThreadUtil$1.call(ThreadUtil.java:52)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x9411fa8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> - <0x4fe21f2e> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> "NioServer.Selector [/127.0.0.1:7925]-3,CustomFailurePolicyTxTest-NodeC-12186" #205597 prio=0 tid=0x3231d nid=NA blocked
> java.lang.Thread.State: BLOCKED
> at sun.nio.ch.WindowsSelectorImpl.putEventOps(WindowsSelectorImpl.java:593)
> - waiting to lock <0x6cc2bf11> (a java.lang.Object)
> at sun.nio.ch.SocketChannelImpl.translateAndSetInterestOps(SocketChannelImpl.java:974)
> at sun.nio.ch.SelectionKeyImpl.nioInterestOps(SelectionKeyImpl.java:105)
> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:83)
> at sun.nio.ch.SelectorImpl.register(SelectorImpl.java:134)
> at java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:212)
> - locked <0x7ae7c881> (a java.lang.Object)
> - locked <0x6a684d8f> (a java.lang.Object)
> at org.jgroups.blocks.cs.NioServer.handleAccept(NioServer.java:93)
> at org.jgroups.blocks.cs.NioBaseServer$Acceptor.run(NioBaseServer.java:138)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (ISPN-10093) PersistenceManagerImpl stop deadlock with topology update
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-10093?page=com.atlassian.jira.plugin... ]
Dan Berindei updated ISPN-10093:
--------------------------------
Comment: was deleted
(was: CustomFailurePolicyTxTest
{noformat}
[ERROR] Test org.infinispan.xsite.backupfailure.tx.CustomFailurePolicyTxTest.destroy has been running for more than 300 seconds. Interrupting the test thread and dumping threads of the test suite process and its children.
Cannot find jstack in T:\opt\windows\x86_64\openjdk-1.8.0\jre, programmatically dumping thread stacks of testsuite process to C:\home\jenkins\workspace\jdg-7.3.x-jdg-func-ispn-testsuite-win-openjdk\a25bd407\infinispan\core\threaddump-org_infinispan_xsite_backupfailure_tx_CustomFailurePolicyTxTest_destroy-2019-05-10.log
Interrupted thread testng-CustomFailurePolicyTxTest (60).
Killed processes 3536
Build was aborted
Aborted by Diego
{noformat}
{noformat}
2019-05-10 14:53:21
Test thread dump:
"VERIFY_SUSPECT.TimerThread-190,CustomFailurePolicyTxTest-NodeC-12186" #230484 prio=0 tid=0x38454 nid=NA timed_waiting
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x859f0d1> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.DelayQueue.poll(DelayQueue.java:273)
at org.jgroups.protocols.VERIFY_SUSPECT.run(VERIFY_SUSPECT.java:181)
at java.lang.Thread.run(Thread.java:748)
"jgroups-98,bridge-org.infinispan.xsite.backupfailure.tx.CustomFailurePolicyTxTest[null, tx=false, 2PC],_CustomFailurePolicyTxTest-NodeC-12186:NYC" #230475 prio=0 tid=0x3844b nid=NA timed_waiting
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x4d5b7fbe> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"jgroups-22,bridge-org.infinispan.xsite.backupfailure.tx.CustomFailurePolicyTxTest[null, tx=false, 2PC],_CustomFailurePolicyTxTest-NodeD-8124:NYC" #230467 prio=0 tid=0x38443 nid=NA timed_waiting
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x43a99927> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"jgroups-21,bridge-org.infinispan.xsite.backupfailure.tx.CustomFailurePolicyTxTest[null, tx=false, 2PC],_CustomFailurePolicyTxTest-NodeD-8124:NYC" #230458 prio=0 tid=0x3843a nid=NA timed_waiting
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x43a99927> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"jgroups-95,bridge-org.infinispan.xsite.backupfailure.tx.CustomFailurePolicyTxTest[null, tx=false, 2PC],_CustomFailurePolicyTxTest-NodeC-12186:NYC" #230386 prio=0 tid=0x383f2 nid=NA timed_waiting
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x4d5b7fbe> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"FD_SOCK conn-handler-18,bridge-org.infinispan.xsite.backupfailure.tx.CustomFailurePolicyTxTest[null, tx=false, 2PC],_CustomFailurePolicyTxTest-NodeC-12186:NYC" #206544 prio=0 tid=0x326d0 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.net.SocketInputStream.read(SocketInputStream.java:224)
at org.jgroups.protocols.FD_SOCK$ClientConnectionHandler.run(FD_SOCK.java:1052)
at java.lang.Thread.run(Thread.java:748)
"FD_SOCK conn-handler-10,bridge-org.infinispan.xsite.backupfailure.tx.CustomFailurePolicyTxTest[null, tx=false, 2PC],_CustomFailurePolicyTxTest-NodeD-8124:NYC" #206541 prio=0 tid=0x326cd nid=NA runnable
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.net.SocketInputStream.read(SocketInputStream.java:224)
at org.jgroups.protocols.FD_SOCK$ClientConnectionHandler.run(FD_SOCK.java:1052)
at java.lang.Thread.run(Thread.java:748)
"FD_SOCK pinger-9,bridge-org.infinispan.xsite.backupfailure.tx.CustomFailurePolicyTxTest[null, tx=false, 2PC],_CustomFailurePolicyTxTest-NodeD-8124:NYC" #206539 prio=0 tid=0x326cb nid=NA runnable
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.net.SocketInputStream.read(SocketInputStream.java:224)
at org.jgroups.protocols.FD_SOCK.run(FD_SOCK.java:431)
at java.lang.Thread.run(Thread.java:748)
"FD_SOCK pinger-17,bridge-org.infinispan.xsite.backupfailure.tx.CustomFailurePolicyTxTest[null, tx=false, 2PC],_CustomFailurePolicyTxTest-NodeC-12186:NYC" #206537 prio=0 tid=0x326c9 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.net.SocketInputStream.read(SocketInputStream.java:224)
at org.jgroups.protocols.FD_SOCK.run(FD_SOCK.java:431)
at java.lang.Thread.run(Thread.java:748)
"Connection.Receiver [127.0.0.1:7203 - 127.0.0.1:58671]-15,bridge-org.infinispan.xsite.backupfailure.tx.CustomFailurePolicyTxTest[null, tx=false, 2PC],_CustomFailurePolicyTxTest-NodeC-12186:NYC" #206532 prio=0 tid=0x326c4 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
- locked <0x7c5c168a> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:294)
at java.lang.Thread.run(Thread.java:748)
"Connection.Receiver [127.0.0.1:58671 - 127.0.0.1:7203]-7,bridge-org.infinispan.xsite.backupfailure.tx.CustomFailurePolicyTxTest[null, tx=false, 2PC],_CustomFailurePolicyTxTest-NodeD-8124:NYC" #206531 prio=0 tid=0x326c3 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
- locked <0x3b44aea8> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:294)
at java.lang.Thread.run(Thread.java:748)
"FD_SOCK acceptor-6,bridge-org.infinispan.xsite.backupfailure.tx.CustomFailurePolicyTxTest[null, tx=false, 2PC],_CustomFailurePolicyTxTest-NodeD-8124:NYC" #206530 prio=0 tid=0x326c2 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at java.net.TwoStacksPlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
- locked <0x2d7480f1> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at org.jgroups.protocols.FD_SOCK$ServerSocketHandler.run(FD_SOCK.java:990)
at java.lang.Thread.run(Thread.java:748)
"TcpServer.Acceptor[7204]-5,bridge-org.infinispan.xsite.backupfailure.tx.CustomFailurePolicyTxTest[null, tx=false, 2PC],_CustomFailurePolicyTxTest-NodeD-8124:NYC" #206529 prio=0 tid=0x326c1 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at java.net.TwoStacksPlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
- locked <0x229b54d9> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at org.jgroups.blocks.cs.TcpServer$Acceptor.run(TcpServer.java:110)
at java.lang.Thread.run(Thread.java:748)
"Timer runner-1,bridge-org.infinispan.xsite.backupfailure.tx.CustomFailurePolicyTxTest[null, tx=false, 2PC],_CustomFailurePolicyTxTest-NodeD-8124:NYC" #206522 prio=0 tid=0x326ba nid=NA timed_waiting
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x3e6ffe46> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:223)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)
at org.jgroups.util.TimeScheduler3.run(TimeScheduler3.java:181)
at java.lang.Thread.run(Thread.java:748)
"FD_SOCK conn-handler-14,bridge-org.infinispan.xsite.backupfailure.tx.CustomFailurePolicyTxTest[null, tx=false, 2PC],_CustomFailurePolicyTxTest-NodeC-12186:NYC" #206277 prio=0 tid=0x325c5 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.net.SocketInputStream.read(SocketInputStream.java:224)
at org.jgroups.protocols.FD_SOCK$ClientConnectionHandler.run(FD_SOCK.java:1052)
at java.lang.Thread.run(Thread.java:748)
"FD_SOCK pinger-13,bridge-org.infinispan.xsite.backupfailure.tx.CustomFailurePolicyTxTest[null, tx=false, 2PC],_CustomFailurePolicyTxTest-NodeB-52489:LON" #206265 prio=0 tid=0x325b9 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.net.SocketInputStream.read(SocketInputStream.java:224)
at org.jgroups.protocols.FD_SOCK.run(FD_SOCK.java:431)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-CustomFailurePolicyTxTest-NodeD-p66444-t6" #205848 prio=0 tid=0x32418 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x204026cd> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-CustomFailurePolicyTxTest-NodeD-p66444-t5" #205843 prio=0 tid=0x32413 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x204026cd> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-CustomFailurePolicyTxTest-NodeD-p66444-t4" #205842 prio=0 tid=0x32412 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x204026cd> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-CustomFailurePolicyTxTest-NodeD-p66444-t3" #205835 prio=0 tid=0x3240b nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x204026cd> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"FD_SOCK conn-handler-12,CustomFailurePolicyTxTest-NodeD-8124" #205811 prio=0 tid=0x323f3 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.net.SocketInputStream.read(SocketInputStream.java:224)
at org.jgroups.protocols.FD_SOCK$ClientConnectionHandler.run(FD_SOCK.java:1052)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-CustomFailurePolicyTxTest-NodeD-p66444-t2" #205810 prio=0 tid=0x323f2 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x204026cd> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"expiration-thread-CustomFailurePolicyTxTest-NodeD-p66469-t1" #205809 prio=0 tid=0x323f1 nid=NA timed_waiting
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x293a593c> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"jgroups-12,CustomFailurePolicyTxTest-NodeC-12186" #205807 prio=0 tid=0x323ef nid=NA timed_waiting
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x51fffd5e> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"timeout-thread-CustomFailurePolicyTxTest-NodeD-p66443-t1" #205803 prio=0 tid=0x323eb nid=NA timed_waiting
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1023b4e9> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-CustomFailurePolicyTxTest-NodeD-p66444-t1" #205802 prio=0 tid=0x323ea nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x204026cd> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Controller-remote-thread-CustomFailurePolicyTxTest-NodeD" #205798 prio=0 tid=0x323e6 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x31749e6e> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl$ControllerThread.run(BlockingTaskAwareExecutorServiceImpl.java:159)
"jgroups-9,CustomFailurePolicyTxTest-NodeD-8124" #205795 prio=0 tid=0x323e3 nid=NA timed_waiting
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x4a7cc2c5> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"jgroups-7,CustomFailurePolicyTxTest-NodeD-8124" #205793 prio=0 tid=0x323e1 nid=NA timed_waiting
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x4a7cc2c5> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"FD_SOCK pinger-7,CustomFailurePolicyTxTest-NodeC-12186" #205789 prio=0 tid=0x323dd nid=NA runnable
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.net.SocketInputStream.read(SocketInputStream.java:224)
at org.jgroups.protocols.FD_SOCK.run(FD_SOCK.java:431)
at java.lang.Thread.run(Thread.java:748)
"NioServer.Selector [/127.0.0.1:7926]-3,CustomFailurePolicyTxTest-NodeD-8124" #205771 prio=0 tid=0x323cb nid=NA runnable
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x6447d3d9> (a sun.nio.ch.Util$3)
- locked <0x694ea744> (a java.util.Collections$UnmodifiableSet)
- locked <0x7cdcd3ed> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.jgroups.blocks.cs.NioBaseServer$Acceptor.doSelect(NioBaseServer.java:157)
at org.jgroups.blocks.cs.NioBaseServer$Acceptor.run(NioBaseServer.java:118)
at java.lang.Thread.run(Thread.java:748)
"TOA-Delivery-Thread-CustomFailurePolicyTxTest-NodeD-8124" #205760 prio=0 tid=0x323c0 nid=NA waiting
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <0x501a4c59> (a java.util.TreeSet)
at java.lang.Object.wait(Object.java:502)
at org.jgroups.protocols.tom.DeliveryManagerImpl.getNextMessagesToDeliver(DeliveryManagerImpl.java:180)
at org.jgroups.protocols.tom.DeliveryThread.run(DeliveryThread.java:52)
"FD_SOCK acceptor-2,CustomFailurePolicyTxTest-NodeD-8124" #205766 prio=0 tid=0x323c6 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at java.net.TwoStacksPlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
- locked <0xa998498> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at org.jgroups.protocols.FD_SOCK$ServerSocketHandler.run(FD_SOCK.java:990)
at java.lang.Thread.run(Thread.java:748)
"Timer runner-1,CustomFailurePolicyTxTest-NodeD-8124" #205765 prio=0 tid=0x323c5 nid=NA timed_waiting
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x16717608> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:223)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)
at org.jgroups.util.TimeScheduler3.run(TimeScheduler3.java:181)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-CustomFailurePolicyTxTest-NodeC-p66404-t6" #205735 prio=0 tid=0x323a7 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xc289c90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-CustomFailurePolicyTxTest-NodeC-p66404-t5" #205731 prio=0 tid=0x323a3 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xc289c90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-CustomFailurePolicyTxTest-NodeC-p66404-t4" #205730 prio=0 tid=0x323a2 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xc289c90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"timeout-thread-CustomFailurePolicyTxTest-NodeC-p66403-t1" #205729 prio=0 tid=0x323a1 nid=NA timed_waiting
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xed271c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-CustomFailurePolicyTxTest-NodeC-p66404-t3" #205704 prio=0 tid=0x32388 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xc289c90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-CustomFailurePolicyTxTest-NodeC-p66404-t2" #205702 prio=0 tid=0x32386 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xc289c90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"expiration-thread-CustomFailurePolicyTxTest-NodeC-p66435-t1" #205700 prio=0 tid=0x32384 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x48a989ce> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-CustomFailurePolicyTxTest-NodeC-p66404-t1" #205679 prio=0 tid=0x3236f nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xc289c90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Controller-remote-thread-CustomFailurePolicyTxTest-NodeC" #205675 prio=0 tid=0x3236b nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x557567e5> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl$ControllerThread.run(BlockingTaskAwareExecutorServiceImpl.java:159)
"TcpServer.Acceptor[7203]-5,bridge-org.infinispan.xsite.backupfailure.tx.CustomFailurePolicyTxTest[null, tx=false, 2PC],_CustomFailurePolicyTxTest-NodeC-12186:NYC" #205653 prio=0 tid=0x32355 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at java.net.TwoStacksPlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
- locked <0x1d71adc7> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at org.jgroups.blocks.cs.TcpServer$Acceptor.run(TcpServer.java:110)
at java.lang.Thread.run(Thread.java:748)
"FD_SOCK acceptor-2,bridge-org.infinispan.xsite.backupfailure.tx.CustomFailurePolicyTxTest[null, tx=false, 2PC],_CustomFailurePolicyTxTest-NodeC-12186:NYC" #205649 prio=0 tid=0x32351 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at java.net.TwoStacksPlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
- locked <0xb0c390> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at org.jgroups.protocols.FD_SOCK$ServerSocketHandler.run(FD_SOCK.java:990)
at java.lang.Thread.run(Thread.java:748)
"Timer runner-1,bridge-org.infinispan.xsite.backupfailure.tx.CustomFailurePolicyTxTest[null, tx=false, 2PC],_CustomFailurePolicyTxTest-NodeC-12186:NYC" #205648 prio=0 tid=0x32350 nid=NA timed_waiting
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x4d3cdb1> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:223)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)
at org.jgroups.util.TimeScheduler3.run(TimeScheduler3.java:181)
at java.lang.Thread.run(Thread.java:748)
"NioServer.Selector [/127.0.0.1:7925]-3,CustomFailurePolicyTxTest-NodeC-12186" #205597 prio=0 tid=0x3231d nid=NA blocked
java.lang.Thread.State: BLOCKED
at sun.nio.ch.WindowsSelectorImpl.putEventOps(WindowsSelectorImpl.java:593)
- waiting to lock <0x6cc2bf11> (a java.lang.Object)
at sun.nio.ch.SocketChannelImpl.translateAndSetInterestOps(SocketChannelImpl.java:974)
at sun.nio.ch.SelectionKeyImpl.nioInterestOps(SelectionKeyImpl.java:105)
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:83)
at sun.nio.ch.SelectorImpl.register(SelectorImpl.java:134)
at java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:212)
- locked <0x7ae7c881> (a java.lang.Object)
- locked <0x6a684d8f> (a java.lang.Object)
at org.jgroups.blocks.cs.NioServer.handleAccept(NioServer.java:93)
at org.jgroups.blocks.cs.NioBaseServer$Acceptor.run(NioBaseServer.java:138)
at java.lang.Thread.run(Thread.java:748)
"TOA-Delivery-Thread-CustomFailurePolicyTxTest-NodeC-12186" #205578 prio=0 tid=0x3230a nid=NA waiting
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <0x76e6597d> (a java.util.TreeSet)
at java.lang.Object.wait(Object.java:502)
at org.jgroups.protocols.tom.DeliveryManagerImpl.getNextMessagesToDeliver(DeliveryManagerImpl.java:180)
at org.jgroups.protocols.tom.DeliveryThread.run(DeliveryThread.java:52)
"Timer runner-1,CustomFailurePolicyTxTest-NodeC-12186" #205591 prio=0 tid=0x32317 nid=NA timed_waiting
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x320134e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:223)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)
at org.jgroups.util.TimeScheduler3.run(TimeScheduler3.java:181)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-ForkChannelRestartTest-NodeD-p49572-t20" #161095 prio=0 tid=0x27547 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1aad031a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-ForkChannelRestartTest-NodeD-p49572-t19" #161094 prio=0 tid=0x27546 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1aad031a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-ForkChannelRestartTest-NodeD-p49572-t18" #161093 prio=0 tid=0x27545 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1aad031a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-ForkChannelRestartTest-NodeD-p49572-t17" #161091 prio=0 tid=0x27543 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1aad031a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-ForkChannelRestartTest-NodeD-p49572-t16" #161088 prio=0 tid=0x27540 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1aad031a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-ForkChannelRestartTest-NodeD-p49572-t15" #161080 prio=0 tid=0x27538 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1aad031a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-ForkChannelRestartTest-NodeD-p49572-t14" #161077 prio=0 tid=0x27535 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1aad031a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-ForkChannelRestartTest-NodeD-p49572-t13" #161076 prio=0 tid=0x27534 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1aad031a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-ForkChannelRestartTest-NodeD-p49572-t12" #161072 prio=0 tid=0x27530 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1aad031a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"remote-thread-ForkChannelRestartTest-NodeD-p49570-t3" #161071 prio=0 tid=0x2752f nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x636e1d69> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-ForkChannelRestartTest-NodeD-p49572-t11" #161067 prio=0 tid=0x2752b nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1aad031a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-ForkChannelRestartTest-NodeD-p49572-t10" #156620 prio=0 tid=0x263cc nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1aad031a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-ForkChannelRestartTest-NodeD-p49572-t9" #156616 prio=0 tid=0x263c8 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1aad031a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-ForkChannelRestartTest-NodeD-p49572-t8" #156613 prio=0 tid=0x263c5 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1aad031a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-ForkChannelRestartTest-NodeD-p49572-t7" #156587 prio=0 tid=0x263ab nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1aad031a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-ForkChannelRestartTest-NodeD-p49572-t6" #156585 prio=0 tid=0x263a9 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1aad031a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-ForkChannelRestartTest-NodeD-p49572-t5" #156556 prio=0 tid=0x2638c nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1aad031a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-ForkChannelRestartTest-NodeD-p49572-t4" #156553 prio=0 tid=0x26389 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1aad031a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"stateTransferExecutor-thread-ForkChannelRestartTest-NodeD-p49573-t6" #154378 prio=0 tid=0x25b0a nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x27078dc8> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-ForkChannelRestartTest-NodeD-p49572-t3" #154372 prio=0 tid=0x25b04 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1aad031a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-ForkChannelRestartTest-NodeD-p49572-t2" #154370 prio=0 tid=0x25b02 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1aad031a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"expiration-thread-ForkChannelRestartTest-NodeD-p49574-t1" #154369 prio=0 tid=0x25b01 nid=NA timed_waiting
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1c9a1bfc> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"timeout-thread-ForkChannelRestartTest-NodeD-p49571-t1" #154368 prio=0 tid=0x25b00 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x6edd7ea6> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"transport-thread-ForkChannelRestartTest-NodeD-p49572-t1" #154367 prio=0 tid=0x25aff nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1aad031a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Controller-remote-thread-ForkChannelRestartTest-NodeD" #154364 prio=0 tid=0x25afc nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x7d4bb341> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl$ControllerThread.run(BlockingTaskAwareExecutorServiceImpl.java:159)
"RxCachedWorkerPoolEvictor-1" #23485 prio=0 tid=0x5bbd nid=NA timed_waiting
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x51924c50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"RxSchedulerPurge-1" #23482 prio=0 tid=0x5bba nid=NA timed_waiting
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x30f15c73> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Transaction Reaper Worker 0" #130 prio=0 tid=0x82 nid=NA waiting
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <0x25bd5395> (a java.util.LinkedList)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.arjuna.coordinator.TransactionReaper.waitForCancellations(TransactionReaper.java:328)
at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:65)
"Transaction Reaper" #129 prio=0 tid=0x81 nid=NA timed_waiting
java.lang.Thread.State: TIMED_WAITING
at java.lang.Object.wait(Native Method)
- waiting on <0x3b265b78> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
at com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThread.java:90)
"testng-RetryMechanismTest" #63 prio=0 tid=0x3f nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x663908ed> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"testng-DistSyncOnePhaseTxStateTransferTest" #62 prio=0 tid=0x3e nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x663908ed> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"testng-DistSyncNonTxStateTransferTest" #61 prio=0 tid=0x3d nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x663908ed> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"testng-CustomFailurePolicyTxTest" #60 prio=0 tid=0x3c nid=NA blocked
java.lang.Thread.State: BLOCKED
at java.nio.channels.spi.AbstractSelectableChannel.removeKey(AbstractSelectableChannel.java:130)
- waiting to lock <0x7ae7c881> (a java.lang.Object)
at java.nio.channels.spi.AbstractSelector.deregister(AbstractSelector.java:188)
at sun.nio.ch.WindowsSelectorImpl.implClose(WindowsSelectorImpl.java:514)
- locked <0x6cc2bf11> (a java.lang.Object)
at sun.nio.ch.SelectorImpl.implCloseSelector(SelectorImpl.java:113)
- locked <0x2ab0aa2d> (a sun.nio.ch.Util$3)
- locked <0x2f9dc993> (a java.util.Collections$UnmodifiableSet)
- locked <0xfeca69a> (a sun.nio.ch.WindowsSelectorImpl)
at java.nio.channels.spi.AbstractSelector.close(AbstractSelector.java:111)
at org.jgroups.util.Util.close(Util.java:428)
at org.jgroups.util.Util.close(Util.java:439)
at org.jgroups.blocks.cs.NioServer.stop(NioServer.java:127)
- locked <0x21bca9ea> (a org.jgroups.blocks.cs.NioServer)
at org.jgroups.protocols.TCP_NIO2.handleDisconnect(TCP_NIO2.java:136)
at org.jgroups.protocols.TP.handleDownEvent(TP.java:1607)
at org.jgroups.protocols.BasicTCP.handleDownEvent(BasicTCP.java:129)
at org.jgroups.protocols.TP.down(TP.java:1082)
at org.jgroups.protocols.Discovery.down(Discovery.java:425)
at org.infinispan.test.fwk.TEST_PING.down(TEST_PING.java:208)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:379)
at org.jgroups.protocols.FD_ALL.down(FD_ALL.java:236)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:101)
at org.jgroups.protocols.pbcast.NAKACK2.down(NAKACK2.java:553)
at org.jgroups.protocols.UNICAST3.down(UNICAST3.java:570)
at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:346)
at org.jgroups.protocols.pbcast.GMS.down(GMS.java:1076)
at org.jgroups.protocols.tom.TOA.down(TOA.java:80)
at org.jgroups.protocols.FlowControl.down(FlowControl.java:296)
at org.jgroups.protocols.RSVP.down(RSVP.java:102)
at org.jgroups.protocols.relay.RELAY2.down(RELAY2.java:380)
at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:908)
at org.jgroups.JChannel.down(JChannel.java:668)
at org.jgroups.JChannel.disconnect(JChannel.java:469)
- locked <0xef29e32> (a org.jgroups.JChannel)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.stop(JGroupsTransport.java:712)
at sun.reflect.GeneratedMethodAccessor148.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
at org.infinispan.commons.util.SecurityActions$$Lambda$154/1125443972.run(Unknown Source)
at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:181)
at org.infinispan.factories.impl.BasicComponentRegistryImpl.performStop(BasicComponentRegistryImpl.java:601)
at org.infinispan.factories.impl.BasicComponentRegistryImpl.stopWrapper(BasicComponentRegistryImpl.java:590)
at org.infinispan.factories.impl.BasicComponentRegistryImpl.stop(BasicComponentRegistryImpl.java:461)
at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:441)
at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:376)
at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:775)
at org.infinispan.xsite.AbstractXSiteTest$$Lambda$4294/1094472832.accept(Unknown Source)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at org.infinispan.xsite.AbstractXSiteTest.killSite(AbstractXSiteTest.java:89)
at org.infinispan.xsite.AbstractXSiteTest.killSites(AbstractXSiteTest.java:82)
at org.infinispan.xsite.AbstractXSiteTest.destroy(AbstractXSiteTest.java:76)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:59)
at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:458)
at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:222)
at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:142)
at org.testng.internal.TestMethodWorker.invokeAfterClassMethods(TestMethodWorker.java:214)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
at org.testng.TestRunner.privateRun(TestRunner.java:648)
at org.testng.TestRunner.run(TestRunner.java:505)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
at org.testng.SuiteRunner.access$000(SuiteRunner.java:40)
at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:489)
at org.testng.internal.thread.ThreadUtil$1.call(ThreadUtil.java:52)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- <0x9411fa8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
- <0x4fe21f2e> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
"testng-test.fwk.TestNGSuiteChecksTest" #59 prio=0 tid=0x3b nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x663908ed> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"testng-DistSyncOnePhaseWriteSkewTxStateTransferTest" #58 prio=0 tid=0x3a nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x663908ed> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"testng-DistSyncTwoPhasesWriteSkewTxStateTransferTest" #57 prio=0 tid=0x39 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x663908ed> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"testng-DistSyncTwoPhasesTxStateTransferTest" #56 prio=0 tid=0x38 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x663908ed> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"RunningTestsRegistry-Worker" #55 prio=0 tid=0x37 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
at org.infinispan.commons.test.RunningTestsRegistry.dumpThreads(RunningTestsRegistry.java:135)
at org.infinispan.commons.test.RunningTestsRegistry.killLongTest(RunningTestsRegistry.java:61)
at org.infinispan.commons.test.RunningTestsRegistry.lambda$registerThreadWithTest$1(RunningTestsRegistry.java:52)
at org.infinispan.commons.test.RunningTestsRegistry$$Lambda$111/1600736286.run(Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- <0x5b4cd08b> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"SeedGenerator Thread" #12 prio=0 tid=0xc nid=NA waiting
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <0x525a614c> (a sun.security.provider.SeedGenerator$ThreadedSeedGenerator)
at java.lang.Object.wait(Object.java:502)
at sun.security.provider.SeedGenerator$ThreadedSeedGenerator.run(SeedGenerator.java:334)
at java.lang.Thread.run(Thread.java:748)
"surefire-forkedjvm-ping-30s" #11 prio=0 tid=0xb nid=NA timed_waiting
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2e2a82c3> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"surefire-forkedjvm-command-thread" #10 prio=0 tid=0xa nid=NA runnable
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:255)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
- locked <0x25d0a83e> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.maven.surefire.booter.MasterProcessCommand.decode(MasterProcessCommand.java:115)
at org.apache.maven.surefire.booter.CommandReader$CommandRunnable.run(CommandReader.java:390)
at java.lang.Thread.run(Thread.java:748)
"Attach Listener" #6 prio=0 tid=0x6 nid=NA runnable
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #5 prio=0 tid=0x5 nid=NA runnable
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 prio=0 tid=0x3 nid=NA waiting
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <0x260f60a5> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
"Reference Handler" #2 prio=0 tid=0x2 nid=NA waiting
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <0x54b5f9c8> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"main" #1 prio=0 tid=0x1 nid=NA timed_waiting
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x6167a697> (a java.util.concurrent.FutureTask)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:426)
at java.util.concurrent.FutureTask.get(FutureTask.java:204)
at java.util.concurrent.AbstractExecutorService.invokeAll(AbstractExecutorService.java:289)
at org.testng.internal.thread.ThreadUtil.execute(ThreadUtil.java:60)
at org.testng.SuiteRunner.runInParallelTestMode(SuiteRunner.java:474)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:418)
at org.testng.SuiteRunner.run(SuiteRunner.java:364)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
at org.testng.TestNG.runSuites(TestNG.java:1049)
at org.testng.TestNG.run(TestNG.java:1017)
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:135)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeMulti(TestNGDirectoryTestSuite.java:193)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:94)
at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:146)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
{noformat})
> PersistenceManagerImpl stop deadlock with topology update
> ---------------------------------------------------------
>
> Key: ISPN-10093
> URL: https://issues.jboss.org/browse/ISPN-10093
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite - Core
> Affects Versions: 10.0.0.Beta3
> Reporter: Dan Berindei
> Assignee: Will Burns
> Priority: Major
> Labels: testsuite_stability
> Fix For: 10.0.0.Beta4
>
> Attachments: threaddump.txt
>
>
> {{DistSyncStoreNotSharedTest.clearContent}} hanged in CI recently:
> {noformat}
> "testng-DistSyncStoreNotSharedTest" #16 prio=5 os_prio=0 cpu=11511.26ms elapsed=435.14s tid=0x00007fdb710b6000 nid=0x3222 waiting on condition [0x00007fdb352d3000]
> java.lang.Thread.State: WAITING (parking)
> at jdk.internal.misc.Unsafe.park(java.base@11/Native Method)
> - parking to wait for <0x00000000c8a22450> (a java.util.concurrent.Semaphore$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(java.base@11/LockSupport.java:194)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11/AbstractQueuedSynchronizer.java:885)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(java.base@11/AbstractQueuedSynchronizer.java:1009)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(java.base@11/AbstractQueuedSynchronizer.java:1324)
> at java.util.concurrent.Semaphore.acquireUninterruptibly(java.base@11/Semaphore.java:504)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.stop(PersistenceManagerImpl.java:222)
> at jdk.internal.reflect.GeneratedMethodAccessor72.invoke(Unknown Source)
> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11/DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(java.base@11/Method.java:566)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> at org.infinispan.commons.util.SecurityActions$$Lambda$237/0x0000000100661c40.run(Unknown Source)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:181)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.performStop(BasicComponentRegistryImpl.java:601)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.stopWrapper(BasicComponentRegistryImpl.java:590)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.stop(BasicComponentRegistryImpl.java:461)
> at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:431)
> at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:366)
> at org.infinispan.cache.impl.CacheImpl.performImmediateShutdown(CacheImpl.java:1160)
> at org.infinispan.cache.impl.CacheImpl.stop(CacheImpl.java:1125)
> at org.infinispan.cache.impl.AbstractDelegatingCache.stop(AbstractDelegatingCache.java:521)
> at org.infinispan.manager.DefaultCacheManager.terminate(DefaultCacheManager.java:747)
> at org.infinispan.manager.DefaultCacheManager.stopCaches(DefaultCacheManager.java:799)
> at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:775)
> at org.infinispan.test.TestingUtil.killCacheManagers(TestingUtil.java:846)
> at org.infinispan.test.MultipleCacheManagersTest.clearContent(MultipleCacheManagersTest.java:158)
> "persistence-thread-DistSyncStoreNotSharedTest-NodeB-p16432-t1" #53654 daemon prio=5 os_prio=0 cpu=1.26ms elapsed=301.93s tid=0x00007fdb3c3d8000 nid=0x8ef waiting on condition [0x00007fdb00055000]
> java.lang.Thread.State: WAITING (parking)
> at jdk.internal.misc.Unsafe.park(java.base@11/Native Method)
> - parking to wait for <0x00000000c8b1fb88> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(java.base@11/LockSupport.java:194)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11/AbstractQueuedSynchronizer.java:885)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(java.base@11/AbstractQueuedSynchronizer.java:1009)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(java.base@11/AbstractQueuedSynchronizer.java:1324)
> at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(java.base@11/ReentrantReadWriteLock.java:738)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.pollStoreAvailability(PersistenceManagerImpl.java:196)
> at org.infinispan.persistence.manager.PersistenceManagerImpl$$Lambda$492/0x00000001007fb440.run(Unknown Source)
> at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11/Executors.java:515)
> at java.util.concurrent.FutureTask.runAndReset(java.base@11/FutureTask.java:305)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11/ScheduledThreadPoolExecutor.java:305)
> "transport-thread-DistSyncStoreNotSharedTest-NodeB-p16424-t5" #53646 daemon prio=5 os_prio=0 cpu=3.15ms elapsed=301.94s tid=0x00007fdb2007a000 nid=0x8e8 waiting on condition [0x00007fdb0b406000]
> java.lang.Thread.State: WAITING (parking)
> at jdk.internal.misc.Unsafe.park(java.base@11/Native Method)
> - parking to wait for <0x00000000c8d2abb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(java.base@11/LockSupport.java:194)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11/AbstractQueuedSynchronizer.java:2081)
> at io.reactivex.internal.operators.flowable.BlockingFlowableIterable$BlockingFlowableIterator.hasNext(BlockingFlowableIterable.java:94)
> at io.reactivex.Flowable.blockingForEach(Flowable.java:5682)
> at org.infinispan.statetransfer.StateConsumerImpl.removeStaleData(StateConsumerImpl.java:1011)
> at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:453)
> at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:202)
> at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:58)
> at org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:114)
> at org.infinispan.topology.LocalTopologyManagerImpl.resetLocalTopologyBeforeRebalance(LocalTopologyManagerImpl.java:437)
> at org.infinispan.topology.LocalTopologyManagerImpl.doHandleRebalance(LocalTopologyManagerImpl.java:519)
> - locked <0x00000000c8b30b30> (a org.infinispan.topology.LocalCacheStatus)
> at org.infinispan.topology.LocalTopologyManagerImpl.lambda$handleRebalance$3(LocalTopologyManagerImpl.java:484)
> at org.infinispan.topology.LocalTopologyManagerImpl$$Lambda$574/0x000000010089a040.run(Unknown Source)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175){noformat}
> [Full thread dump|https://ci.infinispan.org/job/Infinispan/job/master/1133/artifact/core/]
> Somehow the producer thread for the transport-thread iteration is blocked, but without waiting for the persistence mutex. Maybe it's waiting for a topology? Not sure if it's relevant, but the last test to run was {{testClearWithFlag}}, so the data container was empty and the store had 5 entries.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (ISPN-10178) TCP_NIO2 deadlock during stop on Windows
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-10178?page=com.atlassian.jira.plugin... ]
Dan Berindei commented on ISPN-10178:
-------------------------------------
Master no longer uses TCP_NIO2, but it is still the default on branch 9.4.x.
> TCP_NIO2 deadlock during stop on Windows
> ----------------------------------------
>
> Key: ISPN-10178
> URL: https://issues.jboss.org/browse/ISPN-10178
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 9.4.13.Final
> Environment: Windows
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Fix For: 10.0.0.Beta4, 9.4.14.Final
>
>
> {noformat}
> "testng-CustomFailurePolicyTxTest" #60 prio=0 tid=0x3c nid=NA blocked
> java.lang.Thread.State: BLOCKED
> at java.nio.channels.spi.AbstractSelectableChannel.removeKey(AbstractSelectableChannel.java:130)
> - waiting to lock <0x7ae7c881> (a java.lang.Object)
> at java.nio.channels.spi.AbstractSelector.deregister(AbstractSelector.java:188)
> at sun.nio.ch.WindowsSelectorImpl.implClose(WindowsSelectorImpl.java:514)
> - locked <0x6cc2bf11> (a java.lang.Object)
> at sun.nio.ch.SelectorImpl.implCloseSelector(SelectorImpl.java:113)
> - locked <0x2ab0aa2d> (a sun.nio.ch.Util$3)
> - locked <0x2f9dc993> (a java.util.Collections$UnmodifiableSet)
> - locked <0xfeca69a> (a sun.nio.ch.WindowsSelectorImpl)
> at java.nio.channels.spi.AbstractSelector.close(AbstractSelector.java:111)
> at org.jgroups.util.Util.close(Util.java:428)
> at org.jgroups.util.Util.close(Util.java:439)
> at org.jgroups.blocks.cs.NioServer.stop(NioServer.java:127)
> - locked <0x21bca9ea> (a org.jgroups.blocks.cs.NioServer)
> at org.jgroups.protocols.TCP_NIO2.handleDisconnect(TCP_NIO2.java:136)
> at org.jgroups.protocols.TP.handleDownEvent(TP.java:1607)
> at org.jgroups.protocols.BasicTCP.handleDownEvent(BasicTCP.java:129)
> at org.jgroups.protocols.TP.down(TP.java:1082)
> at org.jgroups.protocols.Discovery.down(Discovery.java:425)
> at org.infinispan.test.fwk.TEST_PING.down(TEST_PING.java:208)
> at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:379)
> at org.jgroups.protocols.FD_ALL.down(FD_ALL.java:236)
> at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:101)
> at org.jgroups.protocols.pbcast.NAKACK2.down(NAKACK2.java:553)
> at org.jgroups.protocols.UNICAST3.down(UNICAST3.java:570)
> at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:346)
> at org.jgroups.protocols.pbcast.GMS.down(GMS.java:1076)
> at org.jgroups.protocols.tom.TOA.down(TOA.java:80)
> at org.jgroups.protocols.FlowControl.down(FlowControl.java:296)
> at org.jgroups.protocols.RSVP.down(RSVP.java:102)
> at org.jgroups.protocols.relay.RELAY2.down(RELAY2.java:380)
> at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:908)
> at org.jgroups.JChannel.down(JChannel.java:668)
> at org.jgroups.JChannel.disconnect(JChannel.java:469)
> - locked <0xef29e32> (a org.jgroups.JChannel)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.stop(JGroupsTransport.java:712)
> at sun.reflect.GeneratedMethodAccessor148.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> at org.infinispan.commons.util.SecurityActions$$Lambda$154/1125443972.run(Unknown Source)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:181)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.performStop(BasicComponentRegistryImpl.java:601)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.stopWrapper(BasicComponentRegistryImpl.java:590)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.stop(BasicComponentRegistryImpl.java:461)
> at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:441)
> at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:376)
> at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:775)
> at org.infinispan.xsite.AbstractXSiteTest$$Lambda$4294/1094472832.accept(Unknown Source)
> at java.util.ArrayList.forEach(ArrayList.java:1257)
> at org.infinispan.xsite.AbstractXSiteTest.killSite(AbstractXSiteTest.java:89)
> at org.infinispan.xsite.AbstractXSiteTest.killSites(AbstractXSiteTest.java:82)
> at org.infinispan.xsite.AbstractXSiteTest.destroy(AbstractXSiteTest.java:76)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
> at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:59)
> at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:458)
> at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:222)
> at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:142)
> at org.testng.internal.TestMethodWorker.invokeAfterClassMethods(TestMethodWorker.java:214)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:648)
> at org.testng.TestRunner.run(TestRunner.java:505)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:40)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:489)
> at org.testng.internal.thread.ThreadUtil$1.call(ThreadUtil.java:52)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x9411fa8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> - <0x4fe21f2e> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> "NioServer.Selector [/127.0.0.1:7925]-3,CustomFailurePolicyTxTest-NodeC-12186" #205597 prio=0 tid=0x3231d nid=NA blocked
> java.lang.Thread.State: BLOCKED
> at sun.nio.ch.WindowsSelectorImpl.putEventOps(WindowsSelectorImpl.java:593)
> - waiting to lock <0x6cc2bf11> (a java.lang.Object)
> at sun.nio.ch.SocketChannelImpl.translateAndSetInterestOps(SocketChannelImpl.java:974)
> at sun.nio.ch.SelectionKeyImpl.nioInterestOps(SelectionKeyImpl.java:105)
> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:83)
> at sun.nio.ch.SelectorImpl.register(SelectorImpl.java:134)
> at java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:212)
> - locked <0x7ae7c881> (a java.lang.Object)
> - locked <0x6a684d8f> (a java.lang.Object)
> at org.jgroups.blocks.cs.NioServer.handleAccept(NioServer.java:93)
> at org.jgroups.blocks.cs.NioBaseServer$Acceptor.run(NioBaseServer.java:138)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (ISPN-10178) TCP_NIO2 deadlock during stop on Windows
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-10178?page=com.atlassian.jira.plugin... ]
Dan Berindei updated ISPN-10178:
--------------------------------
Labels: testsuite_stability (was: )
> TCP_NIO2 deadlock during stop on Windows
> ----------------------------------------
>
> Key: ISPN-10178
> URL: https://issues.jboss.org/browse/ISPN-10178
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 9.4.13.Final
> Environment: Windows
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Fix For: 10.0.0.Beta4, 9.4.14.Final
>
>
> {noformat}
> "testng-CustomFailurePolicyTxTest" #60 prio=0 tid=0x3c nid=NA blocked
> java.lang.Thread.State: BLOCKED
> at java.nio.channels.spi.AbstractSelectableChannel.removeKey(AbstractSelectableChannel.java:130)
> - waiting to lock <0x7ae7c881> (a java.lang.Object)
> at java.nio.channels.spi.AbstractSelector.deregister(AbstractSelector.java:188)
> at sun.nio.ch.WindowsSelectorImpl.implClose(WindowsSelectorImpl.java:514)
> - locked <0x6cc2bf11> (a java.lang.Object)
> at sun.nio.ch.SelectorImpl.implCloseSelector(SelectorImpl.java:113)
> - locked <0x2ab0aa2d> (a sun.nio.ch.Util$3)
> - locked <0x2f9dc993> (a java.util.Collections$UnmodifiableSet)
> - locked <0xfeca69a> (a sun.nio.ch.WindowsSelectorImpl)
> at java.nio.channels.spi.AbstractSelector.close(AbstractSelector.java:111)
> at org.jgroups.util.Util.close(Util.java:428)
> at org.jgroups.util.Util.close(Util.java:439)
> at org.jgroups.blocks.cs.NioServer.stop(NioServer.java:127)
> - locked <0x21bca9ea> (a org.jgroups.blocks.cs.NioServer)
> at org.jgroups.protocols.TCP_NIO2.handleDisconnect(TCP_NIO2.java:136)
> at org.jgroups.protocols.TP.handleDownEvent(TP.java:1607)
> at org.jgroups.protocols.BasicTCP.handleDownEvent(BasicTCP.java:129)
> at org.jgroups.protocols.TP.down(TP.java:1082)
> at org.jgroups.protocols.Discovery.down(Discovery.java:425)
> at org.infinispan.test.fwk.TEST_PING.down(TEST_PING.java:208)
> at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:379)
> at org.jgroups.protocols.FD_ALL.down(FD_ALL.java:236)
> at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:101)
> at org.jgroups.protocols.pbcast.NAKACK2.down(NAKACK2.java:553)
> at org.jgroups.protocols.UNICAST3.down(UNICAST3.java:570)
> at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:346)
> at org.jgroups.protocols.pbcast.GMS.down(GMS.java:1076)
> at org.jgroups.protocols.tom.TOA.down(TOA.java:80)
> at org.jgroups.protocols.FlowControl.down(FlowControl.java:296)
> at org.jgroups.protocols.RSVP.down(RSVP.java:102)
> at org.jgroups.protocols.relay.RELAY2.down(RELAY2.java:380)
> at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:908)
> at org.jgroups.JChannel.down(JChannel.java:668)
> at org.jgroups.JChannel.disconnect(JChannel.java:469)
> - locked <0xef29e32> (a org.jgroups.JChannel)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.stop(JGroupsTransport.java:712)
> at sun.reflect.GeneratedMethodAccessor148.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> at org.infinispan.commons.util.SecurityActions$$Lambda$154/1125443972.run(Unknown Source)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:181)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.performStop(BasicComponentRegistryImpl.java:601)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.stopWrapper(BasicComponentRegistryImpl.java:590)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.stop(BasicComponentRegistryImpl.java:461)
> at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:441)
> at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:376)
> at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:775)
> at org.infinispan.xsite.AbstractXSiteTest$$Lambda$4294/1094472832.accept(Unknown Source)
> at java.util.ArrayList.forEach(ArrayList.java:1257)
> at org.infinispan.xsite.AbstractXSiteTest.killSite(AbstractXSiteTest.java:89)
> at org.infinispan.xsite.AbstractXSiteTest.killSites(AbstractXSiteTest.java:82)
> at org.infinispan.xsite.AbstractXSiteTest.destroy(AbstractXSiteTest.java:76)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
> at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:59)
> at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:458)
> at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:222)
> at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:142)
> at org.testng.internal.TestMethodWorker.invokeAfterClassMethods(TestMethodWorker.java:214)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:648)
> at org.testng.TestRunner.run(TestRunner.java:505)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:40)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:489)
> at org.testng.internal.thread.ThreadUtil$1.call(ThreadUtil.java:52)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x9411fa8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> - <0x4fe21f2e> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> "NioServer.Selector [/127.0.0.1:7925]-3,CustomFailurePolicyTxTest-NodeC-12186" #205597 prio=0 tid=0x3231d nid=NA blocked
> java.lang.Thread.State: BLOCKED
> at sun.nio.ch.WindowsSelectorImpl.putEventOps(WindowsSelectorImpl.java:593)
> - waiting to lock <0x6cc2bf11> (a java.lang.Object)
> at sun.nio.ch.SocketChannelImpl.translateAndSetInterestOps(SocketChannelImpl.java:974)
> at sun.nio.ch.SelectionKeyImpl.nioInterestOps(SelectionKeyImpl.java:105)
> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:83)
> at sun.nio.ch.SelectorImpl.register(SelectorImpl.java:134)
> at java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:212)
> - locked <0x7ae7c881> (a java.lang.Object)
> - locked <0x6a684d8f> (a java.lang.Object)
> at org.jgroups.blocks.cs.NioServer.handleAccept(NioServer.java:93)
> at org.jgroups.blocks.cs.NioBaseServer$Acceptor.run(NioBaseServer.java:138)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (ISPN-10178) TCP_NIO2 deadlock during stop on Windows
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-10178?page=com.atlassian.jira.plugin... ]
Dan Berindei updated ISPN-10178:
--------------------------------
Affects Version/s: (was: 10.0.0.Beta3)
> TCP_NIO2 deadlock during stop on Windows
> ----------------------------------------
>
> Key: ISPN-10178
> URL: https://issues.jboss.org/browse/ISPN-10178
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 9.4.13.Final
> Environment: Windows
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Fix For: 10.0.0.Beta4, 9.4.14.Final
>
>
> {noformat}
> "testng-CustomFailurePolicyTxTest" #60 prio=0 tid=0x3c nid=NA blocked
> java.lang.Thread.State: BLOCKED
> at java.nio.channels.spi.AbstractSelectableChannel.removeKey(AbstractSelectableChannel.java:130)
> - waiting to lock <0x7ae7c881> (a java.lang.Object)
> at java.nio.channels.spi.AbstractSelector.deregister(AbstractSelector.java:188)
> at sun.nio.ch.WindowsSelectorImpl.implClose(WindowsSelectorImpl.java:514)
> - locked <0x6cc2bf11> (a java.lang.Object)
> at sun.nio.ch.SelectorImpl.implCloseSelector(SelectorImpl.java:113)
> - locked <0x2ab0aa2d> (a sun.nio.ch.Util$3)
> - locked <0x2f9dc993> (a java.util.Collections$UnmodifiableSet)
> - locked <0xfeca69a> (a sun.nio.ch.WindowsSelectorImpl)
> at java.nio.channels.spi.AbstractSelector.close(AbstractSelector.java:111)
> at org.jgroups.util.Util.close(Util.java:428)
> at org.jgroups.util.Util.close(Util.java:439)
> at org.jgroups.blocks.cs.NioServer.stop(NioServer.java:127)
> - locked <0x21bca9ea> (a org.jgroups.blocks.cs.NioServer)
> at org.jgroups.protocols.TCP_NIO2.handleDisconnect(TCP_NIO2.java:136)
> at org.jgroups.protocols.TP.handleDownEvent(TP.java:1607)
> at org.jgroups.protocols.BasicTCP.handleDownEvent(BasicTCP.java:129)
> at org.jgroups.protocols.TP.down(TP.java:1082)
> at org.jgroups.protocols.Discovery.down(Discovery.java:425)
> at org.infinispan.test.fwk.TEST_PING.down(TEST_PING.java:208)
> at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:379)
> at org.jgroups.protocols.FD_ALL.down(FD_ALL.java:236)
> at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:101)
> at org.jgroups.protocols.pbcast.NAKACK2.down(NAKACK2.java:553)
> at org.jgroups.protocols.UNICAST3.down(UNICAST3.java:570)
> at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:346)
> at org.jgroups.protocols.pbcast.GMS.down(GMS.java:1076)
> at org.jgroups.protocols.tom.TOA.down(TOA.java:80)
> at org.jgroups.protocols.FlowControl.down(FlowControl.java:296)
> at org.jgroups.protocols.RSVP.down(RSVP.java:102)
> at org.jgroups.protocols.relay.RELAY2.down(RELAY2.java:380)
> at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:908)
> at org.jgroups.JChannel.down(JChannel.java:668)
> at org.jgroups.JChannel.disconnect(JChannel.java:469)
> - locked <0xef29e32> (a org.jgroups.JChannel)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.stop(JGroupsTransport.java:712)
> at sun.reflect.GeneratedMethodAccessor148.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> at org.infinispan.commons.util.SecurityActions$$Lambda$154/1125443972.run(Unknown Source)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:181)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.performStop(BasicComponentRegistryImpl.java:601)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.stopWrapper(BasicComponentRegistryImpl.java:590)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.stop(BasicComponentRegistryImpl.java:461)
> at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:441)
> at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:376)
> at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:775)
> at org.infinispan.xsite.AbstractXSiteTest$$Lambda$4294/1094472832.accept(Unknown Source)
> at java.util.ArrayList.forEach(ArrayList.java:1257)
> at org.infinispan.xsite.AbstractXSiteTest.killSite(AbstractXSiteTest.java:89)
> at org.infinispan.xsite.AbstractXSiteTest.killSites(AbstractXSiteTest.java:82)
> at org.infinispan.xsite.AbstractXSiteTest.destroy(AbstractXSiteTest.java:76)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
> at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:59)
> at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:458)
> at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:222)
> at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:142)
> at org.testng.internal.TestMethodWorker.invokeAfterClassMethods(TestMethodWorker.java:214)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:648)
> at org.testng.TestRunner.run(TestRunner.java:505)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:40)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:489)
> at org.testng.internal.thread.ThreadUtil$1.call(ThreadUtil.java:52)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x9411fa8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> - <0x4fe21f2e> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> "NioServer.Selector [/127.0.0.1:7925]-3,CustomFailurePolicyTxTest-NodeC-12186" #205597 prio=0 tid=0x3231d nid=NA blocked
> java.lang.Thread.State: BLOCKED
> at sun.nio.ch.WindowsSelectorImpl.putEventOps(WindowsSelectorImpl.java:593)
> - waiting to lock <0x6cc2bf11> (a java.lang.Object)
> at sun.nio.ch.SocketChannelImpl.translateAndSetInterestOps(SocketChannelImpl.java:974)
> at sun.nio.ch.SelectionKeyImpl.nioInterestOps(SelectionKeyImpl.java:105)
> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:83)
> at sun.nio.ch.SelectorImpl.register(SelectorImpl.java:134)
> at java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:212)
> - locked <0x7ae7c881> (a java.lang.Object)
> - locked <0x6a684d8f> (a java.lang.Object)
> at org.jgroups.blocks.cs.NioServer.handleAccept(NioServer.java:93)
> at org.jgroups.blocks.cs.NioBaseServer$Acceptor.run(NioBaseServer.java:138)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (ISPN-10178) TCP_NIO2 deadlock during stop on Windows
by Dan Berindei (Jira)
Dan Berindei created ISPN-10178:
-----------------------------------
Summary: TCP_NIO2 deadlock during stop on Windows
Key: ISPN-10178
URL: https://issues.jboss.org/browse/ISPN-10178
Project: Infinispan
Issue Type: Bug
Components: Test Suite - Core
Affects Versions: 9.4.13.Final, 10.0.0.Beta3
Environment: Windows
Reporter: Dan Berindei
Assignee: Dan Berindei
Fix For: 10.0.0.Beta4, 9.4.14.Final
{noformat}
"testng-CustomFailurePolicyTxTest" #60 prio=0 tid=0x3c nid=NA blocked
java.lang.Thread.State: BLOCKED
at java.nio.channels.spi.AbstractSelectableChannel.removeKey(AbstractSelectableChannel.java:130)
- waiting to lock <0x7ae7c881> (a java.lang.Object)
at java.nio.channels.spi.AbstractSelector.deregister(AbstractSelector.java:188)
at sun.nio.ch.WindowsSelectorImpl.implClose(WindowsSelectorImpl.java:514)
- locked <0x6cc2bf11> (a java.lang.Object)
at sun.nio.ch.SelectorImpl.implCloseSelector(SelectorImpl.java:113)
- locked <0x2ab0aa2d> (a sun.nio.ch.Util$3)
- locked <0x2f9dc993> (a java.util.Collections$UnmodifiableSet)
- locked <0xfeca69a> (a sun.nio.ch.WindowsSelectorImpl)
at java.nio.channels.spi.AbstractSelector.close(AbstractSelector.java:111)
at org.jgroups.util.Util.close(Util.java:428)
at org.jgroups.util.Util.close(Util.java:439)
at org.jgroups.blocks.cs.NioServer.stop(NioServer.java:127)
- locked <0x21bca9ea> (a org.jgroups.blocks.cs.NioServer)
at org.jgroups.protocols.TCP_NIO2.handleDisconnect(TCP_NIO2.java:136)
at org.jgroups.protocols.TP.handleDownEvent(TP.java:1607)
at org.jgroups.protocols.BasicTCP.handleDownEvent(BasicTCP.java:129)
at org.jgroups.protocols.TP.down(TP.java:1082)
at org.jgroups.protocols.Discovery.down(Discovery.java:425)
at org.infinispan.test.fwk.TEST_PING.down(TEST_PING.java:208)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:379)
at org.jgroups.protocols.FD_ALL.down(FD_ALL.java:236)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:101)
at org.jgroups.protocols.pbcast.NAKACK2.down(NAKACK2.java:553)
at org.jgroups.protocols.UNICAST3.down(UNICAST3.java:570)
at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:346)
at org.jgroups.protocols.pbcast.GMS.down(GMS.java:1076)
at org.jgroups.protocols.tom.TOA.down(TOA.java:80)
at org.jgroups.protocols.FlowControl.down(FlowControl.java:296)
at org.jgroups.protocols.RSVP.down(RSVP.java:102)
at org.jgroups.protocols.relay.RELAY2.down(RELAY2.java:380)
at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:908)
at org.jgroups.JChannel.down(JChannel.java:668)
at org.jgroups.JChannel.disconnect(JChannel.java:469)
- locked <0xef29e32> (a org.jgroups.JChannel)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.stop(JGroupsTransport.java:712)
at sun.reflect.GeneratedMethodAccessor148.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
at org.infinispan.commons.util.SecurityActions$$Lambda$154/1125443972.run(Unknown Source)
at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:181)
at org.infinispan.factories.impl.BasicComponentRegistryImpl.performStop(BasicComponentRegistryImpl.java:601)
at org.infinispan.factories.impl.BasicComponentRegistryImpl.stopWrapper(BasicComponentRegistryImpl.java:590)
at org.infinispan.factories.impl.BasicComponentRegistryImpl.stop(BasicComponentRegistryImpl.java:461)
at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:441)
at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:376)
at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:775)
at org.infinispan.xsite.AbstractXSiteTest$$Lambda$4294/1094472832.accept(Unknown Source)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at org.infinispan.xsite.AbstractXSiteTest.killSite(AbstractXSiteTest.java:89)
at org.infinispan.xsite.AbstractXSiteTest.killSites(AbstractXSiteTest.java:82)
at org.infinispan.xsite.AbstractXSiteTest.destroy(AbstractXSiteTest.java:76)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:59)
at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:458)
at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:222)
at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:142)
at org.testng.internal.TestMethodWorker.invokeAfterClassMethods(TestMethodWorker.java:214)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
at org.testng.TestRunner.privateRun(TestRunner.java:648)
at org.testng.TestRunner.run(TestRunner.java:505)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
at org.testng.SuiteRunner.access$000(SuiteRunner.java:40)
at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:489)
at org.testng.internal.thread.ThreadUtil$1.call(ThreadUtil.java:52)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- <0x9411fa8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
- <0x4fe21f2e> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
"NioServer.Selector [/127.0.0.1:7925]-3,CustomFailurePolicyTxTest-NodeC-12186" #205597 prio=0 tid=0x3231d nid=NA blocked
java.lang.Thread.State: BLOCKED
at sun.nio.ch.WindowsSelectorImpl.putEventOps(WindowsSelectorImpl.java:593)
- waiting to lock <0x6cc2bf11> (a java.lang.Object)
at sun.nio.ch.SocketChannelImpl.translateAndSetInterestOps(SocketChannelImpl.java:974)
at sun.nio.ch.SelectionKeyImpl.nioInterestOps(SelectionKeyImpl.java:105)
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:83)
at sun.nio.ch.SelectorImpl.register(SelectorImpl.java:134)
at java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:212)
- locked <0x7ae7c881> (a java.lang.Object)
- locked <0x6a684d8f> (a java.lang.Object)
at org.jgroups.blocks.cs.NioServer.handleAccept(NioServer.java:93)
at org.jgroups.blocks.cs.NioBaseServer$Acceptor.run(NioBaseServer.java:138)
at java.lang.Thread.run(Thread.java:748)
{noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 4 months
- 1
- 0