DistSyncStoreNotSharedTest.clearContent sometimes hangs ------------------------------------------------------- Key: ISPN-9844 URL: https://issues.jboss.org/browse/ISPN-9844 Project: Infinispan Issue Type: Bug Components: Test Suite - Core Affects Versions: 10.0.0.Alpha2 Reporter: Dan Berindei Assignee: Ryan Emerson Priority: Major Labels: testsuite_stability Fix For: 10.0.0.Beta1 Attachments: DistSyncStoreNotSharedTest_20181221.tar.xz, threaddump-org_infinispan_distribution_DistSyncStoreNotSharedTest_clearContent-2018-12-19-7142.log.xz The main test thread is waiting to acquire the write lock of {{storesMutex}}, and a transport thread is waiting for a publisher to give it the keys of stale segments. There's no thread with a {{storesMutex}} read lock acquisition in the stack (although one thread is trying to acquire the read lock), so either we leak a read lock or we hold the read lock for longer than necessary: {noformat} "testng-DistSyncStoreNotSharedTest" #15 prio=5 os_prio=0 tid=0x00007fd0bce84800 nid=0x1c42 waiting on condition [0x00007fd0888aa000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000dc2a5578> (a java.util.concurrent.Semaphore$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283) at java.util.concurrent.Semaphore.acquireUninterruptibly(Semaphore.java:494) at org.infinispan.persistence.manager.PersistenceManagerImpl.stop(PersistenceManagerImpl.java:215) "transport-thread-DistSyncStoreNotSharedTest-NodeB-p12241-t1" #46086 daemon prio=5 os_prio=0 tid=0x00007fd09406f800 nid=0x5ee8 waiting on condition [0x00007fd05ce9e000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000dc2a4df8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at io.reactivex.internal.operators.flowable.BlockingFlowableIterable$BlockingFlowableIterator.hasNext(BlockingFlowableIterable.java:94) at io.reactivex.Flowable.blockingForEach(Flowable.java:5509) at org.infinispan.statetransfer.StateConsumerImpl.removeStaleData(StateConsumerImpl.java:993) at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:441) "persistence-thread-DistSyncStoreNotSharedTest-NodeB-p12255-t1" #46212 daemon prio=5 os_prio=0 tid=0x00007fd09420a000 nid=0x5f66 waiting on condition [0x00007fd044a89000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000dc2a4870> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283) at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727) at org.infinispan.persistence.manager.PersistenceManagerImpl.pollStoreAvailability(PersistenceManagerImpl.java:189) {noformat}