[JBoss JIRA] (ISPN-8598) ClusteredLockTest.testTryLockWithCountersInParallel random failures
by Zdenek Hostasa (JIRA)
[ https://issues.jboss.org/browse/ISPN-8598?page=com.atlassian.jira.plugin.... ]
Zdenek Hostasa commented on ISPN-8598:
--------------------------------------
This issue seems to be limited to RHEL6 (ot of RHEL 6 and 7)
> ClusteredLockTest.testTryLockWithCountersInParallel random failures
> -------------------------------------------------------------------
>
> Key: ISPN-8598
> URL: https://issues.jboss.org/browse/ISPN-8598
> Project: Infinispan
> Issue Type: Bug
> Components: Clustered Locks
> Affects Versions: 9.2.0.Beta1
> Reporter: Gustavo Fernandes
>
> {noformat}
> [OK: 8, KO: 1, SKIP: 0] Test failed: org.infinispan.lock.ClusteredLockTest.testTryLockWithCountersInParallel
> java.lang.AssertionError: expected:<1> but was:<2>
> at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
> at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:364)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:80)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:245)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:252)
> at org.infinispan.lock.ClusteredLockTest.testTryLockWithCountersInParallel(ClusteredLockTest.java:203)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:38)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:382)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 11 months
- 1
- 0
[JBoss JIRA] (ISPN-8731) Write command times out waiting for wrong topology
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-8731?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-8731:
-------------------------------
Priority: Critical (was: Blocker)
> Write command times out waiting for wrong topology
> --------------------------------------------------
>
> Key: ISPN-8731
> URL: https://issues.jboss.org/browse/ISPN-8731
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite - Query, Test Suite - Server
> Affects Versions: 9.2.0.CR1
> Reporter: Dan Berindei
> Priority: Critical
> Labels: testsuite_stability
> Fix For: 9.2.0.CR2
>
>
> {{MultipleCacheManagersTest.waitForClusterToForm(new String[]{})}} doesn't actually wait for the rebalance to finish everywhere: https://github.com/infinispan/infinispan/pull/5705/files#diff-f9f535214b9...
> Most tests should work regardless of whether we wait for the rebalance to finish or not, it's just a way to reduce the number of test failures when the command retry doesn't work. Waiting for the rebalance to finish is only required when the test needs to install new topologies in a specific order.
> However, some recent test failures in CI show that the effectiveness of {{MultipleCacheManagersTest.waitForClusterToForm(new String[]{})}} does make a difference. Most likely there is a bug in the retry logic during the latter phases of a rebalance, and the retry-specific tests do not cover all the scenarios.
> {noformat}
> 00:23:20,668 DEBUG (remote-thread-ClusteredScriptingTest-NodeI-p218-t5:[]) [ClusterTopologyManagerImpl] Updating cluster-wide current topology for cache ___defaultcache, topology = CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[ClusteredScriptingTest-NodeI-40041: 134, ClusteredScriptingTest-NodeJ-9982: 122]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredScriptingTest-NodeI-40041, ClusteredScriptingTest-NodeJ-9982], persistentUUIDs=[91a47fef-fad6-479c-9ec9-12e9350dc6d5, d2200930-5793-4b74-beea-e001a2b414c1]}, availability mode = AVAILABLE
> 00:23:20,669 DEBUG (remote-thread-ClusteredScriptingTest-NodeI-p218-t5:[]) [ClusterTopologyManagerImpl] Updating cluster-wide stable topology for cache ___defaultcache, topology = CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[ClusteredScriptingTest-NodeI-40041: 134, ClusteredScriptingTest-NodeJ-9982: 122]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredScriptingTest-NodeI-40041, ClusteredScriptingTest-NodeJ-9982], persistentUUIDs=[91a47fef-fad6-479c-9ec9-12e9350dc6d5, d2200930-5793-4b74-beea-e001a2b414c1]}
> 00:23:20,669 DEBUG (transport-thread-ClusteredScriptingTest-NodeI-p220-t6:[Topology-___defaultcache]) [LocalTopologyManagerImpl] Updating local topology for cache ___defaultcache: CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[ClusteredScriptingTest-NodeI-40041: 134, ClusteredScriptingTest-NodeJ-9982: 122]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredScriptingTest-NodeI-40041, ClusteredScriptingTest-NodeJ-9982], persistentUUIDs=[91a47fef-fad6-479c-9ec9-12e9350dc6d5, d2200930-5793-4b74-beea-e001a2b414c1]}
> 00:23:20,669 DEBUG (transport-thread-ClusteredScriptingTest-NodeJ-p234-t6:[Topology-___defaultcache]) [LocalTopologyManagerImpl] Updating local topology for cache ___defaultcache: CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[ClusteredScriptingTest-NodeI-40041: 134, ClusteredScriptingTest-NodeJ-9982: 122]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredScriptingTest-NodeI-40041, ClusteredScriptingTest-NodeJ-9982], persistentUUIDs=[91a47fef-fad6-479c-9ec9-12e9350dc6d5, d2200930-5793-4b74-beea-e001a2b414c1]}
> 00:23:35,673 ERROR (timeout-thread-ClusteredScriptingTest-NodeI-p219-t1:[]) [InvocationContextInterceptor] ISPN000136: Error executing command PutKeyValueCommand, writing keys [/macbeth.txt0]
> org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 6
> at org.infinispan.interceptors.impl.BaseStateTransferInterceptor$CancellableRetry.run(BaseStateTransferInterceptor.java:333) [infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> 00:23:35,673 DEBUG (testng-ClusteredScriptingTest:[]) [DefaultCacheManager] Stopping cache manager ISPN on ClusteredScriptingTest-NodeJ-9982
> 00:23:35,682 DEBUG (remote-thread-ClusteredScriptingTest-NodeI-p218-t5:[]) [ClusterTopologyManagerImpl] Updating cluster-wide current topology for cache ___defaultcache, topology = CacheTopology{id=6, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[ClusteredScriptingTest-NodeI-40041: 256]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredScriptingTest-NodeI-40041], persistentUUIDs=[91a47fef-fad6-479c-9ec9-12e9350dc6d5]}, availability mode = AVAILABLE
> 00:23:35,699 ERROR (testng-ClusteredScriptingTest:[]) [TestSuiteProgress] Test failed: org.infinispan.scripting.ClusteredScriptingTest.testDistributedMapReduceStreamWithFlag([REPL_SYNC])
> org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 6
> at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1636) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1284) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1750) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:217) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:358) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:671) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.scripting.utils.ScriptingUtils.loadData(ScriptingUtils.java:29) ~[test-classes/:?]
> at org.infinispan.scripting.ClusteredScriptingTest$6.call(ClusteredScriptingTest.java:144) ~[test-classes/:?]
> at org.infinispan.test.TestingUtil.withCacheManagers(TestingUtil.java:1522) ~[infinispan-core-9.2.0-SNAPSHOT-tests.jar:9.2.0-SNAPSHOT]
> at org.infinispan.scripting.ClusteredScriptingTest.testDistributedMapReduceStreamWithFlag(ClusteredScriptingTest.java:137) ~[test-classes/:?]
> {noformat}
> {noformat}
> 18:04:17,097 DEBUG (transport-thread-ClusteredCacheTest-NodeB-p3689-t1:[Topology-___defaultcache]) [LocalTopologyManagerImpl] Updating local topology for cache ___defaultcache: CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[ClusteredCacheTest-NodeA-53655: 134, ClusteredCacheTest-NodeB-27398: 122]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredCacheTest-NodeA-53655, ClusteredCacheTest-NodeB-27398], persistentUUIDs=[5f5b2dd6-c570-4c6c-9c2d-92aa5fcd2362, b623ac7a-3c77-471d-88de-e151da823c0c]}
> 18:04:17,097 INFO (testng-ClusteredCacheTest:[]) [TestSuiteProgress] Test starting: org.infinispan.query.blackbox.ClusteredCacheTest.testConditionalRemoveFromNonOwner
> 18:04:17,097 DEBUG (transport-thread-ClusteredCacheTest-NodeA-p3679-t5:[Topology-___defaultcache]) [LocalTopologyManagerImpl] Updating local topology for cache ___defaultcache: CacheTopology{id=5, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[ClusteredCacheTest-NodeA-53655: 134, ClusteredCacheTest-NodeB-27398: 122]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredCacheTest-NodeA-53655, ClusteredCacheTest-NodeB-27398], persistentUUIDs=[5f5b2dd6-c570-4c6c-9c2d-92aa5fcd2362, b623ac7a-3c77-471d-88de-e151da823c0c]}
> 18:04:32,100 ERROR (timeout-thread-ClusteredCacheTest-NodeA-p3678-t1:[]) [InvocationContextInterceptor] ISPN000136: Error executing command PutKeyValueCommand, writing keys [WrappedByteArray{bytes=[B0x010201054E617669..[9], hashCode=-1707624030}]
> org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 6
> at org.infinispan.interceptors.impl.BaseStateTransferInterceptor$CancellableRetry.run(BaseStateTransferInterceptor.java:333) [infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> 18:04:32,101 ERROR (testng-ClusteredCacheTest:[]) [TestSuiteProgress] Test failed: org.infinispan.query.blackbox.ClusteredCacheTest.testConditionalRemoveFromNonOwner
> org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 6
> at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1636) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1284) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1750) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:217) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:358) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:671) ~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
> at org.infinispan.query.blackbox.ClusteredCacheTest.prepareTestData(ClusteredCacheTest.java:140) ~[test-classes/:?]
> at org.infinispan.query.blackbox.ClusteredCacheTest.testConditionalRemoveFrom(ClusteredCacheTest.java:334) ~[test-classes/:?]
> at org.infinispan.query.blackbox.ClusteredCacheTest.testConditionalRemoveFromNonOwner(ClusteredCacheTest.java:298) ~[test-classes/:?]
> 18:04:32,214 DEBUG (testng-ClusteredCacheTest:[]) [DefaultCacheManager] Stopping cache manager ISPN on ClusteredCacheTest-NodeB-27398
> 18:04:32,479 DEBUG (remote-thread-ClusteredCacheTest-NodeA-p3677-t6:[]) [ClusterTopologyManagerImpl] Updating cluster-wide current topology for cache ___defaultcache, topology = CacheTopology{id=6, phase=NO_REBALANCE, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[ClusteredCacheTest-NodeA-53655: 256]}, pendingCH=null, unionCH=null, actualMembers=[ClusteredCacheTest-NodeA-53655], persistentUUIDs=[5f5b2dd6-c570-4c6c-9c2d-92aa5fcd2362]}, availability mode = AVAILABLE
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 11 months
- 1
- 0
[JBoss JIRA] (ISPN-2491) Order of locks in optimistic locking is not strict
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-2491?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-2491:
------------------------------------
[~rvansa] I think this issue was meant to cover transactions that modify the same keys but in a different order. {{Collections.sort()}} is guaranteed to be stable, so transactions that touch the same sets of keys in the same order always worked.
> Order of locks in optimistic locking is not strict
> --------------------------------------------------
>
> Key: ISPN-2491
> URL: https://issues.jboss.org/browse/ISPN-2491
> Project: Infinispan
> Issue Type: Quality Risk
> Components: Transactions
> Affects Versions: 5.1.8.Final, 5.2.0.Beta3
> Reporter: Radim Vansa
> Priority: Minor
>
> In OptimisticLockingInterceptor, the keys are ordered according to their hash. However, the hashes can still collide, which may result in a deadlock if two keys with identical hash (only 32-bit) are sorted to different order. We should try to check if the keys are Comparable or let user provide some comparator class in config, and use the compare of hash only as the last resort.
> In all cases, a warning should be emitted if the compare operation had non-strict result.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 11 months
- 1
- 0
[JBoss JIRA] (ISPN-8724) HotRodRemoteStreamingIT hangs
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-8724?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-8724:
------------------------------------
Even when the test passes, it logs a worrisome exception at the end:
{noformat}
16:05:40,435 WARN [org.infinispan.manager.DefaultCacheManager] (MSC service thread 1-4) ISPN000189: While stopping a cache or cache manager, one of its components failed to stop: java.util.concurrent.CompletionException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
at java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:375)
at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1934)
at org.infinispan.manager.DefaultCacheManager.terminate(DefaultCacheManager.java:681)
at org.infinispan.manager.DefaultCacheManager.stopCaches(DefaultCacheManager.java:727)
at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:704)
at org.infinispan.manager.impl.AbstractDelegatingEmbeddedCacheManager.stop(AbstractDelegatingEmbeddedCacheManager.java:191)
at org.infinispan.server.infinispan.SecurityActions$2.run(SecurityActions.java:98)
at org.infinispan.server.infinispan.SecurityActions$2.run(SecurityActions.java:94)
at org.infinispan.security.Security.doPrivileged(Security.java:76)
at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:69)
at org.infinispan.server.infinispan.SecurityActions.stopAndUnregisterContainer(SecurityActions.java:106)
at org.jboss.as.clustering.infinispan.subsystem.CacheContainerBuilder.stop(CacheContainerBuilder.java:107)
at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83)
at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
at org.infinispan.security.Security.doPrivileged(Security.java:76)
at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:228)
at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:962)
at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:637)
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:582)
at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:468)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
at org.infinispan.counter.impl.CounterModuleLifecycle.lambda$startCaches$0(CounterModuleLifecycle.java:132)
... 1 more
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache ___counter_configuration on node0
at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:233)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
... 21 more
{noformat}
See http://ci.infinispan.org/job/Infinispan/job/master/376/consoleFull (last build before ignoring {{serverKillTest}}).
> HotRodRemoteStreamingIT hangs
> -----------------------------
>
> Key: ISPN-8724
> URL: https://issues.jboss.org/browse/ISPN-8724
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 9.2.0.CR1
> Reporter: Gustavo Fernandes
> Priority: Critical
> Attachments: s1, s2, test
>
>
> This was observed during the execution of test {{HotRodRemoteStreamingIT}}, Line 499 on method stopServer().
> The surefire process waits forever for the two servers to stop.
> One of the servers is waiting for the initial state transfer to complete:
> {noformat}
> "Thread-48" #95 prio=5 os_prio=0 tid=0x00007fdaf48f69f0 nid=0x590a waiting on condition [0x00007fdae1a26000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ee8cfc00> (a java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:231)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> at org.infinispan.commons.util.SecurityActions$$Lambda$170/1509149284.run(Unknown Source)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> at org.infinispan.factories.AbstractComponentRegistry$$Lambda$175/1225674479.run(Unknown Source)
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> at org.infinispan.factories.SecurityActions$$Lambda$176/1784470267.run(Unknown Source)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:228)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:962)
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:637)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:582)
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:468)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
> at org.infinispan.counter.impl.CounterModuleLifecycle.lambda$startCaches$0(CounterModuleLifecycle.java:132)
> at org.infinispan.counter.impl.CounterModuleLifecycle$$Lambda$194/218224883.run(Unknown Source)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> The other server is running a series of operations after the server stops, such
> as {{EmbeddedCacheManager#undefineConfiguration}}, this can be seen on 3 threads:
> MSC service thread 1-1, MSC service thread 1-2 and MSC service thread 1-3
> {noformat}
> "MSC service thread 1-1" #12 prio=5 os_prio=0 tid=0x00007fd4e86720a0 nid=0x56f1 waiting on condition [0x00007fd4bc8f2000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fb922a18> (a java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693)
> at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729)
> at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1934)
> at org.infinispan.manager.DefaultCacheManager.undefineConfiguration(DefaultCacheManager.java:394)
> at org.infinispan.security.actions.UndefineConfigurationAction.run(UndefineConfigurationAction.java:25)
> at org.infinispan.security.actions.UndefineConfigurationAction.run(UndefineConfigurationAction.java:13)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.registry.impl.SecurityActions.undefineConfiguration(SecurityActions.java:39)
> at org.infinispan.registry.impl.InternalCacheRegistryImpl.unregisterInternalCache(InternalCacheRegistryImpl.java:77)
> - locked <0x00000000ee2e1420> (a org.infinispan.registry.impl.InternalCacheRegistryImpl)
> at org.infinispan.server.hotrod.HotRodServer.stop(HotRodServer.java:486)
> at org.infinispan.server.endpoint.subsystem.ProtocolServerService.doStop(ProtocolServerService.java:216)
> at org.infinispan.server.endpoint.subsystem.ProtocolServerService.stop(ProtocolServerService.java:206)
> - locked <0x00000000edd0eae0> (a org.infinispan.server.endpoint.subsystem.ProtocolServerService)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> For some reason, also in the other server, the CounterModule lifecycle is reacting to a
> cache manager start event, and it's hanged as well:
> {noformat}
> "Thread-48" #95 prio=5 os_prio=0 tid=0x00007fd4986a2090 nid=0x575b waiting on condition [0x00007fd48f2f4000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ee91b048> (a java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:231)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> at org.infinispan.commons.util.SecurityActions$$Lambda$170/1758042552.run(Unknown Source)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> at org.infinispan.factories.AbstractComponentRegistry$$Lambda$175/1042194083.run(Unknown Source)
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> at org.infinispan.factories.SecurityActions$$Lambda$176/2118421355.run(Unknown Source)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:228)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:962)
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:637)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:582)
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:468)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
> at org.infinispan.counter.impl.CounterModuleLifecycle.lambda$startCaches$0(CounterModuleLifecycle.java:132)
> at org.infinispan.counter.impl.CounterModuleLifecycle$$Lambda$194/1023465788.run(Unknown Source)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 11 months
- 1
- 0