[JBoss JIRA] (ISPN-10366) ScatteredStateConsumerImpl sets segment state to OWNED before applying values
by Dan Berindei (Jira)
Dan Berindei created ISPN-10366:
-----------------------------------
Summary: ScatteredStateConsumerImpl sets segment state to OWNED before applying values
Key: ISPN-10366
URL: https://issues.jboss.org/browse/ISPN-10366
Project: Infinispan
Issue Type: Bug
Components: Core, Test Suite - Core
Affects Versions: 9.4.15.Final, 10.0.0.Beta3
Reporter: Dan Berindei
Assignee: Radim Vansa
Fix For: 10.0.0.Beta4
{{ScatteredStateConsumerImpl}} uses {{InboundTransferTask}}s only to request keys, then after it received all the keys of a segment it changes the segment state to {{VALUE_TRANSFER}} and starts an asynchronous request to fetch the values and replace the {{RemoteMetadata}} entries with real entries.
{{ScatteredStateConsumerImpl.chunkCounter}} is supposed to delay the state transfer end and the segment state change to {{OWNED}}, but in rare occasions this doesn't happen.
This happened in {{PrefetchTest.testPrefetch12}} while running the test suite with {{taskset -c 1-2}}:
{noformat}
21:54:43,304 TRACE (transport-thread-Test-NodeC-p69907-t5:[Topology-___defaultcache]) [StateConsumerImpl] Received new topology for cache ___defaultcache, isRebalance = true, isMember = true, topology = CacheTopology{id=9, phase=TRANSITORY, rebalanceId=5, currentCH=PartitionerConsistentHash:ScatteredConsistentHash{ns=1, rebalanced=false, owners = (2)[Test-NodeA-39104: 1, Test-NodeC-3746: 0]}, pendingCH=PartitionerConsistentHash:ScatteredConsistentHash{ns=1, rebalanced=true, owners = (2)[Test-NodeA-39104: 0, Test-NodeC-3746: 1]}, unionCH=PartitionerConsistentHash:ScatteredConsistentHash{ns=1, rebalanced=false, owners = (2)[Test-NodeA-39104: 0, Test-NodeC-3746: 1]}, actualMembers=[Test-NodeA-39104, Test-NodeC-3746], persistentUUIDs=[f58e0a9a-dd4e-429a-8464-da64bf001d4e, 1471096f-c59a-4dc9-8f4d-31fbf399a2aa]}
21:54:43,305 TRACE (stateTransferExecutor-thread-Test-NodeC-p69908-t6:[StateRequest-___defaultcache]) [ScatteredStateConsumerImpl] Requesting keys for segments {0} from Test-NodeA-39104
21:54:43,313 TRACE (transport-thread-Test-NodeC-p69907-t5:[Topology-___defaultcache]) [StateConsumerImpl] Topology update processed, stateTransferTopologyId = 9, startRebalance = true, pending CH = PartitionerConsistentHash:ScatteredConsistentHash{ns=1, rebalanced=true, owners = (2)[Test-NodeA-39104: 0, Test-NodeC-3746: 1]}
21:54:43,313 TRACE (transport-thread-Test-NodeC-p69907-t5:[Topology-___defaultcache]) [StateTransferLockImpl] Signalling transaction data received for topology 9
21:54:43,313 TRACE (remote-thread-Test-NodeC-p69905-t2:[]) [TrianglePerCacheInboundInvocationHandler] Calling perform() on StateResponseCommand{cache=___defaultcache, pushTransfer=false, stateChunks=[StateChunk{segmentId=0, cacheEntries=1, isLastChunk=true}], origin=Test-NodeA-39104, topologyId=9, applyState=true}
21:54:43,313 TRACE (stateTransferExecutor-thread-Test-NodeC-p69908-t6:[]) [StateConsumerImpl] Applying new state chunk for segment 0 of cache ___defaultcache from node Test-NodeA-39104: received 1 cache entries
21:54:43,314 TRACE (stateTransferExecutor-thread-Test-NodeC-p69908-t6:[]) [ScatteredVersionManagerImpl] Finished transfer for segment 0 = KEY_TRANSFER -> VALUE_TRANSFER
21:54:43,314 TRACE (stateTransferExecutor-thread-Test-NodeC-p69908-t6:[]) [ScatteredVersionManagerImpl] Node Test-NodeC-3746, segment 0 has all keys in, expects value transfer
21:54:43,314 TRACE (stateTransferExecutor-thread-Test-NodeC-p69908-t6:[]) [ScatteredStateConsumerImpl] Requesting values from segments {0}, for in-memory keys
21:54:43,314 TRACE (stateTransferExecutor-thread-Test-NodeC-p69908-t6:[]) [ScatteredStateConsumerImpl] Retrieving values, chunk counter is 1
21:54:43,314 TRACE (stateTransferExecutor-thread-Test-NodeC-p69908-t6:[]) [JGroupsTransport] Test-NodeC-3746 sending request 11 to Test-NodeA-39104: ClusteredGetAllCommand{keys=[key], flags=[SKIP_OWNERSHIP_CHECK], topologyId=9}
21:54:43,314 TRACE (stateTransferExecutor-thread-Test-NodeC-p69908-t6:[]) [ScatteredStateConsumerImpl] Invalidating versions on Test-NodeC-3746, chunk counter incremented to 2
21:54:43,314 TRACE (stateTransferExecutor-thread-Test-NodeC-p69908-t6:[]) [ScatteredStateConsumerImpl] Versions invalidated on Test-NodeC-3746, chunk counter decremented to 1
21:54:43,314 TRACE (stateTransferExecutor-thread-Test-NodeC-p69908-t6:[]) [StateConsumerImpl] Removing inbound transfers from node {0} for segments Test-NodeA-39104
21:54:43,314 TRACE (stateTransferExecutor-thread-Test-NodeC-p69908-t6:[]) [ScatteredStateConsumerImpl] Inbound transfer removed, chunk counter is 1
21:54:43,314 TRACE (stateTransferExecutor-thread-Test-NodeC-p69908-t6:[]) [StateConsumerImpl] Latch 0
21:54:43,315 TRACE (jgroups-7,Test-NodeC-3746:[]) [JGroupsTransport] Test-NodeC-3746 received response for request 11 from Test-NodeA-39104: SuccessfulResponse([MetadataImmortalCacheValue {value=v0, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=SimpleClusteredVersion{topologyId=7, version=1}}}])
21:54:43,316 TRACE (jgroups-7,Test-NodeC-3746:[]) [BlockingInterceptor] Command blocking before completion of PutKeyValueCommand{key=key, value=v0, flags=[CACHE_MODE_LOCAL, SKIP_REMOTE_LOOKUP, PUT_FOR_STATE_TRANSFER, SKIP_SHARED_CACHE_STORE, SKIP_OWNERSHIP_CHECK, IGNORE_RETURN_VALUES, SKIP_XSITE_BACKUP], commandInvocationId=CommandInvocation:Test-NodeC-3746:121294, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=InternalMetadataImpl{actual=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=SimpleClusteredVersion{topologyId=7, version=1}}, created=-1, lastUsed=-1}, successful=true, topologyId=-1}
21:54:43,316 TRACE (remote-thread-Test-NodeC-p69905-t2:[___defaultcache]) [StateConsumerImpl] After applying the received state the data container of cache ___defaultcache has 1 keys
21:54:43,316 TRACE (remote-thread-Test-NodeC-p69905-t2:[___defaultcache]) [StateConsumerImpl] Segments not received yet for cache ___defaultcache: {}
21:54:43,316 DEBUG (transport-thread-Test-NodeC-p69907-t5:[Topology-___defaultcache]) [StateConsumerImpl] Finished receiving of segments for cache ___defaultcache for topology 9.
21:54:43,316 DEBUG (transport-thread-Test-NodeC-p69907-t5:[Topology-___defaultcache]) [ScatteredVersionManagerImpl] Node Test-NodeC-3746 received values for all segments in topology 9
21:54:43,316 TRACE (transport-thread-Test-NodeC-p69907-t5:[Topology-___defaultcache]) [StateConsumerImpl] Stop keeping track of changed keys for state transfer in topology 9
{noformat}
The test then starts a put operation and expects it to prefetch the previous value, but because the segment is {{OWNED}}, the {{RemoteMetadata}} is ignored:
{noformat}
21:54:43,316 TRACE (ForkThread-1,Test:[]) [InvocationContextInterceptor] Invoked with command PutKeyValueCommand{key=key, value=v1, flags=[], commandInvocationId=CommandInvocation:Test-NodeC-3746:121295, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true, topologyId=-1} and InvocationContext [SingleKeyNonTxInvocationContext{isLocked=false, key=null, cacheEntry=null, origin=null, lockOwner=CommandInvocation:Test-NodeC-3746:121295}]
21:54:43,316 TRACE (ForkThread-1,Test:[]) [EntryFactoryImpl] Retrieved from container MetadataImmortalCacheEntry{key=key, value=null, metadata=RemoteMetadata{address=Test-NodeA-39104, version=1}}
21:54:43,316 TRACE (ForkThread-1,Test:[]) [ScatteredDistributionInterceptor] Committing entry RepeatableReadEntry(108d175b){key=key, value=v1, isCreated=false, isChanged=true, isRemoved=false, isExpired=false, skipLookup=true, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=SimpleClusteredVersion{topologyId=9, version=1}}}, replaced MetadataImmortalCacheEntry{key=key, value=null, metadata=RemoteMetadata{address=Test-NodeA-39104, version=1}}
21:54:53,316 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.scattered.statetransfer.PrefetchTest.testPrefetch12
org.infinispan.test.TestException: java.util.concurrent.TimeoutException
at org.infinispan.util.ControlledRpcManager.uncheckedGet(ControlledRpcManager.java:259) ~[test-classes/:?]
at org.infinispan.util.ControlledRpcManager.expectCommand(ControlledRpcManager.java:124) ~[test-classes/:?]
at org.infinispan.scattered.statetransfer.PrefetchTest.testPrefetch(PrefetchTest.java:110) ~[test-classes/:?]
at org.infinispan.scattered.statetransfer.PrefetchTest.testPrefetch12(PrefetchTest.java:67) ~[test-classes/:?]
{noformat}
On a related note, {{StateConsumerImpl.applyState(pushTransfer=true)}} initializes a {{CountDownLatch(stateChunks.size())}}, but doesn't actually count down if {{stateChunk.getCacheEntries() == null}}, potentially hanging state transfer until it times out.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 6 months
[JBoss JIRA] (ISPN-10363) LazyInitializingExecutorService is not thread-safe
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-10363?page=com.atlassian.jira.plugin... ]
Dan Berindei updated ISPN-10363:
--------------------------------
Git Pull Request: https://github.com/infinispan/infinispan/pull/7096, https://github.com/infinispan/infinispan/pull/7101 (was: https://github.com/infinispan/infinispan/pull/7096)
> LazyInitializingExecutorService is not thread-safe
> --------------------------------------------------
>
> Key: ISPN-10363
> URL: https://issues.jboss.org/browse/ISPN-10363
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite - Core
> Affects Versions: 10.0.0.Beta3, 9.4.15.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Fix For: 10.0.0.Beta4
>
>
> {{LazyInitializingExecutorService.shutdown()}} is not synchronized, so it may miss the executor created on another thread.
> Normally the long time between the first use and stop is long enough for this to be a non-issue, but it does cause random thread leaks in {{PersistenceManagerTest}}:
> {noformat}
> 17:57:06,421 DEBUG (testng-PersistenceManagerTest:[]) [DefaultCacheManager] Started cache manager PersistenceManagerTest-NodeB on null
> 17:57:06,423 INFO (testng-PersistenceManagerTest:[]) [TestSuiteProgress] Test starting: org.infinispan.persistence.PersistenceManagerTest.testProcessAfterStop
> 17:57:06,446 INFO (testng-PersistenceManagerTest:[]) [TestSuiteProgress] Test succeeded: org.infinispan.persistence.PersistenceManagerTest.testProcessAfterStop
> 17:57:06,447 DEBUG (testng-PersistenceManagerTest:[]) [DefaultCacheManager] Stopped cache manager PersistenceManagerTest-NodeB
> 17:58:38,062 WARN (main:[]) [ThreadLeakChecker] Possible leaked thread:
> "async-thread-PersistenceManagerTest-NodeB-p54399-t1" daemon prio=5 tid=0x2ab4c nid=NA waiting
> java.lang.Thread.State: WAITING
> at java.base(a)11.0.3/jdk.internal.misc.Unsafe.park(Native Method)
> at java.base@11.0.3/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
> at java.base@11.0.3/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
> at java.base@11.0.3/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
> at java.base@11.0.3/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
> at java.base@11.0.3/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
> at java.base@11.0.3/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base@11.0.3/java.lang.Thread.run(Thread.java:834)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 6 months
[JBoss JIRA] (ISPN-10364) Persistence (Blocking) thread pool should have a queue
by Will Burns (Jira)
[ https://issues.jboss.org/browse/ISPN-10364?page=com.atlassian.jira.plugin... ]
Will Burns updated ISPN-10364:
------------------------------
Status: Pull Request Sent (was: Coding In Progress)
Git Pull Request: https://github.com/infinispan/infinispan/pull/7100
> Persistence (Blocking) thread pool should have a queue
> ------------------------------------------------------
>
> Key: ISPN-10364
> URL: https://issues.jboss.org/browse/ISPN-10364
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Reporter: Will Burns
> Assignee: Will Burns
> Priority: Major
> Fix For: 10.0.0.Beta4
>
>
> When Persistence executor was added it didn't have a queue. Now that more things are being run on this executor we can end up running things on the wrong thread. This is apparent with this error.
> {code}
> 16:36:47,118 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.distribution.groups.TransactionalGetGroupKeysTest.testRemoveGroupKeysWithPersistenceAndPassivation[PRIMARY_OWNER, DIST_SYNC, TO=true, isolation=READ_COMMITTED]
> org.infinispan.commons.CacheException: Could not commit implicit transaction
> at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1991) ~[classes/:?]
> at org.infinispan.cache.impl.CacheImpl.transactionalRemoveGroup(CacheImpl.java:651) ~[classes/:?]
> at org.infinispan.cache.impl.CacheImpl.removeGroup(CacheImpl.java:634) ~[classes/:?]
> at org.infinispan.cache.impl.CacheImpl.removeGroup(CacheImpl.java:627) ~[classes/:?]
> at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.removeGroup(AbstractDelegatingAdvancedCache.java:342) ~[classes/:?]
> at org.infinispan.distribution.groups.GetGroupKeysTest.testRemoveGroupKeysWithPersistenceAndPassivation(GetGroupKeysTest.java:165) ~[test-classes/:?]
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124) ~[testng-6.14.3.jar:?]
> at org.testng.internal.MethodInvocationHelper$1.runTestMethod(MethodInvocationHelper.java:230) ~[testng-6.14.3.jar:?]
> at org.infinispan.commons.test.TestNGLongTestsHook.run(TestNGLongTestsHook.java:24) ~[infinispan-commons-test-10.0.0-SNAPSHOT.jar:10.0.0-SNAPSHOT]
> at org.testng.internal.MethodInvocationHelper.invokeHookable(MethodInvocationHelper.java:242) ~[testng-6.14.3.jar:?]
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:579) [testng-6.14.3.jar:?]
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:719) [testng-6.14.3.jar:?]
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:989) [testng-6.14.3.jar:?]
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125) [testng-6.14.3.jar:?]
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109) [testng-6.14.3.jar:?]
> at org.testng.TestRunner.privateRun(TestRunner.java:648) [testng-6.14.3.jar:?]
> at org.testng.TestRunner.run(TestRunner.java:505) [testng-6.14.3.jar:?]
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:455) [testng-6.14.3.jar:?]
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:40) [testng-6.14.3.jar:?]
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:489) [testng-6.14.3.jar:?]
> at org.testng.internal.thread.ThreadUtil$1.call(ThreadUtil.java:52) [testng-6.14.3.jar:?]
> at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
> at java.lang.Thread.run(Thread.java:834) [?:?]
> Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1300) ~[narayana-jta-5.9.0.Final.jar:5.9.0.Final (revision: f0935)]
> at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126) ~[narayana-jta-5.9.0.Final.jar:5.9.0.Final (revision: f0935)]
> at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1988) ~[classes/:?]
> ... 28 more
> Suppressed: javax.transaction.xa.XAException
> at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:215) ~[classes/:?]
> at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:161) ~[classes/:?]
> at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:119) ~[classes/:?]
> at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68) ~[classes/:?]
> at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:702) ~[narayana-jta-5.9.0.Final.jar:5.9.0.Final (revision: f0935)]
> at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2386) ~[narayana-jta-5.9.0.Final.jar:5.9.0.Final (revision: f0935)]
> at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1497) ~[narayana-jta-5.9.0.Final.jar:5.9.0.Final (revision: f0935)]
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96) ~[narayana-jta-5.9.0.Final.jar:5.9.0.Final (revision: f0935)]
> at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162) ~[narayana-jta-5.9.0.Final.jar:5.9.0.Final (revision: f0935)]
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1288) ~[narayana-jta-5.9.0.Final.jar:5.9.0.Final (revision: f0935)]
> at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126) ~[narayana-jta-5.9.0.Final.jar:5.9.0.Final (revision: f0935)]
> at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1988) ~[classes/:?]
> at org.infinispan.cache.impl.CacheImpl.transactionalRemoveGroup(CacheImpl.java:651) ~[classes/:?]
> at org.infinispan.cache.impl.CacheImpl.removeGroup(CacheImpl.java:634) ~[classes/:?]
> at org.infinispan.cache.impl.CacheImpl.removeGroup(CacheImpl.java:627) ~[classes/:?]
> at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.removeGroup(AbstractDelegatingAdvancedCache.java:342) ~[classes/:?]
> at org.infinispan.distribution.groups.GetGroupKeysTest.testRemoveGroupKeysWithPersistenceAndPassivation(GetGroupKeysTest.java:165) ~[test-classes/:?]
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124) ~[testng-6.14.3.jar:?]
> at org.testng.internal.MethodInvocationHelper$1.runTestMethod(MethodInvocationHelper.java:230) ~[testng-6.14.3.jar:?]
> at org.infinispan.commons.test.TestNGLongTestsHook.run(TestNGLongTestsHook.java:24) ~[infinispan-commons-test-10.0.0-SNAPSHOT.jar:10.0.0-SNAPSHOT]
> at org.testng.internal.MethodInvocationHelper.invokeHookable(MethodInvocationHelper.java:242) ~[testng-6.14.3.jar:?]
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:579) [testng-6.14.3.jar:?]
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:719) [testng-6.14.3.jar:?]
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:989) [testng-6.14.3.jar:?]
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125) [testng-6.14.3.jar:?]
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109) [testng-6.14.3.jar:?]
> at org.testng.TestRunner.privateRun(TestRunner.java:648) [testng-6.14.3.jar:?]
> at org.testng.TestRunner.run(TestRunner.java:505) [testng-6.14.3.jar:?]
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:455) [testng-6.14.3.jar:?]
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:40) [testng-6.14.3.jar:?]
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:489) [testng-6.14.3.jar:?]
> at org.testng.internal.thread.ThreadUtil$1.call(ThreadUtil.java:52) [testng-6.14.3.jar:?]
> at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
> at java.lang.Thread.run(Thread.java:834) [?:?]
> Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from Test-NodeB-15612, see cause for remote stack trace
> at org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:28) ~[classes/:?]
> at org.infinispan.remoting.transport.impl.MapResponseCollector.addException(MapResponseCollector.java:64) ~[classes/:?]
> at org.infinispan.remoting.transport.impl.MapResponseCollector$IgnoreLeavers.addException(MapResponseCollector.java:102) ~[classes/:?]
> at org.infinispan.remoting.transport.ValidResponseCollector.addResponse(ValidResponseCollector.java:29) ~[classes/:?]
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:91) ~[classes/:?]
> at org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:52) ~[classes/:?]
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1406) ~[classes/:?]
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1309) ~[classes/:?]
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:130) ~[classes/:?]
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1454) ~[classes/:?]
> at org.jgroups.JChannel.up(JChannel.java:775) ~[jgroups-4.1.1.Final.jar:4.1.1.Final]
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:916) ~[jgroups-4.1.1.Final.jar:4.1.1.Final]
> at org.jgroups.protocols.RSVP.up(RSVP.java:164) ~[jgroups-4.1.1.Final.jar:4.1.1.Final]
> at org.jgroups.protocols.FRAG3.up(FRAG3.java:166) ~[jgroups-4.1.1.Final.jar:4.1.1.Final]
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:338) ~[jgroups-4.1.1.Final.jar:4.1.1.Final]
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:338) ~[jgroups-4.1.1.Final.jar:4.1.1.Final]
> at org.jgroups.protocols.tom.TOA.up(TOA.java:119) ~[jgroups-4.1.1.Final.jar:4.1.1.Final]
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:848) ~[jgroups-4.1.1.Final.jar:4.1.1.Final]
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:241) ~[jgroups-4.1.1.Final.jar:4.1.1.Final]
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1019) ~[jgroups-4.1.1.Final.jar:4.1.1.Final]
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:745) ~[jgroups-4.1.1.Final.jar:4.1.1.Final]
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:400) ~[jgroups-4.1.1.Final.jar:4.1.1.Final]
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:590) ~[jgroups-4.1.1.Final.jar:4.1.1.Final]
> at org.jgroups.protocols.Discovery.up(Discovery.java:296) ~[jgroups-4.1.1.Final.jar:4.1.1.Final]
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1265) ~[jgroups-4.1.1.Final.jar:4.1.1.Final]
> at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87) ~[jgroups-4.1.1.Final.jar:4.1.1.Final]
> ... 3 more
> Suppressed: org.infinispan.util.logging.TraceException
> at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41) ~[classes/:?]
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) ~[classes/:?]
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:137) ~[classes/:?]
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) ~[classes/:?]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:202) ~[classes/:?]
> at org.infinispan.interceptors.base.CommandInterceptor.visitCommand(CommandInterceptor.java:163) ~[classes/:?]
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:61) ~[classes/:?]
> at org.infinispan.interceptors.impl.EntryWrappingInterceptor.prepareHandler(EntryWrappingInterceptor.java:193) ~[classes/:?]
> at org.infinispan.interceptors.impl.EntryWrappingInterceptor.wrapEntriesForPrepareAndApply(EntryWrappingInterceptor.java:882) ~[classes/:?]
> at org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:186) ~[classes/:?]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:202) ~[classes/:?]
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) ~[classes/:?]
> at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:53) ~[classes/:?]
> at org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:131) ~[classes/:?]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:202) ~[classes/:?]
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) ~[classes/:?]
> at org.infinispan.interceptors.impl.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:41) ~[classes/:?]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:202) ~[classes/:?]
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) ~[classes/:?]
> at org.infinispan.interceptors.impl.TxInterceptor.handlePrepareCommand(TxInterceptor.java:158) ~[classes/:?]
> at org.infinispan.interceptors.impl.TxInterceptor.visitPrepareCommand(TxInterceptor.java:127) ~[classes/:?]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:202) ~[classes/:?]
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndFinally(BaseAsyncInterceptor.java:155) ~[classes/:?]
> at org.infinispan.interceptors.totalorder.TotalOrderInterceptor.visitPrepareCommand(TotalOrderInterceptor.java:69) ~[classes/:?]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:202) ~[classes/:?]
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) ~[classes/:?]
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) ~[classes/:?]
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:130) ~[classes/:?]
> at org.infinispan.interceptors.totalorder.TotalOrderStateTransferInterceptor.localPrepare(TotalOrderStateTransferInterceptor.java:66) ~[classes/:?]
> at org.infinispan.interceptors.totalorder.TotalOrderStateTransferInterceptor.visitPrepareCommand(TotalOrderStateTransferInterceptor.java:31) ~[classes/:?]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:202) ~[classes/:?]
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) ~[classes/:?]
> at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:53) ~[classes/:?]
> at org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:131) ~[classes/:?]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:202) ~[classes/:?]
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:128) ~[classes/:?]
> at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:89) ~[classes/:?]
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:61) ~[classes/:?]
> at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:53) ~[classes/:?]
> at org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:131) ~[classes/:?]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:202) ~[classes/:?]
> at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:49) ~[classes/:?]
> at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:244) ~[classes/:?]
> at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159) ~[classes/:?]
> at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:119) ~[classes/:?]
> at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68) ~[classes/:?]
> at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:702) ~[narayana-jta-5.9.0.Final.jar:5.9.0.Final (revision: f0935)]
> at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2386) ~[narayana-jta-5.9.0.Final.jar:5.9.0.Final (revision: f0935)]
> at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1497) ~[narayana-jta-5.9.0.Final.jar:5.9.0.Final (revision: f0935)]
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96) ~[narayana-jta-5.9.0.Final.jar:5.9.0.Final (revision: f0935)]
> at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162) ~[narayana-jta-5.9.0.Final.jar:5.9.0.Final (revision: f0935)]
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1288) ~[narayana-jta-5.9.0.Final.jar:5.9.0.Final (revision: f0935)]
> at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126) ~[narayana-jta-5.9.0.Final.jar:5.9.0.Final (revision: f0935)]
> at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1988) ~[classes/:?]
> at org.infinispan.cache.impl.CacheImpl.transactionalRemoveGroup(CacheImpl.java:651) ~[classes/:?]
> at org.infinispan.cache.impl.CacheImpl.removeGroup(CacheImpl.java:634) ~[classes/:?]
> at org.infinispan.cache.impl.CacheImpl.removeGroup(CacheImpl.java:627) ~[classes/:?]
> at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.removeGroup(AbstractDelegatingAdvancedCache.java:342) ~[classes/:?]
> at org.infinispan.distribution.groups.GetGroupKeysTest.testRemoveGroupKeysWithPersistenceAndPassivation(GetGroupKeysTest.java:165) ~[test-classes/:?]
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124) ~[testng-6.14.3.jar:?]
> at org.testng.internal.MethodInvocationHelper$1.runTestMethod(MethodInvocationHelper.java:230) ~[testng-6.14.3.jar:?]
> at org.infinispan.commons.test.TestNGLongTestsHook.run(TestNGLongTestsHook.java:24) ~[infinispan-commons-test-10.0.0-SNAPSHOT.jar:10.0.0-SNAPSHOT]
> at org.testng.internal.MethodInvocationHelper.invokeHookable(MethodInvocationHelper.java:242) ~[testng-6.14.3.jar:?]
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:579) [testng-6.14.3.jar:?]
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:719) [testng-6.14.3.jar:?]
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:989) [testng-6.14.3.jar:?]
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125) [testng-6.14.3.jar:?]
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109) [testng-6.14.3.jar:?]
> at org.testng.TestRunner.privateRun(TestRunner.java:648) [testng-6.14.3.jar:?]
> at org.testng.TestRunner.run(TestRunner.java:505) [testng-6.14.3.jar:?]
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:455) [testng-6.14.3.jar:?]
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:40) [testng-6.14.3.jar:?]
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:489) [testng-6.14.3.jar:?]
> at org.testng.internal.thread.ThreadUtil$1.call(ThreadUtil.java:52) [testng-6.14.3.jar:?]
> at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
> at java.lang.Thread.run(Thread.java:834) [?:?]
> Suppressed: org.infinispan.util.logging.TraceException
> at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41) ~[classes/:?]
> at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:246) ~[classes/:?]
> at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159) ~[classes/:?]
> at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:119) ~[classes/:?]
> at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68) ~[classes/:?]
> at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:702) ~[narayana-jta-5.9.0.Final.jar:5.9.0.Final (revision: f0935)]
> at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2386) ~[narayana-jta-5.9.0.Final.jar:5.9.0.Final (revision: f0935)]
> at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1497) ~[narayana-jta-5.9.0.Final.jar:5.9.0.Final (revision: f0935)]
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96) ~[narayana-jta-5.9.0.Final.jar:5.9.0.Final (revision: f0935)]
> at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162) ~[narayana-jta-5.9.0.Final.jar:5.9.0.Final (revision: f0935)]
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1288) ~[narayana-jta-5.9.0.Final.jar:5.9.0.Final (revision: f0935)]
> at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126) ~[narayana-jta-5.9.0.Final.jar:5.9.0.Final (revision: f0935)]
> at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1988) ~[classes/:?]
> at org.infinispan.cache.impl.CacheImpl.transactionalRemoveGroup(CacheImpl.java:651) ~[classes/:?]
> at org.infinispan.cache.impl.CacheImpl.removeGroup(CacheImpl.java:634) ~[classes/:?]
> at org.infinispan.cache.impl.CacheImpl.removeGroup(CacheImpl.java:627) ~[classes/:?]
> at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.removeGroup(AbstractDelegatingAdvancedCache.java:342) ~[classes/:?]
> at org.infinispan.distribution.groups.GetGroupKeysTest.testRemoveGroupKeysWithPersistenceAndPassivation(GetGroupKeysTest.java:165) ~[test-classes/:?]
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124) ~[testng-6.14.3.jar:?]
> at org.testng.internal.MethodInvocationHelper$1.runTestMethod(MethodInvocationHelper.java:230) ~[testng-6.14.3.jar:?]
> at org.infinispan.commons.test.TestNGLongTestsHook.run(TestNGLongTestsHook.java:24) ~[infinispan-commons-test-10.0.0-SNAPSHOT.jar:10.0.0-SNAPSHOT]
> at org.testng.internal.MethodInvocationHelper.invokeHookable(MethodInvocationHelper.java:242) ~[testng-6.14.3.jar:?]
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:579) [testng-6.14.3.jar:?]
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:719) [testng-6.14.3.jar:?]
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:989) [testng-6.14.3.jar:?]
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125) [testng-6.14.3.jar:?]
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109) [testng-6.14.3.jar:?]
> at org.testng.TestRunner.privateRun(TestRunner.java:648) [testng-6.14.3.jar:?]
> at org.testng.TestRunner.run(TestRunner.java:505) [testng-6.14.3.jar:?]
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:455) [testng-6.14.3.jar:?]
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:40) [testng-6.14.3.jar:?]
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:489) [testng-6.14.3.jar:?]
> at org.testng.internal.thread.ThreadUtil$1.call(ThreadUtil.java:52) [testng-6.14.3.jar:?]
> at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
> at java.lang.Thread.run(Thread.java:834) [?:?]
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.exceptionHandlingCommand(BasePerCacheInboundInvocationHandler.java:155) ~[classes/:?]
> at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.afterCommandException(BaseBlockingRunnable.java:150) ~[classes/:?]
> at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runSync(BaseBlockingRunnable.java:59) ~[classes/:?]
> at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:39) ~[classes/:?]
> ... 3 more
> Caused by: java.lang.AssertionError: Thread name is: remote-thread-Test-NodeB-p3529-t2
> at org.infinispan.persistence.manager.PersistenceManagerImpl.deleteFromAllStoresSync(PersistenceManagerImpl.java:606) ~[classes/:?]
> at org.infinispan.eviction.impl.ActivationManagerImpl.onRemove(ActivationManagerImpl.java:103) ~[classes/:?]
> at org.infinispan.container.impl.AbstractInternalDataContainer.lambda$remove$1(AbstractInternalDataContainer.java:205) ~[classes/:?]
> at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$remap$16(BoundedLocalCache.java:2199) ~[caffeine-2.6.2.jar:?]
> at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1908) ~[?:?]
> at com.github.benmanes.caffeine.cache.BoundedLocalCache.remap(BoundedLocalCache.java:2194) ~[caffeine-2.6.2.jar:?]
> at com.github.benmanes.caffeine.cache.BoundedLocalCache.compute(BoundedLocalCache.java:2146) ~[caffeine-2.6.2.jar:?]
> at com.github.benmanes.caffeine.cache.LocalCache.compute(LocalCache.java:100) ~[caffeine-2.6.2.jar:?]
> at org.infinispan.container.impl.AbstractInternalDataContainer.remove(AbstractInternalDataContainer.java:204) ~[classes/:?]
> at org.infinispan.container.entries.ReadCommittedEntry.commit(ReadCommittedEntry.java:141) ~[classes/:?]
> at org.infinispan.statetransfer.CommitManager.commitEntry(CommitManager.java:130) ~[classes/:?]
> at org.infinispan.statetransfer.CommitManager.commit(CommitManager.java:98) ~[classes/:?]
> at org.infinispan.interceptors.locking.ClusteringDependentLogic$DistributionLogic.commitSingleEntry(ClusteringDependentLogic.java:627) ~[classes/:?]
> at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.lambda$commitEntry$2(ClusteringDependentLogic.java:238) ~[classes/:?]
> at org.infinispan.commons.util.concurrent.CallerRunsRejectOnShutdownPolicy.rejectedExecution(CallerRunsRejectOnShutdownPolicy.java:19) ~[infinispan-commons-10.0.0-SNAPSHOT.jar:10.0.0-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825) ~[?:?]
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355) ~[?:?]
> at org.infinispan.executors.LazyInitializingExecutorService.execute(LazyInitializingExecutorService.java:138) ~[classes/:?]
> at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.commitEntry(ClusteringDependentLogic.java:238) ~[classes/:?]
> at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:602) ~[classes/:?]
> at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:847) ~[classes/:?]
> at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:588) ~[classes/:?]
> at org.infinispan.interceptors.impl.EntryWrappingInterceptor.lambda$new$2(EntryWrappingInterceptor.java:153) ~[classes/:?]
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:86) ~[classes/:?]
> at org.infinispan.interceptors.impl.EntryWrappingInterceptor.prepareHandler(EntryWrappingInterceptor.java:191) ~[classes/:?]
> at org.infinispan.interceptors.impl.EntryWrappingInterceptor.applyModificationsAndThen(EntryWrappingInterceptor.java:905) ~[classes/:?]
> at org.infinispan.interceptors.impl.EntryWrappingInterceptor.wrapEntriesForPrepareAndApply(EntryWrappingInterceptor.java:869) ~[classes/:?]
> at org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:186) ~[classes/:?]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:202) ~[classes/:?]
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) ~[classes/:?]
> at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:53) ~[classes/:?]
> at org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:131) ~[classes/:?]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:202) ~[classes/:?]
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) ~[classes/:?]
> at org.infinispan.interceptors.impl.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:41) ~[classes/:?]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:202) ~[classes/:?]
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:188) ~[classes/:?]
> at org.infinispan.interceptors.impl.TxInterceptor.handlePrepareCommand(TxInterceptor.java:138) ~[classes/:?]
> at org.infinispan.interceptors.impl.TxInterceptor.visitPrepareCommand(TxInterceptor.java:127) ~[classes/:?]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:202) ~[classes/:?]
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndFinally(BaseAsyncInterceptor.java:155) ~[classes/:?]
> at org.infinispan.interceptors.totalorder.TotalOrderInterceptor.visitPrepareCommand(TotalOrderInterceptor.java:95) ~[classes/:?]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:202) ~[classes/:?]
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) ~[classes/:?]
> at org.infinispan.interceptors.BaseAsyncInterceptor.asyncInvokeNext(BaseAsyncInterceptor.java:232) ~[classes/:?]
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:46) ~[classes/:?]
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:61) ~[classes/:?]
> at org.infinispan.interceptors.totalorder.TotalOrderStateTransferInterceptor.remotePrepare(TotalOrderStateTransferInterceptor.java:55) ~[classes/:?]
> at org.infinispan.interceptors.totalorder.TotalOrderStateTransferInterceptor.visitPrepareCommand(TotalOrderStateTransferInterceptor.java:33) ~[classes/:?]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:202) ~[classes/:?]
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) ~[classes/:?]
> at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:53) ~[classes/:?]
> at org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:131) ~[classes/:?]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:202) ~[classes/:?]
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:128) ~[classes/:?]
> at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:89) ~[classes/:?]
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:61) ~[classes/:?]
> at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:53) ~[classes/:?]
> at org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:131) ~[classes/:?]
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:202) ~[classes/:?]
> at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:49) ~[classes/:?]
> at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:226) ~[classes/:?]
> at org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:122) ~[classes/:?]
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:118) ~[classes/:?]
> at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runSync(BaseBlockingRunnable.java:55) ~[classes/:?]
> at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:39) ~[classes/:?]
> {code}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 6 months
[JBoss JIRA] (ISPN-9257) ClustertopologyManagerTest.testAbruptLeaveAfterGetStatus2[SCATTERED_SYNC, tx=false] random failures
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-9257?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-9257:
-------------------------------
Git Pull Request: https://github.com/infinispan/infinispan/pull/7094, https://github.com/infinispan/infinispan/pull/7097 (was: https://github.com/infinispan/infinispan/pull/7094)
> ClustertopologyManagerTest.testAbruptLeaveAfterGetStatus2[SCATTERED_SYNC, tx=false] random failures
> ---------------------------------------------------------------------------------------------------
>
> Key: ISPN-9257
> URL: https://issues.jboss.org/browse/ISPN-9257
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 9.3.0.CR1
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Minor
> Labels: testsuite_stability
> Fix For: 10.0.0.Beta4, 9.4.16.Final
>
> Attachments: ISPN-8731_wrong_topology_2018-05-18_ClusterTopologyManagerTest-infinispan-core.log.gz
>
>
> The test kills the coordinator NodeA, then while NodeB is trying to recover the caches it also kills NodeC. It expects NodeB to start a rebalance with 2 nodes and discards it, in order to test that it can process the 1-node rebalance first:
> {noformat}
> 00:34:06,582 DEBUG (transport-thread-test-NodeB-p12-t6:[testCache]) [ClusterTopologyManagerTest] Discarding rebalance command CacheTopology{id=8, phase=TRANSITORY, rebalanceId=5, currentCH=ScatteredConsistentHash{ns=256, rebalanced=false, owners = (2)[test-NodeB-49590: 85, test-NodeC-58596: 85]}, pendingCH=ScatteredConsistentHash{ns=256, rebalanced=true, owners = (2)[test-NodeB-49590: 128, test-NodeC-58596: 128]}, unionCH=null, actualMembers=[test-NodeB-49590, test-NodeC-58596], persistentUUIDs=[6b96414e-15d8-4350-aa3c-4fb4fc34e888, d47dc4a9-2a95-4bb1-a83b-bb8a27c9999f]}
> 00:34:06,609 DEBUG (transport-thread-test-NodeB-p12-t2:[Topology-testCache]) [LocalTopologyManagerImpl] Updating local topology for cache testCache: CacheTopology{id=9, phase=TRANSITORY, rebalanceId=5, currentCH=ScatteredConsistentHash{ns=256, rebalanced=false, owners = (1)[test-NodeB-49590: 85]}, pendingCH=ScatteredConsistentHash{ns=256, rebalanced=false, owners = (1)[test-NodeB-49590: 128]}, unionCH=null, actualMembers=[test-NodeB-49590], persistentUUIDs=[6b96414e-15d8-4350-aa3c-4fb4fc34e888]}
> 00:34:06,609 DEBUG (transport-thread-test-NodeB-p12-t2:[Topology-testCache]) [LocalTopologyManagerImpl] Installing fake cache topology CacheTopology{id=8, phase=NO_REBALANCE, rebalanceId=4, currentCH=ScatteredConsistentHash{ns=256, rebalanced=false, owners = (1)[test-NodeB-49590: 85]}, pendingCH=null, unionCH=null, actualMembers=[test-NodeB-49590], persistentUUIDs=[6b96414e-15d8-4350-aa3c-4fb4fc34e888]} for cache testCache
> {noformat}
> Unfortunately {{PreferAvailabilityStrategy}} has changed a bit and the rebalance ids don't always match the expectations of the test, so that the 1-node rebalance is discarded instead:
> {noformat}
> 09:46:10,530 DEBUG (transport-thread-Test-NodeB-p54539-t3:[testCache]) [Test] Discarding rebalance command CacheTopology{id=9, phase=TRANSITORY, rebalanceId=5, currentCH=ScatteredConsistentHash{ns=256, rebalanced=false, owners = (1)[Test-NodeB-62039: 85]}, pendingCH=ScatteredConsistentHash{ns=256, rebalanced=true, owners = (1)[Test-NodeB-62039: 256]}, unionCH=null, actualMembers=[Test-NodeB-62039], persistentUUIDs=[0ed7be74-4485-489b-baee-28c461c9e5de]}
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 6 months
[JBoss JIRA] (ISPN-8329) ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus2 random failures with scattered cache
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-8329?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-8329:
-------------------------------
Git Pull Request: https://github.com/infinispan/infinispan/pull/7094, https://github.com/infinispan/infinispan/pull/7097 (was: https://github.com/infinispan/infinispan/pull/7094)
> ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus2 random failures with scattered cache
> ----------------------------------------------------------------------------------------------
>
> Key: ISPN-8329
> URL: https://issues.jboss.org/browse/ISPN-8329
> Project: Infinispan
> Issue Type: Enhancement
> Components: Core
> Reporter: Tristan Tarrant
> Assignee: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Fix For: 10.0.0.Beta4, 9.4.16.Final
>
>
> Error Message
> Timed out waiting for rebalancing to complete on node ClusterTopologyManagerTest[SCATTERED_SYNC, tx=false]-NodeB-47665, current topology is CacheTopology{id=8, rebalanceId=4, currentCH=PartitionerConsistentHash:ScatteredConsistentHash{ns=256, rebalanced=false, owners = (1)[ClusterTopologyManagerTest[SCATTERED_SYNC, tx=false]-NodeB-47665: 85]}, pendingCH=null, unionCH=null, phase=NO_REBALANCE, actualMembers=[ClusterTopologyManagerTest[SCATTERED_SYNC, tx=false]-NodeB-47665], persistentUUIDs=[63b3a997-f229-475b-a14c-9c892f608ba0]}. rebalanceInProgress=false, currentChIsBalanced=false
> Stacktrace
> java.lang.RuntimeException: Timed out waiting for rebalancing to complete on node ClusterTopologyManagerTest[SCATTERED_SYNC, tx=false]-NodeB-47665, current topology is CacheTopology{id=8, rebalanceId=4, currentCH=PartitionerConsistentHash:ScatteredConsistentHash{ns=256, rebalanced=false, owners = (1)[ClusterTopologyManagerTest[SCATTERED_SYNC, tx=false]-NodeB-47665: 85]}, pendingCH=null, unionCH=null, phase=NO_REBALANCE, actualMembers=[ClusterTopologyManagerTest[SCATTERED_SYNC, tx=false]-NodeB-47665], persistentUUIDs=[63b3a997-f229-475b-a14c-9c892f608ba0]}. rebalanceInProgress=false, currentChIsBalanced=false
> at org.infinispan.test.TestingUtil.waitForNoRebalance(TestingUtil.java:386)
> at org.infinispan.statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus2(ClusterTopologyManagerTest.java:430)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> ... Removed 16 stack frames
>
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 6 months