[JBoss JIRA] (ISPN-9512) *TxPartitionAndMerge*Test tests hang during teardown
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-9512?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-9512:
------------------------------------
Oops, I reduced the number of transport threads and the size of the queue in ISPN-9067 (https://github.com/infinispan/infinispan/commit/ef9735d778a9e687f272b8db3...), and that's what's causing the tests to hang.
I see 2 problems that we should tackle here:
# {{ClusterTopologyManagerImpl.executeOnClusterAsync()}} invokes the command on the local node before sending it to the other nodes. This probably made sense back when the remote invocation could only be synchronous, but now it would be better to send the command first, then execute on the local node, and then wait for responses.
# {{ClusterCacheStatus.doMergePartitions()}} should not hold the {{ClusterCacheStatus}} monitor while broadcasting the rebalance command, because there's always a chance that the local invocation will happen on the caller thread.
Going further we need to move away from blocking RPCs both in {{ClusterTopologyManagerImpl}} (ISPN-8955) and in {{StateConsumerImpl}}, and also think about unifying the thread pools but using {{LimitedExecutor}} e.g. to apply state.
> *TxPartitionAndMerge*Test tests hang during teardown
> ----------------------------------------------------
>
> Key: ISPN-9512
> URL: https://issues.jboss.org/browse/ISPN-9512
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Labels: testsuite_stability
> Fix For: 9.4.0.CR3
>
> Attachments: master_20180913-1119_PessimisticTxPartitionAndMergeDuringRollbackTest-infinispan-core.log.gz, threaddump-org_infinispan_partitionhandling_PessimisticTxPartitionAndMergeDuringRollbackTest_clearContent-2018-09-13-13828.log
>
>
> Not sure what changed recently, but the thread dumps show a state transfer executor thread blocked waiting for a clustered listeners response. The stack includes two instances of {{ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution()}}, which suggests that at some point all the state transfer executor threads (6) and async transport threads (4) were busy, and the transport thread pool queue (10) was also full.
> {noformat}
> "stateTransferExecutor-thread-PessimisticTxPartitionAndMergeDuringRollbackTest-NodeC-p57758-t1" #192601 daemon prio=5 os_prio=0 tid=0x00007f7094031800 nid=0x5b27 waiting on condition [0x00007f70190ce000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000d470b0f8> (a java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1695)
> at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1775)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
> at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:93)
> at org.infinispan.remoting.rpc.RpcManagerImpl.blocking(RpcManagerImpl.java:262)
> at org.infinispan.statetransfer.StateConsumerImpl.getClusterListeners(StateConsumerImpl.java:895)
> at org.infinispan.statetransfer.StateConsumerImpl.fetchClusterListeners(StateConsumerImpl.java:453)
> at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:309)
> at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:197)
> at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:54)
> at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:117)
> at org.infinispan.topology.LocalTopologyManagerImpl.doHandleRebalance(LocalTopologyManagerImpl.java:517)
> - locked <0x00000000cc304f88> (a org.infinispan.topology.LocalCacheStatus)
> at org.infinispan.topology.LocalTopologyManagerImpl.lambda$handleRebalance$3(LocalTopologyManagerImpl.java:475)
> at org.infinispan.topology.LocalTopologyManagerImpl$$Lambda$429/1368424830.run(Unknown Source)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:37)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:227)
> at java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:2038)
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
> at org.infinispan.executors.LazyInitializingExecutorService.execute(LazyInitializingExecutorService.java:121)
> at org.infinispan.executors.LimitedExecutor.tryExecute(LimitedExecutor.java:151)
> at org.infinispan.executors.LimitedExecutor.executeInternal(LimitedExecutor.java:118)
> at org.infinispan.executors.LimitedExecutor.execute(LimitedExecutor.java:108)
> at org.infinispan.topology.LocalTopologyManagerImpl.handleRebalance(LocalTopologyManagerImpl.java:473)
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:199)
> at org.infinispan.topology.CacheTopologyControlCommand.invokeAsync(CacheTopologyControlCommand.java:160)
> at org.infinispan.commands.ReplicableCommand.invoke(ReplicableCommand.java:44)
> at org.infinispan.topology.ClusterTopologyManagerImpl.lambda$executeOnClusterAsync$5(ClusterTopologyManagerImpl.java:600)
> at org.infinispan.topology.ClusterTopologyManagerImpl$$Lambda$304/909965247.run(Unknown Source)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:2038)
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
> at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
> at org.infinispan.executors.LazyInitializingExecutorService.submit(LazyInitializingExecutorService.java:91)
> at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterAsync(ClusterTopologyManagerImpl.java:596)
> at org.infinispan.topology.ClusterTopologyManagerImpl.broadcastRebalanceStart(ClusterTopologyManagerImpl.java:437)
> at org.infinispan.topology.ClusterCacheStatus.startQueuedRebalance(ClusterCacheStatus.java:903)
> - locked <0x00000000cc305138> (a org.infinispan.topology.ClusterCacheStatus)
> at org.infinispan.topology.ClusterCacheStatus.queueRebalance(ClusterCacheStatus.java:140)
> - locked <0x00000000cc305138> (a org.infinispan.topology.ClusterCacheStatus)
> at org.infinispan.partitionhandling.impl.PreferConsistencyStrategy.updateMembersAndRebalance(PreferConsistencyStrategy.java:299)
> at org.infinispan.partitionhandling.impl.PreferConsistencyStrategy.onPartitionMerge(PreferConsistencyStrategy.java:245)
> at org.infinispan.topology.ClusterCacheStatus.doMergePartitions(ClusterCacheStatus.java:642)
> - locked <0x00000000cc305138> (a org.infinispan.topology.ClusterCacheStatus)
> at org.infinispan.topology.ClusterTopologyManagerImpl.lambda$recoverClusterStatus$4(ClusterTopologyManagerImpl.java:494)
> at org.infinispan.topology.ClusterTopologyManagerImpl$$Lambda$578/46555845.run(Unknown Source)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:37)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:227)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> All partition and merge tests seem to be affected: PessimisticTxPartitionAndMergeDuringPrepareTest, PessimisticTxPartitionAndMergeDuringRollbackTest, PessimisticTxPartitionAndMergeDuringRuntimeTest, OptimisticTxPartitionAndMergeDuringCommitTest, OptimisticTxPartitionAndMergeDuringPrepareTest, and OptimisticTxPartitionAndMergeDuringRollbackTest.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months
[JBoss JIRA] (ISPN-6161) [Remote query] Indexes should be created only for annotated message types
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-6161?page=com.atlassian.jira.plugin.... ]
Adrian Nistor updated ISPN-6161:
--------------------------------
Summary: [Remote query] Indexes should be created only for annotated message types (was: Indexes should be created only for annotated message types)
> [Remote query] Indexes should be created only for annotated message types
> -------------------------------------------------------------------------
>
> Key: ISPN-6161
> URL: https://issues.jboss.org/browse/ISPN-6161
> Project: Infinispan
> Issue Type: Task
> Components: Remote Querying
> Reporter: Tristan Tarrant
> Assignee: Adrian Nistor
> Fix For: 10.0.0.Final
>
>
> Modify behaviour of indexing for protobuf entities: Current behaviour is: if the message types is not annotated then we index all its fields; if some fields are annotated then index those fields only. New behaviour: index only the annotated fields.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months
[JBoss JIRA] (ISPN-2575) KeyTransformer registration is required on all nodes of the cluster, in case of custom keys
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-2575?page=com.atlassian.jira.plugin.... ]
Adrian Nistor updated ISPN-2575:
--------------------------------
Fix Version/s: 9.4.0.CR3
(was: 9.4.0.Final)
> KeyTransformer registration is required on all nodes of the cluster, in case of custom keys
> -------------------------------------------------------------------------------------------
>
> Key: ISPN-2575
> URL: https://issues.jboss.org/browse/ISPN-2575
> Project: Infinispan
> Issue Type: Enhancement
> Components: Embedded Querying
> Affects Versions: 5.2.0.Beta5
> Reporter: Anna Manukyan
> Assignee: Adrian Nistor
> Priority: Minor
> Fix For: 9.4.0.CR3
>
> Attachments: ClusteredCacheTest.java
>
>
> The case is the following:
> I have a clustered cache on which I want to perform a search. I'm doing the following:
> I'm initializing SearchManager on node1, I'm registering a custom key transformer for my key using the created searchmanager, but then when I'm trying to put data into the cache on node1 (which is in REPL_SYNC mode with cache on node2), I'm getting the exception:
> java.lang.IllegalArgumentException: Indexing only works with entries keyed on Strings, primitives and classes that have the @Transformable annotation - you passed in a class org.infinispan.query.test.CustomKey3. Alternatively, see org.infinispan.query.SearchManager#registerKeyTransformer
> When I'm initializing the SearchManager using node2 cache and register the keyTransformer on it as well, then everything works perfectly, even though I am not using the second created SearchManager.
> The test which reproduces the issue is attached to the jira. Please see the test case: testSearchKeyTransformer()
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months
[JBoss JIRA] (ISPN-4443) Key Transformer cannot be registered early enough
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-4443?page=com.atlassian.jira.plugin.... ]
Adrian Nistor commented on ISPN-4443:
-------------------------------------
Both this and ISPN-2575 will be fixed by adding a new element in the xml config that allows registration of key transformers upfront, so they will be there when the cache starts.
> Key Transformer cannot be registered early enough
> -------------------------------------------------
>
> Key: ISPN-4443
> URL: https://issues.jboss.org/browse/ISPN-4443
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 6.0.2.Final
> Reporter: Rich DiCroce
> Assignee: Adrian Nistor
>
> I have a replicated mode cache that has an index stored on each node using the RAM directory provider. I can't annotate the key class with @Transformable, so I'm using SearchManagerImplementor#registerKeyTransformer() instead. When the second node starts up, I get the below exception for about 50 milliseconds because the first node is putting stuff in the cache, and the cache has been started on the second node, but the Transformer hasn't been registered yet.
> As far as I can tell, this is a catch-22 that is impossible to avoid. I have to obtain a Cache in order to register the Transformer, but getting the cache causes the cache to be started before I can register the Transformer. The exceptions stop after the Transformer is registered, but I don't know what the state of the index is relative to the contents of the cache. My application will have a serious problem if an entry is in the cache but not the index.
> This problem is similar to ISPN-2575, but not the same.
> {code}
> 2014-06-25 17:51:48,601 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (remote-thread-0) ISPN000136: Execution error: java.lang.IllegalArgumentException: Indexing only works with entries keyed on Strings, primitives and classes that have the @Transformable annotation - you passed in a class com.sgi.song.common.message.SONGv1.address.EndpointAddress. Alternatively, see org.infinispan.query.SearchManager#registerKeyTransformer
> at org.infinispan.query.backend.KeyTransformationHandler.keyToString(KeyTransformationHandler.java:153)
> at org.infinispan.query.backend.QueryInterceptor.keyToString(QueryInterceptor.java:332)
> at org.infinispan.query.backend.QueryInterceptor.updateIndexes(QueryInterceptor.java:221)
> at org.infinispan.query.backend.QueryInterceptor.processPutKeyValueCommand(QueryInterceptor.java:518)
> at org.infinispan.query.backend.QueryInterceptor.visitPutKeyValueCommand(QueryInterceptor.java:164)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPutKeyValueCommand(PessimisticLockingInterceptor.java:111)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:255)
> at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:186)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:206)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:141)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:148)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:134)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
> at org.infinispan.statetransfer.StateConsumerImpl.doApplyState(StateConsumerImpl.java:543)
> at org.infinispan.statetransfer.StateConsumerImpl.applyState(StateConsumerImpl.java:495)
> at org.infinispan.statetransfer.StateResponseCommand.perform(StateResponseCommand.java:62)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:95)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:50)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:172)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [rt.jar:1.8.0_05]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [rt.jar:1.8.0_05]
> at java.lang.Thread.run(Unknown Source) [rt.jar:1.8.0_05]
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months