[JBoss JIRA] (ISPN-9127) Remote commands can access components before they are started
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-9127?page=com.atlassian.jira.plugin.... ]
Work on ISPN-9127 started by Dan Berindei.
------------------------------------------
> Remote commands can access components before they are started
> -------------------------------------------------------------
>
> Key: ISPN-9127
> URL: https://issues.jboss.org/browse/ISPN-9127
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.2.2.Final, 9.3.0.Alpha1
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Labels: testsuite_stability
>
> {{PerCacheInboundInvocationHandler.handle()}} may be called before the component was started, because {{GlobalInboundInvocationHandler}} fetches it from the component registry without any checks. {{CommandsFactoryImpl.initializeReplicableCommand()}} doesn't wait for the components that it injects into remote commands to be started, either.
> This started causing random test failures in {{ConcurrentStartForkChannelTest}} after ISPN-8515, which moved most initialization work from {{init()}} methods to {{start()}} methods. Because {{StateProviderImpl}} starts after {{StateTransferManagerImpl}}, it's possible for a node to receive a {{StateRequestCommand}} before {{StateProviderImpl}} has initialized:
> {noformat}
> 16:15:09,549 TRACE (remote-thread-Test-NodeB-p51957-t2:[org.infinispan.CONFIG]) [StateProviderImpl] Starting outbound transfer to node Test-NodeA for cache null, topology id 2, segments {0-255}
> 16:15:09,551 WARN (remote-thread-Test-NodeB-p51957-t2:[]) [NonTotalOrderPerCacheInboundInvocationHandler] ISPN000071: Caught exception when handling command StateRequestCommand{cache=org.infinispan.CONFIG, origin=Test-NodeA, type=START_STATE_TRANSFER, topologyId=2, segments={0-255}}
> java.lang.IllegalArgumentException: chunkSize must be greater than 0
> at org.infinispan.statetransfer.OutboundTransferTask.<init>(OutboundTransferTask.java:114) ~[classes/:?]
> at org.infinispan.statetransfer.StateProviderImpl.startOutboundTransfer(StateProviderImpl.java:273) ~[classes/:?]
> at org.infinispan.statetransfer.StateRequestCommand.invokeAsync(StateRequestCommand.java:101) ~[classes/:?]
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94) ~[classes/:?]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months
[JBoss JIRA] (ISPN-9512) *TxPartitionAndMerge*Test tests hang during teardown
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-9512?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-9512:
----------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
> *TxPartitionAndMerge*Test tests hang during teardown
> ----------------------------------------------------
>
> Key: ISPN-9512
> URL: https://issues.jboss.org/browse/ISPN-9512
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Labels: testsuite_stability
> Fix For: 9.4.0.CR3
>
> Attachments: master_20180913-1119_PessimisticTxPartitionAndMergeDuringRollbackTest-infinispan-core.log.gz, threaddump-org_infinispan_partitionhandling_PessimisticTxPartitionAndMergeDuringRollbackTest_clearContent-2018-09-13-13828.log
>
>
> Not sure what changed recently, but the thread dumps show a state transfer executor thread blocked waiting for a clustered listeners response. The stack includes two instances of {{ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution()}}, which suggests that at some point all the state transfer executor threads (6) and async transport threads (4) were busy, and the transport thread pool queue (10) was also full.
> {noformat}
> "stateTransferExecutor-thread-PessimisticTxPartitionAndMergeDuringRollbackTest-NodeC-p57758-t1" #192601 daemon prio=5 os_prio=0 tid=0x00007f7094031800 nid=0x5b27 waiting on condition [0x00007f70190ce000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000d470b0f8> (a java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1695)
> at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1775)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
> at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:93)
> at org.infinispan.remoting.rpc.RpcManagerImpl.blocking(RpcManagerImpl.java:262)
> at org.infinispan.statetransfer.StateConsumerImpl.getClusterListeners(StateConsumerImpl.java:895)
> at org.infinispan.statetransfer.StateConsumerImpl.fetchClusterListeners(StateConsumerImpl.java:453)
> at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:309)
> at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:197)
> at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:54)
> at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:117)
> at org.infinispan.topology.LocalTopologyManagerImpl.doHandleRebalance(LocalTopologyManagerImpl.java:517)
> - locked <0x00000000cc304f88> (a org.infinispan.topology.LocalCacheStatus)
> at org.infinispan.topology.LocalTopologyManagerImpl.lambda$handleRebalance$3(LocalTopologyManagerImpl.java:475)
> at org.infinispan.topology.LocalTopologyManagerImpl$$Lambda$429/1368424830.run(Unknown Source)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:37)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:227)
> at java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:2038)
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
> at org.infinispan.executors.LazyInitializingExecutorService.execute(LazyInitializingExecutorService.java:121)
> at org.infinispan.executors.LimitedExecutor.tryExecute(LimitedExecutor.java:151)
> at org.infinispan.executors.LimitedExecutor.executeInternal(LimitedExecutor.java:118)
> at org.infinispan.executors.LimitedExecutor.execute(LimitedExecutor.java:108)
> at org.infinispan.topology.LocalTopologyManagerImpl.handleRebalance(LocalTopologyManagerImpl.java:473)
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:199)
> at org.infinispan.topology.CacheTopologyControlCommand.invokeAsync(CacheTopologyControlCommand.java:160)
> at org.infinispan.commands.ReplicableCommand.invoke(ReplicableCommand.java:44)
> at org.infinispan.topology.ClusterTopologyManagerImpl.lambda$executeOnClusterAsync$5(ClusterTopologyManagerImpl.java:600)
> at org.infinispan.topology.ClusterTopologyManagerImpl$$Lambda$304/909965247.run(Unknown Source)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:2038)
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
> at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
> at org.infinispan.executors.LazyInitializingExecutorService.submit(LazyInitializingExecutorService.java:91)
> at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterAsync(ClusterTopologyManagerImpl.java:596)
> at org.infinispan.topology.ClusterTopologyManagerImpl.broadcastRebalanceStart(ClusterTopologyManagerImpl.java:437)
> at org.infinispan.topology.ClusterCacheStatus.startQueuedRebalance(ClusterCacheStatus.java:903)
> - locked <0x00000000cc305138> (a org.infinispan.topology.ClusterCacheStatus)
> at org.infinispan.topology.ClusterCacheStatus.queueRebalance(ClusterCacheStatus.java:140)
> - locked <0x00000000cc305138> (a org.infinispan.topology.ClusterCacheStatus)
> at org.infinispan.partitionhandling.impl.PreferConsistencyStrategy.updateMembersAndRebalance(PreferConsistencyStrategy.java:299)
> at org.infinispan.partitionhandling.impl.PreferConsistencyStrategy.onPartitionMerge(PreferConsistencyStrategy.java:245)
> at org.infinispan.topology.ClusterCacheStatus.doMergePartitions(ClusterCacheStatus.java:642)
> - locked <0x00000000cc305138> (a org.infinispan.topology.ClusterCacheStatus)
> at org.infinispan.topology.ClusterTopologyManagerImpl.lambda$recoverClusterStatus$4(ClusterTopologyManagerImpl.java:494)
> at org.infinispan.topology.ClusterTopologyManagerImpl$$Lambda$578/46555845.run(Unknown Source)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:37)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:227)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> All partition and merge tests seem to be affected: PessimisticTxPartitionAndMergeDuringPrepareTest, PessimisticTxPartitionAndMergeDuringRollbackTest, PessimisticTxPartitionAndMergeDuringRuntimeTest, OptimisticTxPartitionAndMergeDuringCommitTest, OptimisticTxPartitionAndMergeDuringPrepareTest, and OptimisticTxPartitionAndMergeDuringRollbackTest.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months
[JBoss JIRA] (ISPN-2575) Key Transformer registration is required on all nodes of the cluster, in case of custom keys
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-2575?page=com.atlassian.jira.plugin.... ]
Adrian Nistor commented on ISPN-2575:
-------------------------------------
The PR fixes this by offering an alternative to SearchManagerImplementor.registerKeyTransformer, but SearchManagerImplementor.registerKeyTransformer continues to cause issues and will be addressed in ISPN-9513.
> Key Transformer registration is required on all nodes of the cluster, in case of custom keys
> --------------------------------------------------------------------------------------------
>
> Key: ISPN-2575
> URL: https://issues.jboss.org/browse/ISPN-2575
> Project: Infinispan
> Issue Type: Enhancement
> Components: Embedded Querying
> Affects Versions: 5.2.0.Beta5
> Reporter: Anna Manukyan
> Assignee: Adrian Nistor
> Priority: Minor
> Fix For: 9.4.0.CR3
>
> Attachments: ClusteredCacheTest.java
>
>
> The case is the following:
> I have a clustered cache on which I want to perform a search. I'm doing the following:
> I'm initializing SearchManager on node1, I'm registering a custom key transformer for my key using the created searchmanager, but then when I'm trying to put data into the cache on node1 (which is in REPL_SYNC mode with cache on node2), I'm getting the exception:
> java.lang.IllegalArgumentException: Indexing only works with entries keyed on Strings, primitives and classes that have the @Transformable annotation - you passed in a class org.infinispan.query.test.CustomKey3. Alternatively, see org.infinispan.query.SearchManager#registerKeyTransformer
> When I'm initializing the SearchManager using node2 cache and register the keyTransformer on it as well, then everything works perfectly, even though I am not using the second created SearchManager.
> The test which reproduces the issue is attached to the jira. Please see the test case: testSearchKeyTransformer()
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months
[JBoss JIRA] (ISPN-2575) Key Transformer registration is required on all nodes of the cluster, in case of custom keys
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-2575?page=com.atlassian.jira.plugin.... ]
Adrian Nistor updated ISPN-2575:
--------------------------------
Summary: Key Transformer registration is required on all nodes of the cluster, in case of custom keys (was: KeyTransformer registration is required on all nodes of the cluster, in case of custom keys)
> Key Transformer registration is required on all nodes of the cluster, in case of custom keys
> --------------------------------------------------------------------------------------------
>
> Key: ISPN-2575
> URL: https://issues.jboss.org/browse/ISPN-2575
> Project: Infinispan
> Issue Type: Enhancement
> Components: Embedded Querying
> Affects Versions: 5.2.0.Beta5
> Reporter: Anna Manukyan
> Assignee: Adrian Nistor
> Priority: Minor
> Fix For: 9.4.0.CR3
>
> Attachments: ClusteredCacheTest.java
>
>
> The case is the following:
> I have a clustered cache on which I want to perform a search. I'm doing the following:
> I'm initializing SearchManager on node1, I'm registering a custom key transformer for my key using the created searchmanager, but then when I'm trying to put data into the cache on node1 (which is in REPL_SYNC mode with cache on node2), I'm getting the exception:
> java.lang.IllegalArgumentException: Indexing only works with entries keyed on Strings, primitives and classes that have the @Transformable annotation - you passed in a class org.infinispan.query.test.CustomKey3. Alternatively, see org.infinispan.query.SearchManager#registerKeyTransformer
> When I'm initializing the SearchManager using node2 cache and register the keyTransformer on it as well, then everything works perfectly, even though I am not using the second created SearchManager.
> The test which reproduces the issue is attached to the jira. Please see the test case: testSearchKeyTransformer()
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months