[JBoss JIRA] (ISPN-6118) CacheNotifierImplInitialTransferDistTest random failures
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-6118?page=com.atlassian.jira.plugin.... ]
William Burns updated ISPN-6118:
--------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
> CacheNotifierImplInitialTransferDistTest random failures
> --------------------------------------------------------
>
> Key: ISPN-6118
> URL: https://issues.jboss.org/browse/ISPN-6118
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 8.1.0.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Labels: testsuite_stability
> Fix For: 9.0.0.Alpha1
>
>
> The failures are probably related to the default consistent hash factory change.
> {noformat}
> 18:54:56,255 ERROR (testng-CacheNotifierImplInitialTransferDistTest:) [UnitTestTestNGListener] Test testCreateAfterIterationBeganAndSegmentNotCompleteValueNonOwnerClustered(org.infinispan.notifications.cachelistener.CacheNotifierImplInitialTransferDistTest) failed.
> java.util.concurrent.TimeoutException: Timed out waiting for event pre_complete_segment_invoked
> at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:43) ~[test-classes/:?]
> at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:33) ~[test-classes/:?]
> at org.infinispan.notifications.cachelistener.CacheNotifierImplInitialTransferDistTest.testIterationBeganAndSegmentNotComplete(CacheNotifierImplInitialTransferDistTest.java:519) ~[test-classes/:?]
> at org.infinispan.notifications.cachelistener.CacheNotifierImplInitialTransferDistTest.testCreateAfterIterationBeganAndSegmentNotCompleteValueNonOwnerClustered(CacheNotifierImplInitialTransferDistTest.java:622) ~[test-classes/:?]
> {noformat}
> {noformat}
> 18:55:06,637 ERROR (testng-CacheNotifierImplInitialTransferDistTest:) [UnitTestTestNGListener] Test testCreateAfterIterationBeganAndSegmentNotCompleteValueOwnerClustered(org.infinispan.notifications.cachelistener.CacheNotifierImplInitialTransferDistTest) failed.
> java.util.concurrent.TimeoutException: Timed out waiting for event pre_complete_segment_invoked
> at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:43) ~[test-classes/:?]
> at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:33) ~[test-classes/:?]
> at org.infinispan.notifications.cachelistener.CacheNotifierImplInitialTransferDistTest.testIterationBeganAndSegmentNotComplete(CacheNotifierImplInitialTransferDistTest.java:519) ~[test-classes/:?]
> at org.infinispan.notifications.cachelistener.CacheNotifierImplInitialTransferDistTest.testCreateAfterIterationBeganAndSegmentNotCompleteValueOwnerClustered(CacheNotifierImplInitialTransferDistTest.java:633) ~[test-classes/:?]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (ISPN-6322) Infinispan can miss incoming commands with JGroupsChannelLookup
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-6322?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-6322:
-------------------------------
Description:
Normally, the JGroupsTransport startup sequence goes like this:
# Create the {{Channel}}
# Create the {{CommandAwareRpcDispatcher}} and install it as an {{UpHandler}}
# Connect the channel
This way, every {{RequestCorrelator}} message received by the channel is passed up to {{CommandAwareRpcDispatcher}}, which executes the appropriate command.
When using a {{JGroupsChannelLookup}}, the lookup implementation is allowed to return a {{Channel}} instance that is already connected ({{shouldConnect() == false}}). That means there is now a window where the channel doesn't have an {{UpHandler}}, and messages sent to this node are discarded.
Normally a node only receives commands after it sent a join request to the coordinator. There are however a few exceptions:
# On startup, {{LocalTopologyManagerImpl}} sends the join request to the JGroups coordinator, which may not have the {{UpHandler}} yet. This seems to be responsible for the recent hanging in {{ConcurrentStartTest}}. We have a workaround here, to use a smaller timeout on the {{CacheTopologyControlCommand(JOIN)}} command, and retry it on {{TimeoutException}}.
# When a node becomes coordinator, {{ClusterTopologyManagerImpl}} broadcasts a {{GET_STATUS}} request to all cluster members, and expects a response from each of them. The same workaround with a smaller timeout and retries might work here.
# In replicated mode, write commands are broadcasted to all cluster members. There is some commented out code in {{RpcManagerImpl.invokeRemotelyAsync()}} that might fix it by only waiting for responses from the cache topology members.
We should consider deprecating {{JGroupsChannelLookup.shouldConnect()}} and requiring that the channel is only connected by {{JGroupsTransport}}. Assuming that works with {{ForkChannel}}, of course.
was:
Normally, the JGroupsTransport startup sequence goes like this:
# Create the {{Channel}}
# Create the {{CommandAwareRpcDispatcher}} and install it as an {{UpHandler}}
# Connect the channel
This way, every {{RequestCorrelator}} message received by the channel is passed up to {{CommandAwareRpcDispatcher}}, which executes the appropriate command.
When using a {{JGroupsChannelLookup}}, the lookup implementation is allowed to return a {{Channel}} instance that is already connected ({{shouldConnect() == false}}). That means there is now a window where the channel doesn't have an {{UpHandler}}, and messages sent to this node are discarded.
Normally a node only receives commands after it sent a join request to the coordinator. There are however a few exceptions:
# On startup, {[LocalTopologyManagerImpl}} sends the join request to the JGroups coordinator, which may not have the {{UpHandler}} yet. This seems to be responsible for the recent hanging in {{ConcurrentStartTest}}. We have a workaround here, to use a smaller timeout on the {{CacheTopologyControlCommand(JOIN)}} command, and retry it on {{TimeoutException}}.
# When a node becomes coordinator, {{ClusterTopologyManagerImpl}} broadcasts a {{GET_STATUS}} request to all cluster members, and expects a response from each of them. The same workaround with a smaller timeout and retries might work here.
# In replicated mode, write commands are broadcasted to all cluster members. There is some commented out code in {{RpcManagerImpl.invokeRemotelyAsync()}} that might fix it by only waiting for responses from the cache topology members.
We should consider deprecating {{JGroupsChannelLookup.shouldConnect()}} and requiring that the channel is only connected by {{JGroupsTransport}}. Assuming that works with {{ForkChannel}}, of course.
> Infinispan can miss incoming commands with JGroupsChannelLookup
> ---------------------------------------------------------------
>
> Key: ISPN-6322
> URL: https://issues.jboss.org/browse/ISPN-6322
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 8.2.0.CR1, 8.1.2.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 9.0.0.Alpha1
>
>
> Normally, the JGroupsTransport startup sequence goes like this:
> # Create the {{Channel}}
> # Create the {{CommandAwareRpcDispatcher}} and install it as an {{UpHandler}}
> # Connect the channel
> This way, every {{RequestCorrelator}} message received by the channel is passed up to {{CommandAwareRpcDispatcher}}, which executes the appropriate command.
> When using a {{JGroupsChannelLookup}}, the lookup implementation is allowed to return a {{Channel}} instance that is already connected ({{shouldConnect() == false}}). That means there is now a window where the channel doesn't have an {{UpHandler}}, and messages sent to this node are discarded.
> Normally a node only receives commands after it sent a join request to the coordinator. There are however a few exceptions:
> # On startup, {{LocalTopologyManagerImpl}} sends the join request to the JGroups coordinator, which may not have the {{UpHandler}} yet. This seems to be responsible for the recent hanging in {{ConcurrentStartTest}}. We have a workaround here, to use a smaller timeout on the {{CacheTopologyControlCommand(JOIN)}} command, and retry it on {{TimeoutException}}.
> # When a node becomes coordinator, {{ClusterTopologyManagerImpl}} broadcasts a {{GET_STATUS}} request to all cluster members, and expects a response from each of them. The same workaround with a smaller timeout and retries might work here.
> # In replicated mode, write commands are broadcasted to all cluster members. There is some commented out code in {{RpcManagerImpl.invokeRemotelyAsync()}} that might fix it by only waiting for responses from the cache topology members.
> We should consider deprecating {{JGroupsChannelLookup.shouldConnect()}} and requiring that the channel is only connected by {{JGroupsTransport}}. Assuming that works with {{ForkChannel}}, of course.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (ISPN-6374) Unauthorized access while running javascript on secured cluster
by Anna Manukyan (JIRA)
Anna Manukyan created ISPN-6374:
-----------------------------------
Summary: Unauthorized access while running javascript on secured cluster
Key: ISPN-6374
URL: https://issues.jboss.org/browse/ISPN-6374
Project: Infinispan
Issue Type: Bug
Components: Tasks
Reporter: Anna Manukyan
Assignee: Tristan Tarrant
The following exception is thrown when trying to execute a distributed script on REPL/DIST 2 node secured cluster.
{code}
java.security.PrivilegedActionException: java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.lang.SecurityException: ISPN000287: Unauthorized access: subject 'null' lacks 'ADMIN' permission
at org.infinispan.security.Security.doAs(Security.java:145)
at org.infinispan.scripting.ReplicatedSecuredScriptingTest.testDistributedScriptExecutionWithRole(ReplicatedSecuredScriptingTest.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
at org.testng.TestRunner.privateRun(TestRunner.java:767)
at org.testng.TestRunner.run(TestRunner.java:617)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:343)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:305)
at org.testng.SuiteRunner.run(SuiteRunner.java:254)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
at org.testng.TestNG.run(TestNG.java:1057)
at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:72)
at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:122)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.lang.SecurityException: ISPN000287: Unauthorized access: subject 'null' lacks 'ADMIN' permission
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at org.infinispan.scripting.ReplicatedSecuredScriptingTest$9.run(ReplicatedSecuredScriptingTest.java:163)
at org.infinispan.scripting.ReplicatedSecuredScriptingTest$9.run(ReplicatedSecuredScriptingTest.java:158)
at org.infinispan.security.Security.doAs(Security.java:143)
... 29 more
Caused by: java.util.concurrent.ExecutionException: java.lang.SecurityException: ISPN000287: Unauthorized access: subject 'null' lacks 'ADMIN' permission
at org.infinispan.commons.util.concurrent.NotifyingFutureImpl.get(NotifyingFutureImpl.java:80)
at org.infinispan.util.concurrent.CompletableFutures.lambda$toCompletableFuture$1(CompletableFutures.java:47)
at org.infinispan.commons.util.concurrent.BaseNotifyingFuture.fireListeners(BaseNotifyingFuture.java:37)
at org.infinispan.commons.util.concurrent.NotifyingFutureImpl.notifyException(NotifyingFutureImpl.java:105)
at org.infinispan.distexec.DefaultExecutorService$LocalDistributedTaskPart$1.doLocalInvoke(DefaultExecutorService.java:1116)
at org.infinispan.distexec.DefaultExecutorService$LocalDistributedTaskPart$1.call(DefaultExecutorService.java:1102)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.SecurityException: ISPN000287: Unauthorized access: subject 'null' lacks 'ADMIN' permission
at org.infinispan.security.impl.AuthorizationHelper.checkPermission(AuthorizationHelper.java:86)
at org.infinispan.security.impl.AuthorizationManagerImpl.checkPermission(AuthorizationManagerImpl.java:42)
at org.infinispan.security.impl.SecureCacheImpl.getCacheManager(SecureCacheImpl.java:466)
at org.infinispan.scripting.impl.DistributedScript.setEnvironment(DistributedScript.java:35)
at org.infinispan.commands.read.DistributedExecuteCommand.perform(DistributedExecuteCommand.java:105)
at org.infinispan.distexec.DefaultExecutorService$LocalDistributedTaskPart$1.doLocalInvoke(DefaultExecutorService.java:1112)
... 5 more
{code}
You can find the source of the test here:
https://github.com/andyuk1986/infinispan/blob/cc001620fe199a72f165bb82b19...
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (ISPN-6373) Cache stop order not respected for ShardIndexManager
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-6373?page=com.atlassian.jira.plugin.... ]
Adrian Nistor updated ISPN-6373:
--------------------------------
Fix Version/s: 8.2.1.Final
> Cache stop order not respected for ShardIndexManager
> ----------------------------------------------------
>
> Key: ISPN-6373
> URL: https://issues.jboss.org/browse/ISPN-6373
> Project: Infinispan
> Issue Type: Bug
> Components: Embedded Querying
> Affects Versions: 8.2.0.Final
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
> Fix For: 9.0.0.Final, 9.0.0.Alpha1, 8.2.1.Final
>
>
> Query test suite is throwing tons of
> {noformat} ISPN000323: Cache 'LuceneIndexesLocking' is in 'TERMINATED' state and so it does not accept new invocations. Either restart it or recreate the cache container.
> {noformat}
> for test {{ClusteredCacheWithShardedIndexManagerTest}} since stop dependencies are not added correctly during initialization.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (ISPN-6373) Cache stop order not respected for ShardIndexManager
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-6373?page=com.atlassian.jira.plugin.... ]
Adrian Nistor commented on ISPN-6373:
-------------------------------------
Integrated in 8.2.x too.
> Cache stop order not respected for ShardIndexManager
> ----------------------------------------------------
>
> Key: ISPN-6373
> URL: https://issues.jboss.org/browse/ISPN-6373
> Project: Infinispan
> Issue Type: Bug
> Components: Embedded Querying
> Affects Versions: 8.2.0.Final
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
> Fix For: 9.0.0.Final, 9.0.0.Alpha1, 8.2.1.Final
>
>
> Query test suite is throwing tons of
> {noformat} ISPN000323: Cache 'LuceneIndexesLocking' is in 'TERMINATED' state and so it does not accept new invocations. Either restart it or recreate the cache container.
> {noformat}
> for test {{ClusteredCacheWithShardedIndexManagerTest}} since stop dependencies are not added correctly during initialization.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years