[JBoss JIRA] (ISPN-7099) Error executing command RollbackCommand due to timed out waiting for topology
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/ISPN-7099?page=com.atlassian.jira.plugin.... ]
Radoslav Husar reassigned ISPN-7099:
------------------------------------
Assignee: (was: Paul Ferraro)
> Error executing command RollbackCommand due to timed out waiting for topology
> -----------------------------------------------------------------------------
>
> Key: ISPN-7099
> URL: https://issues.jboss.org/browse/ISPN-7099
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 8.2.4.Final
> Reporter: Radoslav Husar
> Labels: infinispan_clean_shutdown
>
> Occured on server during eap-7x-failover-http-session-shutdown-dist-sync scenario.
> Server log stacktrace:
> {code}
> 00:14:37,012 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-6) ISPN000136: Error executing command RollbackCommand, writing keys [SessionCreationMetaDataKey(896a6XPyj99XiHDouVS5eojwHiGrjYJ2N-rOrzh8), SessionAttributesKey(896a6XPyj99XiHDouVS5eojwHiGrjYJ2N-rOrzh8), SessionAccessMetaDataKey(896a6XPyj99XiHDouVS5eojwHiGrjYJ2N-rOrzh8)]: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 28
> [JBossINF] at org.infinispan.statetransfer.StateTransferLockImpl.reportErrorAfterWait(StateTransferLockImpl.java:159)
> [JBossINF] at org.infinispan.statetransfer.StateTransferLockImpl.waitForTransactionData(StateTransferLockImpl.java:98)
> [JBossINF] at org.infinispan.interceptors.base.BaseStateTransferInterceptor.waitForTransactionData(BaseStateTransferInterceptor.java:97)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:249)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.visitRollbackCommand(StateTransferInterceptor.java:97)
> [JBossINF] at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:114)
> [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:83)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:117)
> [JBossINF] at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:117)
> [JBossINF] at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335)
> [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:231)
> [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:202)
> [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159)
> [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:114)
> [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:401)
> [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.commitResources(DummyTransaction.java:448)
> [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.runCommit(DummyTransaction.java:321)
> [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:108)
> [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:71)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:81)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:814)
> [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:572)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1668)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1668)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1668)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1668)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:207)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:810)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> {code}
> Followed by:
> {code}
> 00:14:37,012 WARN [org.infinispan.transaction.impl.TransactionCoordinator] (transport-thread--p15-t24) ISPN000141: Could not rollback prepared 1PC transaction. This transaction will be rolled back by the recovery process, if enabled. Transaction: LocalXaTransaction{xid=DummyXid{id=13473}} LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=[], backupKeyLocks=[], topologyId=26, stateTransferFlag=null} org.infinispan.transaction.xa.LocalXaTransaction@43a2: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 28
> [JBossINF] at org.infinispan.statetransfer.StateTransferLockImpl.reportErrorAfterWait(StateTransferLockImpl.java:159)
> ...
> {code}
> Server log:
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-http-...
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 8 months
[JBoss JIRA] (ISPN-6985) FlagsReplicationTest.testScenario random failures
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-6985?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-6985:
-------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/4606
> FlagsReplicationTest.testScenario random failures
> -------------------------------------------------
>
> Key: ISPN-6985
> URL: https://issues.jboss.org/browse/ISPN-6985
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite - Core
> Affects Versions: 8.2.4.Final
> Reporter: Ivan Straka
> Assignee: Dan Berindei
> Labels: testsuite_stability
> Fix For: 9.0.0.Beta1
>
>
> {code:java}
> Stacktrace
> java.lang.AssertionError
> at org.infinispan.replication.FlagsReplicationTest.haveSecondaryThreadReleaseLock(FlagsReplicationTest.java:80)
> at org.infinispan.replication.FlagsReplicationTest.testScenario(FlagsReplicationTest.java:63)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
> at java.lang.reflect.Method.invoke(Method.java:508)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:38)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:382)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask.run(FutureTask.java:277)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.lang.Thread.run(Thread.java:785)
> Standard Output
> [UnitTestTestNGListener] Test testScenario(org.infinispan.replication.FlagsReplicationTest) failed.
> 11:17:50,928 ERROR (testng-FlagsReplicationTest) [UnitTestTestNGListener] Test testScenario(org.infinispan.replication.FlagsReplicationTest) failed.
> java.lang.AssertionError
> at org.infinispan.replication.FlagsReplicationTest.haveSecondaryThreadReleaseLock(FlagsReplicationTest.java:80)
> at org.infinispan.replication.FlagsReplicationTest.testScenario(FlagsReplicationTest.java:63)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
> at java.lang.reflect.Method.invoke(Method.java:508)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)[UnitTestTestNGListener] Test suite progress: tests succeeded: 1533, failed: 4, skipped: 0.
> {code}
> [details|https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-infin...]
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 8 months
[JBoss JIRA] (ISPN-6985) FlagsReplicationTest.testScenario random failures
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-6985?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-6985:
-------------------------------
Status: Resolved (was: Pull Request Sent)
Fix Version/s: 9.0.0.Beta1
Resolution: Duplicate Issue
> FlagsReplicationTest.testScenario random failures
> -------------------------------------------------
>
> Key: ISPN-6985
> URL: https://issues.jboss.org/browse/ISPN-6985
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite - Core
> Affects Versions: 8.2.4.Final
> Reporter: Ivan Straka
> Assignee: Dan Berindei
> Labels: testsuite_stability
> Fix For: 9.0.0.Beta1
>
>
> {code:java}
> Stacktrace
> java.lang.AssertionError
> at org.infinispan.replication.FlagsReplicationTest.haveSecondaryThreadReleaseLock(FlagsReplicationTest.java:80)
> at org.infinispan.replication.FlagsReplicationTest.testScenario(FlagsReplicationTest.java:63)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
> at java.lang.reflect.Method.invoke(Method.java:508)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:38)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:382)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask.run(FutureTask.java:277)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.lang.Thread.run(Thread.java:785)
> Standard Output
> [UnitTestTestNGListener] Test testScenario(org.infinispan.replication.FlagsReplicationTest) failed.
> 11:17:50,928 ERROR (testng-FlagsReplicationTest) [UnitTestTestNGListener] Test testScenario(org.infinispan.replication.FlagsReplicationTest) failed.
> java.lang.AssertionError
> at org.infinispan.replication.FlagsReplicationTest.haveSecondaryThreadReleaseLock(FlagsReplicationTest.java:80)
> at org.infinispan.replication.FlagsReplicationTest.testScenario(FlagsReplicationTest.java:63)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
> at java.lang.reflect.Method.invoke(Method.java:508)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)[UnitTestTestNGListener] Test suite progress: tests succeeded: 1533, failed: 4, skipped: 0.
> {code}
> [details|https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-infin...]
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 8 months
[JBoss JIRA] (ISPN-7097) FlagsReplicationTest fails randomly
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-7097?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-7097:
-------------------------------
Status: Resolved (was: Pull Request Sent)
Fix Version/s: 9.0.0.Beta1
Resolution: Done
> FlagsReplicationTest fails randomly
> -----------------------------------
>
> Key: ISPN-7097
> URL: https://issues.jboss.org/browse/ISPN-7097
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Reporter: Radim Vansa
> Assignee: Radim Vansa
> Fix For: 9.0.0.Beta1
>
>
> The test uses another thread from {{AbstractInfinispanTest}}'s thread pool to execute isolated transaction, but it calls {{fork()}} twice: once to start the transaction and acquire the lock, and once to commit it. When it does not get the same thread for both locking and commit, it fails.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 8 months
[JBoss JIRA] (ISPN-7097) FlagsReplicationTest fails randomly
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-7097?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-7097:
-------------------------------
Description: The test uses another thread from {{AbstractInfinispanTest}}'s thread pool to execute isolated transaction, but it calls {{fork()}} twice: once to start the transaction and acquire the lock, and once to commit it. When it does not get the same thread for both locking and commit, it fails. (was: The test uses another thread from fork join pool to execute isolated transaction, but when it does not get the same thread for both locking and commit, it fails.)
> FlagsReplicationTest fails randomly
> -----------------------------------
>
> Key: ISPN-7097
> URL: https://issues.jboss.org/browse/ISPN-7097
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Reporter: Radim Vansa
> Assignee: Radim Vansa
>
> The test uses another thread from {{AbstractInfinispanTest}}'s thread pool to execute isolated transaction, but it calls {{fork()}} twice: once to start the transaction and acquire the lock, and once to commit it. When it does not get the same thread for both locking and commit, it fails.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 8 months
[JBoss JIRA] (ISPN-6985) FlagsReplicationTest.testScenario random failures
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-6985?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-6985:
-------------------------------
Status: Open (was: New)
> FlagsReplicationTest.testScenario random failures
> -------------------------------------------------
>
> Key: ISPN-6985
> URL: https://issues.jboss.org/browse/ISPN-6985
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite - Core
> Affects Versions: 8.2.4.Final
> Reporter: Ivan Straka
> Assignee: Dan Berindei
> Labels: testsuite_stability
>
> {code:java}
> Stacktrace
> java.lang.AssertionError
> at org.infinispan.replication.FlagsReplicationTest.haveSecondaryThreadReleaseLock(FlagsReplicationTest.java:80)
> at org.infinispan.replication.FlagsReplicationTest.testScenario(FlagsReplicationTest.java:63)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
> at java.lang.reflect.Method.invoke(Method.java:508)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:38)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:382)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask.run(FutureTask.java:277)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.lang.Thread.run(Thread.java:785)
> Standard Output
> [UnitTestTestNGListener] Test testScenario(org.infinispan.replication.FlagsReplicationTest) failed.
> 11:17:50,928 ERROR (testng-FlagsReplicationTest) [UnitTestTestNGListener] Test testScenario(org.infinispan.replication.FlagsReplicationTest) failed.
> java.lang.AssertionError
> at org.infinispan.replication.FlagsReplicationTest.haveSecondaryThreadReleaseLock(FlagsReplicationTest.java:80)
> at org.infinispan.replication.FlagsReplicationTest.testScenario(FlagsReplicationTest.java:63)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
> at java.lang.reflect.Method.invoke(Method.java:508)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)[UnitTestTestNGListener] Test suite progress: tests succeeded: 1533, failed: 4, skipped: 0.
> {code}
> [details|https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-infin...]
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 8 months
[JBoss JIRA] (ISPN-7098) FunctionalDistributionTest.testDistributionFromPrimaryOwner random failures
by Dan Berindei (JIRA)
Dan Berindei created ISPN-7098:
----------------------------------
Summary: FunctionalDistributionTest.testDistributionFromPrimaryOwner random failures
Key: ISPN-7098
URL: https://issues.jboss.org/browse/ISPN-7098
Project: Infinispan
Issue Type: Bug
Components: Core, Test Suite - Core
Affects Versions: 9.0.0.Alpha4
Reporter: Dan Berindei
Priority: Critical
Fix For: 9.0.0.Beta1
{{FunctionalDistributionTest.testDistributionFromPrimaryOwner}} assumes 100ms is enough for the command to be executed on both backup owners, but that's not always true -- e.g. when TRACE logging is enabled.
{noformat}
15:27:36,079 TRACE (async-thread-Test-NodeC-p36570-t1:[]) [InvocationContextInterceptor] Invoked with command ReadWriteKeyCommand {key=testDistributionFromPrimaryOwner, flags=[]} and InvocationContext [org.infinispan.context.SingleKeyNonTxInvocationContext@44ffeec2]
15:27:36,124 TRACE (async-thread-Test-NodeC-p36570-t1:[]) [JGroupsTransport] dests=[Test-NodeC-21089, Test-NodeD-31203], command=SingleRpcCommand{cacheName='dist', command=ReadWriteKeyCommand {key=testDistributionFromPrimaryOwner, flags=[]}}, mode=ASYNCHRONOUS, timeout=15000
15:27:36,140 TRACE (async-thread-Test-NodeC-p36570-t1:[]) [DefaultDataContainer] Store MetadataImmortalCacheEntry{key=testDistributionFromPrimaryOwner, value=1, metadata=MetaParamsInternalMetadata{params=MetaParams{metas=[]}}} in container
15:27:36,144 TRACE (Incoming-1,Test-NodeD-31203:[]) [InvocationContextInterceptor] Invoked with command ReadWriteKeyCommand {key=testDistributionFromPrimaryOwner, flags=[]} and InvocationContext [org.infinispan.context.SingleKeyNonTxInvocationContext@73111d9]
15:27:36,241 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.functional.FunctionalDistributionTest.testDistributionFromPrimaryOwner
java.lang.NullPointerException
at org.infinispan.functional.FunctionalDistributionTest.lambda$iterate$8(FunctionalDistributionTest.java:101) ~[test-classes/:?]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_101]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_101]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_101]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) ~[?:1.8.0_101]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_101]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_101]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_101]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_101]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_101]
at org.infinispan.functional.FunctionalDistributionTest.iterate(FunctionalDistributionTest.java:102) ~[test-classes/:?]
at org.infinispan.functional.FunctionalDistributionTest.doTestDistribution(FunctionalDistributionTest.java:80) ~[test-classes/:?]
at org.infinispan.functional.FunctionalDistributionTest.testDistributionFromPrimaryOwner(FunctionalDistributionTest.java:46) ~[test-classes/:?]
15:27:36,259 TRACE (Incoming-1,Test-NodeD-31203:[]) [DefaultDataContainer] Store MetadataImmortalCacheEntry{key=testDistributionFromPrimaryOwner, value=1, metadata=MetaParamsInternalMetadata{params=MetaParams{metas=[]}}} in container
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 8 months
[JBoss JIRA] (ISPN-7064) RPC to leaver times out instead of finishing immediately
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-7064?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-7064:
-------------------------------
Attachment: MultipleNodesLeavingTest_ISPN-5469_async_remote_perform_20160913.log
Another failure in `MultipleNodesLeavingTest`.
> RPC to leaver times out instead of finishing immediately
> --------------------------------------------------------
>
> Key: ISPN-7064
> URL: https://issues.jboss.org/browse/ISPN-7064
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite - Core
> Affects Versions: 9.0.0.Alpha4, 8.2.4.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Labels: testsuite_stability
> Fix For: 9.0.0.Beta1
>
> Attachments: MultipleNodesLeavingTest_ISPN-5469_async_remote_perform_20160913.log
>
>
> This causes random failures in {{ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus2}}. Node D leaves and nodes E and F start a rebalance. Then node F also leaves, but {{DISCARD}} is enabled first, so the {{CacheTopologyControlCommand(LEAVE)}} synchronous RPC blocks until node F installs a view by itself.
> {noformat}
> 14:05:17,187 INFO (Incoming-1,Test-NodeF-12732:[]) [JGroupsTransport] ISPN000094: Received new cluster view for channel ISPN: [Test-NodeE-18953|3] (2) [Test-NodeE-18953, Test-NodeF-12732]
> 14:05:17,261 TRACE (testng-Test:[]) [JGroupsTransport] dests=[Test-NodeE-18953], command=CacheTopologyControlCommand{cache=testCache, type=LEAVE, sender=Test-NodeF-12732, joinInfo=null, topologyId=0, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=3}, mode=SYNCHRONOUS, timeout=240000
> 14:05:22,546 DEBUG (Thread-88606:[]) [GMS] Test-NodeF-12732: installing view [Test-NodeF-12732|4] (1) [Test-NodeF-12732]
> 14:05:22,589 DEBUG (testng-Test:[]) [LocalTopologyManagerImpl] Error sending the leave request for cache testCache to coordinator
> {noformat}
> Node F then cancels its inbound state transfers by sending a synchronous {{StateRequestCommand(CANCEL_STATE_TRANSFER)}} RPC to node F. This RPC sometimes hits JGRP-2103 and it only times out after 5 minutes, causing the test to fail.
> {noformat}
> 14:05:22,608 TRACE (testng-Test:[]) [JGroupsTransport] dests=[Test-NodeE-18953], command=StateRequestCommand{cache=testCache, origin=Test-NodeF-12732, type=CANCEL_STATE_TRANSFER, topologyId=7, segments=[130, 3, ...]}, mode=SYNCHRONOUS_IGNORE_LEAVERS, timeout=240000
> 14:05:22,608 TRACE (testng-Test:[]) [RequestCorrelator] Test-NodeF-12732: invoking multicast RPC [req-id=134161]
> 14:05:22,609 DEBUG (Thread-88606:[]) [JGroupsTransport] New view accepted: [Test-NodeF-12732|4] (1) [Test-NodeF-12732]
> 14:09:22,608 TRACE (timeout-thread-Test-NodeF-p48767-t1:[]) [JGroupsTransport] Responses: Responses{
> Test-NodeE-18953: sender=Test-NodeE-18953, received=false, suspected=true}
> 14:09:22,608 TRACE (timeout-thread-Test-NodeF-p48767-t1:[]) [RpcManagerImpl] Replication exception
> org.infinispan.util.concurrent.TimeoutException: Replication timeout
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$2(JGroupsTransport.java:659) ~[classes/:?]
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) ~[?:1.8.0_101]
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ~[?:1.8.0_101]
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) [?:1.8.0_101]
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) [?:1.8.0_101]
> at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47) [classes/:?]
> at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:15) [classes/:?]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_101]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_101]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_101]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
> Suppressed: org.infinispan.remoting.RpcException: Test-NodeE-18953 was suspected
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addSuppressedExceptions(JGroupsTransport.java:707) ~[classes/:?]
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$2(JGroupsTransport.java:659) ~[classes/:?]
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) ~[?:1.8.0_101]
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ~[?:1.8.0_101]
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) [?:1.8.0_101]
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) [?:1.8.0_101]
> at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47) [classes/:?]
> at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:15) [classes/:?]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_101]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_101]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_101]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
> 14:09:22,609 DEBUG (testng-Test:[]) [InboundTransferTask] Caught an exception while cancelling state transfer for segments [130, 3, ...] from Test-NodeE-18953
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 8 months
[JBoss JIRA] (ISPN-6970) Upgrade to JGroups 4.0.0.Alpha1
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/ISPN-6970?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on ISPN-6970:
--------------------------------
I tested 4.0.0.Alpha1 against 4.0.0.Beta1 on cluster01-04, using {{UPerf}}.
Performance of Beta1 was slightly better, but I definitely did not see a performance regression between Alpha1 and Beta1.
> Upgrade to JGroups 4.0.0.Alpha1
> -------------------------------
>
> Key: ISPN-6970
> URL: https://issues.jboss.org/browse/ISPN-6970
> Project: Infinispan
> Issue Type: Component Upgrade
> Components: Build process, Core
> Affects Versions: 9.0.0.Alpha4
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 9.0.0.Beta1
>
>
> This will allow us to experiment with the changes in the JGroups 4.0 before it's set in stone.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 8 months