[JBoss JIRA] (ISPN-1896) ClusteredGetCommands should never fail with a SuspectException
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-1896?page=com.atlassian.jira.plugin.... ]
Dan Berindei resolved ISPN-1896.
--------------------------------
Fix Version/s: 7.0.3.Final
Resolution: Done
Fixed with ISPN-5032
> ClusteredGetCommands should never fail with a SuspectException
> --------------------------------------------------------------
>
> Key: ISPN-1896
> URL: https://issues.jboss.org/browse/ISPN-1896
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 5.1.6.FINAL
> Reporter: Dan Berindei
> Fix For: 7.0.3.Final
>
>
> I have seen this exception in the core test suite logs:
> {noformat}
> 2012-03-02 15:07:19,718 ERROR (testng-VersionedDistStateTransferTest) [org.infinispan.test.fwk.UnitTestTestNGListener] Method testStateTransfer(org.infinispan.container.versioning.VersionedDistStateTransferTest) threw an exceptionorg.infinispan.CacheException: SuspectedException
> at org.infinispan.util.Util.rewrapAsCacheException(Util.java:524)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:168)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:478)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148)
> at org.infinispan.distribution.DistributionManagerImpl.retrieveFromRemoteSource(DistributionManagerImpl.java:169)
> at org.infinispan.interceptors.DistributionInterceptor.realRemoteGet(DistributionInterceptor.java:212)
> at org.infinispan.interceptors.DistributionInterceptor.remoteGetAndStoreInL1(DistributionInterceptor.java:194)
> at org.infinispan.interceptors.DistributionInterceptor.remoteGetBeforeWrite(DistributionInterceptor.java:440)
> at org.infinispan.interceptors.DistributionInterceptor.handleWriteCommand(DistributionInterceptor.java:455)
> at org.infinispan.interceptors.DistributionInterceptor.visitPutKeyValueCommand(DistributionInterceptor.java:274)
> ...
> Caused by: SuspectedException
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:349)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:263)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:163)
> ... 60 more
> {noformat}
> The remote get command should return null instead of failing, even if it had a single target node.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 8 months
[JBoss JIRA] (ISPN-2752) IllegalStateException on shutdown
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-2752?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-2752:
-------------------------------
Status: Pull Request Sent (was: Open)
> IllegalStateException on shutdown
> ---------------------------------
>
> Key: ISPN-2752
> URL: https://issues.jboss.org/browse/ISPN-2752
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 5.2.0.CR2
> Reporter: Michal Linhard
> Priority: Minor
>
> This happens on graceful shutdown of 8 or 6 nodes of 5.2.0.CR2:
> {code}
> 07:17:45,892 ERROR [org.infinispan.topology.ClusterTopologyManagerImpl] ISPN000196: Failed to recover cluster state after the current node became the coordinator: java.util.concurrent.ExecutionException: org.infinispan.CacheException: Remote (node0005/default) failed unexpectedly
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232) [rt.jar:1.6.0_33]
> at java.util.concurrent.FutureTask.get(FutureTask.java:91) [rt.jar:1.6.0_33]
> at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterSync(ClusterTopologyManagerImpl.java:567) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at org.infinispan.topology.ClusterTopologyManagerImpl.recoverClusterStatus(ClusterTopologyManagerImpl.java:432) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleNewView(ClusterTopologyManagerImpl.java:231) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.handleViewChange(ClusterTopologyManagerImpl.java:597) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_33]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_33]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_33]
> at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_33]
> at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation$1.run(AbstractListenerImpl.java:212) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_33]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_33]
> at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_33]
> Caused by: org.infinispan.CacheException: Remote (node0005/default) failed unexpectedly
> at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:96) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:541) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at org.infinispan.topology.ClusterTopologyManagerImpl$2.call(ClusterTopologyManagerImpl.java:549) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at org.infinispan.topology.ClusterTopologyManagerImpl$2.call(ClusterTopologyManagerImpl.java:546) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_33]
> at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_33]
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_33]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_33]
> at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_33]
> at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA-redhat-2.jar:2.0.0.GA-redhat-2]
> Caused by: java.lang.IllegalStateException: transport was closed
> at org.jgroups.blocks.GroupRequest.transportClosed(GroupRequest.java:273) [jgroups-3.2.6.Final-redhat-1.jar:3.2.6.Final-redhat-1]
> at org.jgroups.blocks.RequestCorrelator.stop(RequestCorrelator.java:269) [jgroups-3.2.6.Final-redhat-1.jar:3.2.6.Final-redhat-1]
> at org.jgroups.blocks.MessageDispatcher.stop(MessageDispatcher.java:152) [jgroups-3.2.6.Final-redhat-1.jar:3.2.6.Final-redhat-1]
> at org.jgroups.blocks.MessageDispatcher.channelDisconnected(MessageDispatcher.java:455) [jgroups-3.2.6.Final-redhat-1.jar:3.2.6.Final-redhat-1]
> at org.jgroups.Channel.notifyChannelDisconnected(Channel.java:507) [jgroups-3.2.6.Final-redhat-1.jar:3.2.6.Final-redhat-1]
> at org.jgroups.JChannel.disconnect(JChannel.java:363) [jgroups-3.2.6.Final-redhat-1.jar:3.2.6.Final-redhat-1]
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.stop(JGroupsTransport.java:258) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_33]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_33]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_33]
> at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_33]
> at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:883) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:690) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:568) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at org.infinispan.factories.GlobalComponentRegistry.stop(GlobalComponentRegistry.java:260) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:742) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at org.infinispan.manager.AbstractDelegatingEmbeddedCacheManager.stop(AbstractDelegatingEmbeddedCacheManager.java:179) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at org.jboss.as.clustering.infinispan.subsystem.EmbeddedCacheManagerService.stop(EmbeddedCacheManagerService.java:76) [jboss-datagrid-infinispan-6.1.0.ER9-redhat-1.jar:6.1.0.ER9-redhat-1]
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.2.GA-redhat-2.jar:1.0.2.GA-redhat-2]
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.2.GA-redhat-2.jar:1.0.2.GA-redhat-2]
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_33]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_33]
> at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_33]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 8 months
[JBoss JIRA] (ISPN-5471) ConditionalOperationsConcurrentWriteSkewTest random failures
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-5471?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-5471:
-------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/3468
> ConditionalOperationsConcurrentWriteSkewTest random failures
> ------------------------------------------------------------
>
> Key: ISPN-5471
> URL: https://issues.jboss.org/browse/ISPN-5471
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 7.2.1.Final
> Reporter: Dan Berindei
> Labels: testsuite_stability
> Fix For: 8.0.0.Alpha1
>
>
> The latches in {{ConditionalOperationsConcurrentWriteSkewTest}} are {{volatile}}, so the waiters should save the latches in local variables before waiting on them.
> {noformat}
> java.util.concurrent.ExecutionException: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:206)
> at org.infinispan.api.ConditionalOperationsConcurrentWriteSkewTest.doSimpleConcurrentTest(ConditionalOperationsConcurrentWriteSkewTest.java:131)
> at org.infinispan.api.ConditionalOperationsConcurrentWriteSkewTest.testSimpleConcurrentReplace(ConditionalOperationsConcurrentWriteSkewTest.java:62)
> Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1212)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
> at org.infinispan.api.ConditionalOperationsConcurrentWriteSkewTest$1.call(ConditionalOperationsConcurrentWriteSkewTest.java:94)
> at org.infinispan.api.ConditionalOperationsConcurrentWriteSkewTest$1.call(ConditionalOperationsConcurrentWriteSkewTest.java:89)
> at org.infinispan.test.AbstractInfinispanTest$LoggingCallable.call(AbstractInfinispanTest.java:422)
> ... 4 more
> Suppressed: javax.transaction.xa.XAException
> at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:213)
> at org.infinispan.transaction.impl.TransactionCoordinator.commitInternal(TransactionCoordinator.java:224)
> at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:163)
> at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:110)
> at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:698)
> at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2364)
> at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1518)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96)
> at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)
> ... 8 more
> Caused by: java.lang.NullPointerException
> at org.infinispan.api.ConditionalOperationsConcurrentWriteSkewTest$CommandInterceptorController.visitCommitCommand(ConditionalOperationsConcurrentWriteSkewTest.java:219)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.VersionedEntryWrappingInterceptor.visitCommitCommand(VersionedEntryWrappingInterceptor.java:67)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:133)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitCommitCommand(AbstractTxLockingInterceptor.java:86)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.NotificationInterceptor.visitCommitCommand(NotificationInterceptor.java:43)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.TxInterceptor.visitCommitCommand(TxInterceptor.java:224)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:133)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommitCommand(TransactionSynchronizerInterceptor.java:54)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:207)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitCommitCommand(StateTransferInterceptor.java:100)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:133)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
> at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:133)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
> at org.infinispan.transaction.impl.TransactionCoordinator.commitInternal(TransactionCoordinator.java:221)
> ... 16 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 8 months
[JBoss JIRA] (ISPN-5471) ConditionalOperationsConcurrentWriteSkewTest random failures
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-5471?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-5471:
-------------------------------
Status: Open (was: New)
> ConditionalOperationsConcurrentWriteSkewTest random failures
> ------------------------------------------------------------
>
> Key: ISPN-5471
> URL: https://issues.jboss.org/browse/ISPN-5471
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 7.2.1.Final
> Reporter: Dan Berindei
> Labels: testsuite_stability
> Fix For: 8.0.0.Alpha1
>
>
> The latches in {{ConditionalOperationsConcurrentWriteSkewTest}} are {{volatile}}, so the waiters should save the latches in local variables before waiting on them.
> {noformat}
> java.util.concurrent.ExecutionException: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:206)
> at org.infinispan.api.ConditionalOperationsConcurrentWriteSkewTest.doSimpleConcurrentTest(ConditionalOperationsConcurrentWriteSkewTest.java:131)
> at org.infinispan.api.ConditionalOperationsConcurrentWriteSkewTest.testSimpleConcurrentReplace(ConditionalOperationsConcurrentWriteSkewTest.java:62)
> Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1212)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
> at org.infinispan.api.ConditionalOperationsConcurrentWriteSkewTest$1.call(ConditionalOperationsConcurrentWriteSkewTest.java:94)
> at org.infinispan.api.ConditionalOperationsConcurrentWriteSkewTest$1.call(ConditionalOperationsConcurrentWriteSkewTest.java:89)
> at org.infinispan.test.AbstractInfinispanTest$LoggingCallable.call(AbstractInfinispanTest.java:422)
> ... 4 more
> Suppressed: javax.transaction.xa.XAException
> at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:213)
> at org.infinispan.transaction.impl.TransactionCoordinator.commitInternal(TransactionCoordinator.java:224)
> at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:163)
> at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:110)
> at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:698)
> at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2364)
> at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1518)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96)
> at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)
> ... 8 more
> Caused by: java.lang.NullPointerException
> at org.infinispan.api.ConditionalOperationsConcurrentWriteSkewTest$CommandInterceptorController.visitCommitCommand(ConditionalOperationsConcurrentWriteSkewTest.java:219)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.VersionedEntryWrappingInterceptor.visitCommitCommand(VersionedEntryWrappingInterceptor.java:67)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:133)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitCommitCommand(AbstractTxLockingInterceptor.java:86)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.NotificationInterceptor.visitCommitCommand(NotificationInterceptor.java:43)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.TxInterceptor.visitCommitCommand(TxInterceptor.java:224)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:133)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommitCommand(TransactionSynchronizerInterceptor.java:54)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:207)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitCommitCommand(StateTransferInterceptor.java:100)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:133)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
> at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:133)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
> at org.infinispan.transaction.impl.TransactionCoordinator.commitInternal(TransactionCoordinator.java:221)
> ... 16 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 8 months
[JBoss JIRA] (ISPN-5471) ConditionalOperationsConcurrentWriteSkewTest random failures
by Dan Berindei (JIRA)
Dan Berindei created ISPN-5471:
----------------------------------
Summary: ConditionalOperationsConcurrentWriteSkewTest random failures
Key: ISPN-5471
URL: https://issues.jboss.org/browse/ISPN-5471
Project: Infinispan
Issue Type: Bug
Components: Test Suite - Core
Affects Versions: 7.2.1.Final
Reporter: Dan Berindei
Fix For: 8.0.0.Alpha1
The latches in {{ConditionalOperationsConcurrentWriteSkewTest}} are {{volatile}}, so the waiters should save the latches in local variables before waiting on them.
{noformat}
java.util.concurrent.ExecutionException: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at org.infinispan.api.ConditionalOperationsConcurrentWriteSkewTest.doSimpleConcurrentTest(ConditionalOperationsConcurrentWriteSkewTest.java:131)
at org.infinispan.api.ConditionalOperationsConcurrentWriteSkewTest.testSimpleConcurrentReplace(ConditionalOperationsConcurrentWriteSkewTest.java:62)
Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1212)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at org.infinispan.api.ConditionalOperationsConcurrentWriteSkewTest$1.call(ConditionalOperationsConcurrentWriteSkewTest.java:94)
at org.infinispan.api.ConditionalOperationsConcurrentWriteSkewTest$1.call(ConditionalOperationsConcurrentWriteSkewTest.java:89)
at org.infinispan.test.AbstractInfinispanTest$LoggingCallable.call(AbstractInfinispanTest.java:422)
... 4 more
Suppressed: javax.transaction.xa.XAException
at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:213)
at org.infinispan.transaction.impl.TransactionCoordinator.commitInternal(TransactionCoordinator.java:224)
at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:163)
at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:110)
at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:698)
at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2364)
at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1518)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)
... 8 more
Caused by: java.lang.NullPointerException
at org.infinispan.api.ConditionalOperationsConcurrentWriteSkewTest$CommandInterceptorController.visitCommitCommand(ConditionalOperationsConcurrentWriteSkewTest.java:219)
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.VersionedEntryWrappingInterceptor.visitCommitCommand(VersionedEntryWrappingInterceptor.java:67)
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:133)
at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitCommitCommand(AbstractTxLockingInterceptor.java:86)
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.NotificationInterceptor.visitCommitCommand(NotificationInterceptor.java:43)
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.TxInterceptor.visitCommitCommand(TxInterceptor.java:224)
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:133)
at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommitCommand(TransactionSynchronizerInterceptor.java:54)
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:207)
at org.infinispan.statetransfer.StateTransferInterceptor.visitCommitCommand(StateTransferInterceptor.java:100)
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:133)
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:133)
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:39)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
at org.infinispan.transaction.impl.TransactionCoordinator.commitInternal(TransactionCoordinator.java:221)
... 16 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 8 months
[JBoss JIRA] (ISPN-5106) Deadlock on GlobalComponentRegistry when starting a cluster
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-5106?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-5106:
-----------------------------------------------
Sebastian Łaskawiec <slaskawi(a)redhat.com> changed the Status of [bug 1217380|https://bugzilla.redhat.com/show_bug.cgi?id=1217380] from MODIFIED to ON_QA
> Deadlock on GlobalComponentRegistry when starting a cluster
> -----------------------------------------------------------
>
> Key: ISPN-5106
> URL: https://issues.jboss.org/browse/ISPN-5106
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Reporter: Jakub Markos
> Assignee: Dan Berindei
> Priority: Critical
> Fix For: 7.2.0.Final
>
> Attachments: dumps_and_logs.zip
>
>
> We have a test which starts 4 server nodes, and sometimes they fail to complete the startup. This happens with the current snapshot.
> It appears there's a deadlock on intrinsic locks on GlobalComponentRegistry, since the CacheTopologyControlCommand.POLICY_GET_STATUS is sent with the lock acquired but this lock is also needed for injecting dependencies when the command is processed on the remote node.
> Here are the relevant parts from the dumps, node02:
> {code}
> "remote-thread--p3-t1" daemon prio=10 tid=0x00007f7a00002800 nid=0x487f waiting for monitor entry [0x00007f796bbfa000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:262)
> - waiting to lock <0x000000060365b6b8> (a org.infinispan.factories.GlobalComponentRegistry)
> at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:227)
> at org.infinispan.factories.AbstractComponentRegistry.wireDependencies(AbstractComponentRegistry.java:132)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$2.run(GlobalInboundInvocationHandler.java:156)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Locked ownable synchronizers:
> - <0x0000000615af46d0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "MSC service thread 1-16" prio=10 tid=0x00007f79ec071800 nid=0x4839 waiting on condition [0x00007f7a40239000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0000000614d47e60> (a java.util.concurrent.FutureTask)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:422)
> at java.util.concurrent.FutureTask.get(FutureTask.java:199)
> at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterSync(ClusterTopologyManagerImpl.java:432)
> at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterSync(ClusterTopologyManagerImpl.java:385)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:368)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:359)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:281)
> - locked <0x000000060420d4a8> (a java.lang.Object)
> at org.infinispan.topology.ClusterTopologyManagerImpl.start(ClusterTopologyManagerImpl.java:103)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
> - locked <0x000000060365b6b8> (a org.infinispan.factories.GlobalComponentRegistry)
> at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:221)
> - locked <0x000000060365b6b8> (a org.infinispan.factories.GlobalComponentRegistry)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:580)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:546)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:423)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:437)
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:89)
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:80)
> at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:116)
> at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:113)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:60)
> at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:121)
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:79)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Locked ownable synchronizers:
> - <0x0000000653444750> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> {code}
> and node03
> {code}
> "remote-thread--p3-t1" daemon prio=10 tid=0x00007f016c079000 nid=0x1a43 waiting for monitor entry [0x00007f0114396000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:262)
> - waiting to lock <0x0000000609c2bf50> (a org.infinispan.factories.GlobalComponentRegistry)
> at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:227)
> at org.infinispan.factories.AbstractComponentRegistry.wireDependencies(AbstractComponentRegistry.java:132)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$2.run(GlobalInboundInvocationHandler.java:156)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Locked ownable synchronizers:
> - <0x0000000615a05750> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "MSC service thread 1-16" prio=10 tid=0x00007f015c071800 nid=0x19ff waiting on condition [0x00007f01b0558000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0000000615025bb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at org.jgroups.util.CondVar.waitFor(CondVar.java:64)
> at org.jgroups.blocks.Request.waitForResults(Request.java:195)
> at org.jgroups.blocks.Request.responsesComplete(Request.java:181)
> at org.jgroups.blocks.Request.execute(Request.java:89)
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:409)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:374)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:188)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:562)
> at org.infinispan.topology.ClusterTopologyManagerImpl.start(ClusterTopologyManagerImpl.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
> - locked <0x0000000609c2bf50> (a org.infinispan.factories.GlobalComponentRegistry)
> at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:221)
> - locked <0x0000000609c2bf50> (a org.infinispan.factories.GlobalComponentRegistry)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:580)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:546)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:423)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:437)
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:89)
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:80)
> at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:116)
> at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:113)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:60)
> at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:121)
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:79)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Locked ownable synchronizers:
> - <0x00000006534e9628> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 8 months
[JBoss JIRA] (ISPN-5386) Tx succeeds on coord, while being rollbacked on other participants due to Tx pruning
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-5386?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-5386:
-----------------------------------------------
Sebastian Łaskawiec <slaskawi(a)redhat.com> changed the Status of [bug 1212795|https://bugzilla.redhat.com/show_bug.cgi?id=1212795] from MODIFIED to ON_QA
> Tx succeeds on coord, while being rollbacked on other participants due to Tx pruning
> ------------------------------------------------------------------------------------
>
> Key: ISPN-5386
> URL: https://issues.jboss.org/browse/ISPN-5386
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Reporter: Matej Čimbora
> Assignee: Dan Berindei
> Fix For: 7.2.0.Final
>
>
> All participants of transaction share the same topology. TX gets successfully prepared & commited on coordinator.
> {code}
> 03:49:27,759 DEBUG [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-1,edg-perf08-48196) New view accepted: [edg-perf08-48196|18] (5) [edg-perf08-48196, edg-perf01-23632, edg-perf02-34805, edg-perf03-16232, edg-perf04-41106]
> 03:49:41,051 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (transport-thread-9) Installing new cache topology CacheTopology{id=53, rebalanceId=19, currentCH=DefaultConsistentHash{ns = 512, owners = (5)[edg-perf08-48196: 103+101, edg-perf01-23632: 102+103, edg-perf02-34805: 102+103, edg-perf03-16232: 102+103, edg-perf04-41106: 103+102]}, pendingCH=null, unionCH=null, actualMembers=[edg-perf08-48196, edg-perf01-23632, edg-perf02-34805, edg-perf03-16232, edg-perf04-41106]} on cache testCache
> ...
> 03:51:34,005 TRACE [org.infinispan.remoting.rpc.RpcManagerImpl] (DefaultStressor-1) edg-perf08-48196 invoking PrepareCommand { ... gtx=GlobalTransaction:<edg-perf08-48196>:13330:local, cacheName='testCache', topologyId=53} to recipient list [edg-perf03-16232, edg-perf08-48196, edg-perf02-34805, edg-perf04-41106, edg-perf01-23632]
> 03:51:36,329 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (DefaultStressor-1) Responses: [sender=edg-perf03-16232,received=true, suspected=false] [sender=edg-perf02-34805, received=true, suspected=false] [sender=edg-perf04-41106, received=true, suspected=false] [sender=edg-perf01-23632, received=true, suspected=false]
> 03:51:36,342 TRACE [org.infinispan.remoting.rpc.RpcManagerImpl] (DefaultStressor-1) edg-perf08-48196 invoking CommitCommand {gtx=GlobalTransaction:<edg-perf08-48196>:13330:local, cacheName='testCache', topologyId=53} to recipient list [edg-perf03-16232, edg-perf08-48196, edg-perf02-34805, edg-perf04-41106, edg-perf01-23632] with options RpcOptions{timeout=60000, unit=MILLISECONDS, fifoOrder=false, totalOrder=false, responseFilter=null, responseMode=SYNCHRONOUS_IGNORE_LEAVERS, skipReplicationQueue=false}
> 03:51:36,703 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (DefaultStressor-1) Responses: [sender=edg-perf03-16232, retval=SuccessfulResponse{responseValue=null} , received=true, suspected=false] [sender=edg-perf02-34805, retval=SuccessfulResponse{responseValue=null} , received=true, suspected=false] [sender=edg-perf04-41106, retval=SuccessfulResponse{responseValue=null} , received=true, suspected=false] [sender=edg-perf01-23632, retval=SuccessfulResponse{responseValue=null} , received=true, suspected=false]
> {code}
> The problem is, that other participating nodes rollback it, as TX with higher id was completed before. Successfull response is returned for both prepare & commit commands.
> {code}
> 03:49:58,190 TRACE [org.infinispan.transaction.TransactionTable] (remote-thread-499) Marking transaction GlobalTransaction:<edg-perf08-48196>:13337:local as completed
> ...
> 03:51:34,122 TRACE [org.infinispan.transaction.TransactionTable] (remote-thread-593) Created and registered remote transaction RemoteTransaction{ ... lookedUpEntries={}, lockedKeys=null, backupKeyLocks=null, lookedUpEntriesTopology=2147483647, isMarkedForRollback=false, tx=GlobalTransaction:<edg-perf08-48196>:13330:remote, state=null}
> 03:51:34,073 TRACE [org.infinispan.remoting.InboundInvocationHandlerImpl] (remote-thread-593) Calling perform() on PrepareCommand { ... gtx=GlobalTransaction:<edg-perf08-48196>:13330:remote, cacheName='testCache', topologyId=53}
> 03:51:34,342 TRACE [org.infinispan.interceptors.TxInterceptor] (remote-thread-593) Rolling back remote transaction GlobalTransaction:<edg-perf08-48196>:13330:remote because either already completed (true) or originator no longer in the cluster (false).
> 03:51:34,639 TRACE [org.infinispan.remoting.InboundInvocationHandlerImpl] (remote-thread-593) About to send back response null for command PrepareCommand { ... gtx=GlobalTransaction:<edg-perf08-48196>:13330:remote, cacheName='testCache', topologyId=53}
> 03:51:36,355 TRACE [org.infinispan.remoting.InboundInvocationHandlerImpl] (remote-thread-589) Calling perform() on CommitCommand {gtx=GlobalTransaction:<edg-perf08-48196>:13330:remote, cacheName='testCache', topologyId=53}
> 03:51:36,355 TRACE [org.infinispan.commands.tx.AbstractTransactionBoundaryCommand] (remote-thread-589) Did not find a RemoteTransaction for GlobalTransaction:<edg-perf08-48196>:13330:remote
> 03:51:36,355 TRACE [org.infinispan.remoting.InboundInvocationHandlerImpl] (remote-thread-589) About to send back response SuccessfulResponse{responseValue=null} for command CommitCommand {gtx=GlobalTransaction:<edg-perf08-48196>:13330:remote, cacheName='testCache', topologyId=53}
> {code}
> Exception response should be returned instead to avoid incorrect assumptions about presence of updated entry in the cache.
> [~dan.berindei] spotted lastPrunedTxId modifications are not logged, let's make sure they are.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 8 months
[JBoss JIRA] (ISPN-5261) Transaction is recommited by joining node during merge
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-5261?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-5261:
-----------------------------------------------
Sebastian Łaskawiec <slaskawi(a)redhat.com> changed the Status of [bug 1198267|https://bugzilla.redhat.com/show_bug.cgi?id=1198267] from MODIFIED to ON_QA
> Transaction is recommited by joining node during merge
> ------------------------------------------------------
>
> Key: ISPN-5261
> URL: https://issues.jboss.org/browse/ISPN-5261
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 7.1.1.Final
> Reporter: Matej Čimbora
> Assignee: Dan Berindei
> Fix For: 7.2.1.Final
>
>
> Transaction gets recommited after a node joins a cluster during merge and requests retransmission of previous messages, overwriting updates which happened till that point.
> 4 nodes - edg-perf01-04
> Scenario:
> 1. edg-perf01 begins a new transaction & updates value of K
> {code}
> 06:50:50,421 TRACE [org.infinispan.transaction.xa.TransactionXaAdapter] (DefaultStressor-7) start called on tx GlobalTransaction:<edg-perf01-19349>:3516:local
> {code}
> The latest view on edg-perf01 is
> {code}
> 06:50:35,103 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,edg-perf01-19349) ISPN000093: Received new, MERGED cluster view: MergeView::[edg-perf01-19349|5] (3) [edg-perf01-19349, edg-perf04-20753, edg-perf02-19191], 1 subgroups: [edg-perf01-19349|4] (2) [edg-perf01-19349, edg-perf02-19191]
> {code}
> 2. Prepare command is sent with nodes edg-perf02 & edg-perf04 responding successfully
> {code}
> 06:50:51,186 TRACE [org.infinispan.remoting.rpc.RpcManagerImpl] (DefaultStressor-7) edg-perf01-19349 invoking PrepareCommand ... topologyId=12} to recipient list [edg-perf02-19191, edg-perf04-20753, edg-perf01-19349] ...
> {code}
> check message id & seqno
> {code}
> (06:50:51,195 TRACE [org.jgroups.protocols.TCP] (DefaultStressor-7) edg-perf01-19349: sending msg to null, src=edg-perf01-19349, headers are RequestCorrelator: id=200, type=REQ, id=17134, rsp_expected=true, exclusion_list=[edg-perf01-19349], NAKACK2: [MSG, seqno=1020], 20903: slaveIndex=0, TCP: [cluster_name=default])
> {code}
> 3. Transaction gets commited & value of K gets updated multiple times on edg-perf01 from this point on
> 4. edg-perf03 receives a new view containing all nodes
> {code}
> 06:50:53,963 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,edg-perf03-14171) ISPN000093: Received new, MERGED cluster view: MergeView::[edg-perf01-19349|6] (4) [edg-perf01-19349, edg-perf03-14171, edg-perf04-20753, edg-perf02-19191], 3 subgroups: [edg-perf03-14171|4] (2) [edg-perf03-14171, edg-perf04-20753], [edg-perf01-19349|5] (3) [edg-perf01-19349, edg-perf04-20753, edg-perf02-19191], [edg-perf01-19349|4] (2) [edg-perf01-19349, edg-perf02-19191]
> {code}
> 5. edg-perf03 apparently requests retransmission of previous messages (containing prepare from step#2) & edg-perf01 sends the response
> Request
> {code}
> 06:51:35,241 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (Timer-3,edg-perf03-14171) edg-perf03-14171: sending XMIT_REQ ((79): {1015-1093}) to edg-perf01-19349
> {code}
> Response
> {code}
> 06:51:35,299 TRACE [org.jgroups.protocols.TCP] (INT-14,edg-perf03-14171) edg-perf03-14171: received [dst: edg-perf03-14171, src: edg-perf01-19349 (4 headers), size=975 bytes, flags=OOB|DONT_BUNDLE|NO_TOTAL_ORDER|INTERNAL], headers are RequestCorrelator: id=200, type=REQ, id=17134, rsp_expected=true, exclusion_list=[edg-perf01-19349], NAKACK2: [XMIT_RSP, seqno=1020], 20903: slaveIndex=0, TCP: [cluster_name=default]
> {code}
> 6. edg-perf03 prepares the old transaction & eventually commits it again
> {code}
> 06:51:35,304 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (INT-14,edg-perf03-14171) Attempting to execute command: PrepareCommand.....gtx=GlobalTransaction:<edg-perf01-19349>:3516:local, cacheName='testCache', topologyId=12} [sender=edg-perf01-19349]
> 06:51:35,304 TRACE [org.infinispan.statetransfer.StateTransferLockImpl] (INT-14,edg-perf03-14171) Checking if transaction data was received for topology 12, current topology is 14
> 06:51:35,305 TRACE [org.infinispan.transaction.TransactionTable] (remote-thread-40) Created and registered remote transaction ... tx=GlobalTransaction:<edg-perf01-19349>:3516:remote, ...
> 06:51:35,308 TRACE [org.infinispan.remoting.rpc.RpcManagerImpl] (remote-thread-40) edg-perf03-14171 invoking PrepareCommand ... gtx=GlobalTransaction:<edg-perf01-19349>:3516:remote, cacheName='testCache', topologyId=15} to recipient list [edg-perf04-20753, edg-perf02-19191] ...
> {code}
> Successful commit responses
> {code}
> 06:51:36,750 TRACE [org.infinispan.remoting.rpc.RpcManagerImpl] (remote-thread-44) Response(s) to CommitCommand {gtx=GlobalTransaction:<edg-perf01-19349>:3516:remote, cacheName='testCache', topologyId=15} is {edg-perf02-19191=SuccessfulResponse{responseValue=null} , edg-perf04-20753=SuccessfulResponse{responseValue=null} }
> {code}
> 7. When get(K) is invoked on edg-perf01, stale value is returned (the one from step #6, ignoring updates which have happened since step #3)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 8 months
[JBoss JIRA] (ISPN-5462) Transaction prepare is not replicated to new owners during state transfer
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-5462?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-5462:
-----------------------------------------------
Sebastian Łaskawiec <slaskawi(a)redhat.com> changed the Status of [bug 1221166|https://bugzilla.redhat.com/show_bug.cgi?id=1221166] from MODIFIED to ON_QA
> Transaction prepare is not replicated to new owners during state transfer
> -------------------------------------------------------------------------
>
> Key: ISPN-5462
> URL: https://issues.jboss.org/browse/ISPN-5462
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 7.2.1.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Labels: testsuite_stability
> Fix For: 7.2.2.Final, 8.0.0.Alpha1
>
>
> This is related to ISPN-5460, and I've seen it in the same map/reduce tests.
> {{TransactionTable}} updates its topology id *before* the new topology is installed in {{StateConsumerImpl}}. This means a transaction could be created with the new topology id, while the prepare is replicated to the owners in the old topology - meaning the new owners never receive the prepare and the commit then fails.
> Note: without the ISPN-4546 fix, it would have reported success, but it wouldn't have updated the keys.
> {noformat}
> 09:25:57,749 TRACE (asyncTransportThread-0,NodeA:) [StateConsumerImpl] Received new topology for cache resultCache2, isRebalance = true, isMember = true, topology = CacheTopology{id=1, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 1, owners = (1)[NodeA-6285: 1]}, pendingCH=ReplicatedConsistentHash{ns = 1, owners = (2)[NodeA-6285: 1, NodeB-17038: 0]}, unionCH=ReplicatedConsistentHash{ns = 1, owners = (2)[NodeA-6285: 1, NodeB-17038: 0]}, actualMembers=[NodeA-6285, NodeB-17038]}
> 09:25:57,749 TRACE (asyncTransportThread-1,NodeA:) [TransactionTable] Created a new local transaction: LocalXaTransaction{xid=null} LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=1, stateTransferFlag=null} org.infinispan.transaction.xa.LocalXaTransaction@3c96
> 09:25:57,749 TRACE (asyncTransportThread-1,NodeA:) [TransactionXaAdapter] end called on tx GlobalTransaction:<NodeA-6285>:15510:local(resultCache2)
> 09:25:57,749 TRACE (asyncTransportThread-1,NodeA:) [StateTransferInterceptor] handleTxCommand for command PrepareCommand {modifications=[PutKeyValueCommand{key=Boston, value=1, flags=null, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedMetadata{version=null}, successful=true}], onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-6285>:15510:local, cacheName='resultCache2', topologyId=-1}, origin null
> 09:25:57,749 TRACE (asyncTransportThread-1,NodeA:) [JGroupsTransport] dests=[NodeA-6285], command=PrepareCommand {modifications=[PutKeyValueCommand{key=Boston, value=1, flags=null, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedMetadata{version=null}, successful=true}], onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-6285>:15510:local, cacheName='resultCache2', topologyId=0}, mode=SYNCHRONOUS_IGNORE_LEAVERS, timeout=15000
> 09:25:57,749 TRACE (asyncTransportThread-1,NodeA:) [LocalTransaction] Adding remote locks on [NodeA-6285]. Remote locks are null
> 09:25:57,749 TRACE (asyncTransportThread-1,NodeA:) [TransactionCoordinator] Committing transaction GlobalTransaction:<NodeA-6285>:15510:local
> 09:25:57,749 TRACE (asyncTransportThread-0,NodeA:) [StateConsumerImpl] Lock State Transfer in Progress for topology ID 1
> // StateConsumerImpl's topology is updated here
> 09:25:57,749 TRACE (asyncTransportThread-0,NodeA:) [StateConsumerImpl] Start keeping track of keys for rebalance
> 09:25:57,751 TRACE (remote-thread-5,NodeB:) [StateTransferManagerImpl] Installing new cache topology CacheTopology{id=1, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 1, owners = (1)[NodeA-6285: 1]}, pendingCH=ReplicatedConsistentHash{ns = 1, owners = (2)[NodeA-6285: 1, NodeB-17038: 0]}, unionCH=ReplicatedConsistentHash{ns = 1, owners = (2)[NodeA-6285: 1, NodeB-17038: 0]}, actualMembers=[NodeA-6285, NodeB-17038]} on cache resultCache2
> 09:25:57,752 TRACE (remote-thread-5,NodeB:) [StateConsumerImpl] Requesting transactions for segments [0] of cache resultCache2 from node NodeA-6285
> 09:25:57,752 TRACE (remote-thread-2,NodeA:) [StateProviderImpl] Skipping transaction LocalXaTransaction{xid=< formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffffc0a80a66:ab12:55504ae3:5806, node_name=1, branch_uid=0:ffffc0a80a66:ab12:55504ae3:5807, subordinatenodename=null, eis_name=0 >} LocalTransaction{remoteLockedNodes=[NodeA-6285], isMarkedForRollback=false, lockedKeys=[Boston], backupKeyLocks=null, topologyId=1, stateTransferFlag=null} org.infinispan.transaction.xa.LocalXaTransaction@3c96 as it was started in the current topology or by a leaver
> 09:25:57,753 TRACE (remote-thread-3,NodeB:) [InboundInvocationHandlerImpl] Calling perform() on CommitCommand {gtx=GlobalTransaction:<NodeA-6285>:15510:remote, cacheName='resultCache2', topologyId=1}
> 09:25:57,753 TRACE (remote-thread-3,NodeB:) [AbstractTransactionBoundaryCommand] Did not find a RemoteTransaction for GlobalTransaction:<NodeA-6285>:15510:remote, completed successfully? false
> 09:25:57,758 ERROR (remote-thread-3,NodeB:) [InboundInvocationHandlerImpl] ISPN000260: Exception executing command
> java.lang.IllegalStateException: Remote transaction not found
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 8 months