[JBoss JIRA] (ISPN-4743) Rebalance can hang after the coordinator and another node leave
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-4743?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-4743:
-------------------------------
Status: Open (was: New)
> Rebalance can hang after the coordinator and another node leave
> ---------------------------------------------------------------
>
> Key: ISPN-4743
> URL: https://issues.jboss.org/browse/ISPN-4743
> Project: Infinispan
> Issue Type: Bug
> Components: Core, State Transfer
> Affects Versions: 7.0.0.Beta2
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Labels: testsuite_stability
> Fix For: 7.0.0.CR1
>
>
> This caused a failure in {{ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus}}.
> When the coordinator changes, the new coordinator first sends a {{CacheTopologyControlCommand(type=CH_UPDATE)}} to reset any ongoing rebalance, then a {{CacheTopologyControlCommand(type=REBALANCE_START)}} to start a new rebalance with the remaining members. If another node leaves afterwards, the coordinator sends yet another {{CacheTopologyControlCommand(type=CH_UPDATE)}} to remove the leaver from the CHs.
> If one node (in this case the coordinator itself) processes the last {{CH_UPDATE}} before the other two commands, it will fail to confirm the rebalance, and the cache will stay in "rebalancing" state until another node joins or leaves.
--
This message was sent by Atlassian JIRA
(v6.3.1#6329)
11 years, 6 months
[JBoss JIRA] (ISPN-4746) NPE when preloading cache with DeltaAware values
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-4746?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes reassigned ISPN-4746:
---------------------------------------
Assignee: Gustavo Fernandes (was: Dan Berindei)
> NPE when preloading cache with DeltaAware values
> ------------------------------------------------
>
> Key: ISPN-4746
> URL: https://issues.jboss.org/browse/ISPN-4746
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 7.0.0.Beta2
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
>
> {code:java}
> public class DeltaAwarePreloadTest extends MultipleCacheManagersTest {
> private static final int CLUSTER_SIZE = 1;
> @Override
> protected void createCacheManagers() throws Throwable {
> ConfigurationBuilder c = getDefaultClusteredCacheConfig(CacheMode.DIST_SYNC, false);
> c.persistence().addStore(new DummyInMemoryStoreConfigurationBuilder(c.persistence()).storeName(getClass().getSimpleName())).preload(true);
> createCluster(c, CLUSTER_SIZE);
> waitForClusterToForm();
> }
> @Test
> public void testPreloadOnStart() throws PersistenceException {
> Cache<Object, Object> cache = caches().get(0);
> cache.put(1, new TestDeltaAware());
> cache.stop();
> cache.start();
> }
> }
> {code}
> During preload, the {{NonTxDistributionInterceptor}} decides that is requires values from previous owners and tries to check if the local node is the primary owner. At this point, the WriteCH is null since no topology was ever updated. Here's the stacktrace:
> {code}
> Caused by: org.infinispan.persistence.spi.PersistenceException: Unable to preload!
> at org.infinispan.persistence.manager.PersistenceManagerImpl.preloadKey(PersistenceManagerImpl.java:620)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.access$000(PersistenceManagerImpl.java:70)
> at org.infinispan.persistence.manager.PersistenceManagerImpl$1.processEntry(PersistenceManagerImpl.java:228)
> at org.infinispan.persistence.dummy.DummyInMemoryStore.process(DummyInMemoryStore.java:165)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.preload(PersistenceManagerImpl.java:220)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
> ... 37 more
> Caused by: java.lang.NullPointerException
> at org.infinispan.distribution.impl.DistributionManagerImpl.getWriteConsistentHash(DistributionManagerImpl.java:115)
> at org.infinispan.distribution.impl.DistributionManagerImpl.getConsistentHash(DistributionManagerImpl.java:105)
> at org.infinispan.distribution.impl.DistributionManagerImpl.getPrimaryLocation(DistributionManagerImpl.java:95)
> at org.infinispan.interceptors.locking.ClusteringDependentLogic$DistributionLogic.localNodeIsPrimaryOwner(ClusteringDependentLogic.java:395)
> at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.remoteGetBeforeWrite(NonTxDistributionInterceptor.java:131)
> at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:195)
> at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:72)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.DistCacheWriterInterceptor.visitPutKeyValueCommand(DistCacheWriterInterceptor.java:72)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java:113)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:376)
> at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:464)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.1#6329)
11 years, 6 months
[JBoss JIRA] (ISPN-4732) Can't acquire locks with lockAcquisitionTimeout = 0 immediately after topology change
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-4732?page=com.atlassian.jira.plugin.... ]
Adrian Nistor updated ISPN-4732:
--------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
Integrated. Thanks!
> Can't acquire locks with lockAcquisitionTimeout = 0 immediately after topology change
> -------------------------------------------------------------------------------------
>
> Key: ISPN-4732
> URL: https://issues.jboss.org/browse/ISPN-4732
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 7.0.0.Beta2
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Blocker
> Labels: testsuite_stability
> Fix For: 7.0.0.CR1
>
>
> After a topology change, as long as there are still transactions from the previous topology, it's not possible to acquire locks with a lockAcquisitionTimeout of 0. {{AbstractTxLockingInterceptor}} first checks for backup locks, and throws a TimeoutException if the remaining time is <= 0. But {{waitForTransactionsToComplete}} already throws a TimeoutException if the timeout was exceeded, so the additional check is not needed.
> This caused a random failure in SyncDistTxClusterExtendedStatisticLogicTest.testConditionalPutTxAndReadOnlyTx:
> {noformat}
> 20:43:28,771 ERROR (testng-SyncDistTxClusterExtendedStatisticLogicTest:) [UnitTestTestNGListener] Test testConditionalPutTxAndReadOnlyTx(org.infinispan.stats.logic.SyncDistTxClusterExtendedStatisticLogicTest) failed.
> javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1177)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
> at org.infinispan.stats.BaseTxClusterExtendedStatisticLogicTest.testStats(BaseTxClusterExtendedStatisticLogicTest.java:242)
> at org.infinispan.stats.BaseTxClusterExtendedStatisticLogicTest.testConditionalPutTxAndReadOnlyTx(BaseTxClusterExtendedStatisticLogicTest.java:104)
> Caused by: org.infinispan.commons.CacheException: Could not prepare.
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:46)
> at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:273)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:93)
> at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
> ... 23 more
> Caused by: javax.transaction.xa.XAException
> at org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:141)
> at org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:104)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:44)
> ... 28 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: Could not acquire lock on MagicKey#KEY_38{b4f7313c@NodeA-6769/0} on behalf of transaction GlobalTransaction:<NodeA-6769>:418:local.Lock is being held by null
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.newTimeoutException(AbstractTxLockingInterceptor.java:200)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockKeyAndCheckOwnership(AbstractTxLockingInterceptor.java:158)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAndRegisterBackupLock(AbstractTxLockingInterceptor.java:98)
> at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.acquireAllLocks(OptimisticLockingInterceptor.java:262)
> at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:84)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:36)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.stats.wrappers.ExtendedStatisticInterceptor.visitPrepareCommand(ExtendedStatisticInterceptor.java:122)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:124)
> at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:111)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:97)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:42)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:217)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:147)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:75)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:97)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:108)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:97)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.util.TransactionTrackInterceptor.visitPrepareCommand(TransactionTrackInterceptor.java:81)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
> at org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:121)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.1#6329)
11 years, 6 months
[JBoss JIRA] (ISPN-4747) Update README files in quickstarts
by Jiří Holuša (JIRA)
[ https://issues.jboss.org/browse/ISPN-4747?page=com.atlassian.jira.plugin.... ]
Jiří Holuša updated ISPN-4747:
------------------------------
Description:
In quickstarts README files there is often an instruction to change ISPN subsystem configuration (for example in hotrod-endpoint, hotrod-secured, etc.)
However, the ISPN subsystem version does not match latest JDG server version, so we should adjust it accordingly.
Also, if there are some places where README file can be improved (design, typo), do it.
was:
In quickstarts README files there is often an instruction to change ISPN subsystem configuration (for example in hotrod-endpoint, hotrod-secured, etc.)
However, the ISPN subsystem version does not match latest JDG server version, so we should adjust it accordingly.
> Update README files in quickstarts
> ----------------------------------
>
> Key: ISPN-4747
> URL: https://issues.jboss.org/browse/ISPN-4747
> Project: Infinispan
> Issue Type: Enhancement
> Components: Demos and Tutorials
> Reporter: Jiří Holuša
> Assignee: Jiří Holuša
> Priority: Minor
>
> In quickstarts README files there is often an instruction to change ISPN subsystem configuration (for example in hotrod-endpoint, hotrod-secured, etc.)
> However, the ISPN subsystem version does not match latest JDG server version, so we should adjust it accordingly.
> Also, if there are some places where README file can be improved (design, typo), do it.
--
This message was sent by Atlassian JIRA
(v6.3.1#6329)
11 years, 6 months
[JBoss JIRA] (ISPN-4747) Update README files in quickstarts
by Jiří Holuša (JIRA)
Jiří Holuša created ISPN-4747:
---------------------------------
Summary: Update README files in quickstarts
Key: ISPN-4747
URL: https://issues.jboss.org/browse/ISPN-4747
Project: Infinispan
Issue Type: Enhancement
Components: Demos and Tutorials
Reporter: Jiří Holuša
Assignee: Jiří Holuša
Priority: Minor
In quickstarts README files there is often an instruction to change ISPN subsystem configuration (for example in hotrod-endpoint, hotrod-secured, etc.)
However, the ISPN subsystem version does not match latest JDG server version, so we should adjust it accordingly.
--
This message was sent by Atlassian JIRA
(v6.3.1#6329)
11 years, 6 months