[JBoss JIRA] (ISPN-4016) Write operations in invalidation mode can fail with OutdatedTopologyException
by Divya Mehra (JIRA)
[ https://issues.jboss.org/browse/ISPN-4016?page=com.atlassian.jira.plugin.... ]
Divya Mehra updated ISPN-4016:
------------------------------
Labels: 630 (was: )
> Write operations in invalidation mode can fail with OutdatedTopologyException
> -----------------------------------------------------------------------------
>
> Key: ISPN-4016
> URL: https://issues.jboss.org/browse/ISPN-4016
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 6.0.1.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Labels: 630
> Fix For: 7.0.0.Alpha2
>
>
> I introduced this problem with the fix for ISPN-3873.
> Invalidation commands now have a topology id, so they can wait for the initial topology to be installed on a joiner. However, that means EntryWrappingInterceptor also checks the topology id, and if it has changed it will throw an OutdatedTopologyException. The exception is propagated all the way to the caller.
> OutdatedTopologyExceptions are not useful in invalidation mode, since the invalidation is always sent to the entire cluster. So EntryWrappingInterceptor should ignore the topology id in invalidation mode.
> {noformat}
> Tests run: 4052, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 313.505 sec <<< FAILURE!testInvalidationDuringStateTransfer(org.infinispan.statetransfer.NonTxStateTransferInvalidationTest) Time elapsed: 0.004 sec <<< FAILURE!java.util.concurrent.ExecutionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from NonTxStateTransferInvalidationTest-NodeB-5833, see cause for remote stack trace
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:202)
> at org.infinispan.commons.util.concurrent.NotifyingFutureImpl.get(NotifyingFutureImpl.java:84)
> at org.infinispan.statetransfer.NonTxStateTransferInvalidationTest.testInvalidationDuringStateTransfer(NonTxStateTransferInvalidationTest.java:115)
> Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from NonTxStateTransferInvalidationTest-NodeB-5833, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:41)
> at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:66)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:547)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:280)
> at org.infinispan.interceptors.InvalidationInterceptor.invalidateAcrossCluster(InvalidationInterceptor.java:227)
> at org.infinispan.interceptors.InvalidationInterceptor.handleInvalidate(InvalidationInterceptor.java:143)
> at org.infinispan.interceptors.InvalidationInterceptor.visitPutKeyValueCommand(InvalidationInterceptor.java:80)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:326)
> at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:407)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:164)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:68)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:148)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:134)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1403)
> at org.infinispan.CacheImpl.putInternal(CacheImpl.java:881)
> at org.infinispan.CacheImpl.access$100(CacheImpl.java:106)
> at org.infinispan.CacheImpl$2.call(CacheImpl.java:1015) ... 4 more
> Caused by: org.infinispan.statetransfer.OutdatedTopologyException: Cache topology changed while the command was executing: expected 2, got 3
> at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:347)
> at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:407)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitInvalidateCommand(EntryWrappingInterceptor.java:139)
> at org.infinispan.commands.write.InvalidateCommand.acceptVisitor(InvalidateCommand.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitInvalidateCommand(AbstractLockingInterceptor.java:87)
> at org.infinispan.commands.write.InvalidateCommand.acceptVisitor(InvalidateCommand.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:111)
> at org.infinispan.commands.write.InvalidateCommand.acceptVisitor(InvalidateCommand.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:111)
> at org.infinispan.commands.write.InvalidateCommand.acceptVisitor(InvalidateCommand.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:111)
> at org.infinispan.commands.write.InvalidateCommand.acceptVisitor(InvalidateCommand.java:118)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
> at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:39)
> at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:48)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:95)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:50)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:178) ... 3 moreResults :
> Failed tests: NonTxStateTransferInvalidationTest.testInvalidationDuringStateTransfer:115 » Execution
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 3 months
[JBoss JIRA] (ISPN-4012) Timeout errors in the mass indexing tests
by Divya Mehra (JIRA)
[ https://issues.jboss.org/browse/ISPN-4012?page=com.atlassian.jira.plugin.... ]
Divya Mehra updated ISPN-4012:
------------------------------
Labels: 630 (was: )
> Timeout errors in the mass indexing tests
> -----------------------------------------
>
> Key: ISPN-4012
> URL: https://issues.jboss.org/browse/ISPN-4012
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Query
> Affects Versions: 6.0.1.Final
> Reporter: Dan Berindei
> Assignee: Sanne Grinovero
> Labels: 630
> Fix For: 7.0.0.Alpha2
>
>
> Both DistProgrammaticMassIndexTest and TopologyAwareDistMassIndexingTest failed in CI today: http://ci.infinispan.org/viewLog.html?buildId=6578&tab=buildResultsDiv&bu...
> {noformat}
> org.infinispan.util.concurrent.TimeoutException: Node DistProgrammaticMassIndexTest-NodeC-7069 timed out
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:174)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:521)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:280)
> at org.infinispan.query.indexmanager.InfinispanCommandsBackend.sendCommand(InfinispanCommandsBackend.java:87)
> at org.infinispan.query.indexmanager.InfinispanCommandsBackend.applyWork(InfinispanCommandsBackend.java:81)
> at org.infinispan.query.indexmanager.MasterSwitchDelegatingQueueProcessor.applyWork(MasterSwitchDelegatingQueueProcessor.java:53)
> at org.hibernate.search.indexes.impl.DirectoryBasedIndexManager.performOperations(DirectoryBasedIndexManager.java:126)
> at org.hibernate.search.backend.impl.WorkQueuePerIndexSplitter.commitOperations(WorkQueuePerIndexSplitter.java:63)
> at org.hibernate.search.backend.impl.BatchedQueueingProcessor.performWorks(BatchedQueueingProcessor.java:99)
> at org.hibernate.search.backend.impl.TransactionalWorker.performWork(TransactionalWorker.java:105)
> at org.infinispan.query.backend.QueryInterceptor.performSearchWorks(QueryInterceptor.java:232)
> at org.infinispan.query.backend.QueryInterceptor.purgeAllIndexes(QueryInterceptor.java:207)
> at org.infinispan.query.backend.QueryInterceptor.purgeAllIndexes(QueryInterceptor.java:199)
> at org.infinispan.query.impl.massindex.MapReduceMassIndexer.wipeExistingIndexes(MapReduceMassIndexer.java:33)
> at org.infinispan.query.impl.massindex.MapReduceMassIndexer.start(MapReduceMassIndexer.java:24)
> at org.infinispan.query.distributed.DistributedMassIndexingTest.rebuildIndexes(DistributedMassIndexingTest.java:75)
> at org.infinispan.query.distributed.DistributedMassIndexingTest.testReindexing(DistributedMassIndexingTest.java:63)
> Caused by: org.jgroups.TimeoutException: timeout sending message to DistProgrammaticMassIndexTest-NodeC-7069
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:349)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:167)
> ... 37 more
> {noformat}
> Full logs: http://ci.infinispan.org/repository/download/bt2/6578:id/logs-638.zip
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 3 months
[JBoss JIRA] (ISPN-4034) Cache RPCs should not wait for replies from nodes that have not joined the cache
by Divya Mehra (JIRA)
[ https://issues.jboss.org/browse/ISPN-4034?page=com.atlassian.jira.plugin.... ]
Divya Mehra updated ISPN-4034:
------------------------------
Labels: 630 (was: )
> Cache RPCs should not wait for replies from nodes that have not joined the cache
> --------------------------------------------------------------------------------
>
> Key: ISPN-4034
> URL: https://issues.jboss.org/browse/ISPN-4034
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 6.0.1.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Labels: 630
> Fix For: 7.0.0.Final
>
>
> In a replicated cache, commands are broadcasted to the entire cluster and the originator waits for some kind of response from all the nodes in the cluster, even they are not members of the cache.
> If a node doesn't have Infinispan's RpcDispatcher installed, it will never send any response, and replicated cache operations on the other nodes will all time out. See MissingRpcDispatcherTest for an example.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 3 months
[JBoss JIRA] (ISPN-4032) Listeners on originator node in DIST are notified even when not an owner
by Divya Mehra (JIRA)
[ https://issues.jboss.org/browse/ISPN-4032?page=com.atlassian.jira.plugin.... ]
Divya Mehra updated ISPN-4032:
------------------------------
Labels: 630 (was: )
> Listeners on originator node in DIST are notified even when not an owner
> ------------------------------------------------------------------------
>
> Key: ISPN-4032
> URL: https://issues.jboss.org/browse/ISPN-4032
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Listeners
> Affects Versions: 6.0.1.Final
> Reporter: William Burns
> Assignee: Dan Berindei
> Labels: 630
> Fix For: 7.0.0.Final
>
>
> Currently we notify listeners installed on the originator node when they perform a write command. This is done even when the originator node doesn't own the key. These notifications should only be fired by primary owner/backup owners normally and only primary if configured to do so.
> The DistListenerTest currently even asserts this behavior around line ~85.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 3 months
[JBoss JIRA] (ISPN-4027) TransactionTable.start() initialize the TxServiceCleanup thread pool even when the cache is NON_TRANSACTIONAL
by Divya Mehra (JIRA)
[ https://issues.jboss.org/browse/ISPN-4027?page=com.atlassian.jira.plugin.... ]
Divya Mehra updated ISPN-4027:
------------------------------
Labels: 630 (was: )
> TransactionTable.start() initialize the TxServiceCleanup thread pool even when the cache is NON_TRANSACTIONAL
> -------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-4027
> URL: https://issues.jboss.org/browse/ISPN-4027
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 6.0.1.Final
> Reporter: Guillermo GARCIA OCHOA
> Assignee: Mircea Markus
> Labels: 630
>
> In the {{TransactionTable.start()}} each cache creates a thread pool and a job is scheduled to clean up completed transactions.
> {code:java}
> private void start() {
> ...
> totalOrder = configuration.transaction().transactionProtocol().isTotalOrder();
> if (!totalOrder) {
> // Periodically run a task to cleanup the transaction table from completed transactions.
> ThreadFactory tf = new ThreadFactory() {
> @Override
> public Thread newThread(Runnable r) {
> String address = rpcManager != null ? rpcManager.getTransport().getAddress().toString() : "local";
> Thread th = new Thread(r, "TxCleanupService," + cacheName + "," + address);
> th.setDaemon(true);
> return th;
> }
> };
> executorService = Executors.newSingleThreadScheduledExecutor(tf);
> long interval = configuration.transaction().reaperWakeUpInterval();
> executorService.scheduleAtFixedRate(new Runnable() {
> @Override
> public void run() {
> cleanupCompletedTransactions();
> }
> }, interval, interval, TimeUnit.MILLISECONDS);
> }
> }
> {code}
> As you can see in the code, even is the cache is {{NON_TRANSACTIONAL}} the job is scheduled, consuming resources to do nothing (the {{completedTransactions}} map is always empty)
> Maybe I'm missing something, but our application profiling is showing us that these threads do nothing but they are consuming precious resources because we have more than 1000 {{NON_TRANSACTIONAL}} caches.
> (i) This can be considered when solving ISPN-3702 too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 3 months
[JBoss JIRA] (ISPN-4049) Race condition during startup if a JDBCstore use the datasource
by Divya Mehra (JIRA)
[ https://issues.jboss.org/browse/ISPN-4049?page=com.atlassian.jira.plugin.... ]
Divya Mehra updated ISPN-4049:
------------------------------
Labels: 630 (was: )
> Race condition during startup if a JDBCstore use the datasource
> ---------------------------------------------------------------
>
> Key: ISPN-4049
> URL: https://issues.jboss.org/browse/ISPN-4049
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 6.0.1.Final
> Reporter: Wolf-Dieter Fink
> Assignee: Mircea Markus
> Labels: 630
>
> It looks like that the infinispan subsystem does not have a dependency to the datasource subsystem and both are started in parallel.
> Sporadically it happens that the JDBCstore is accessing the datasource for preloading the cache but the datasource is not ready.
> The test is a single StringKeyedJdbcStore cache (replicated, preloaded, no eviction)
> From the log it is shown that the DS is bound a couple of millis after the failure:
> xx:xx:xx,849 ERROR
> Caused by: org.infinispan.CacheException: Unable to start cache loaders
> Caused by: org.infinispan.loaders.CacheLoaderException: javax.naming.NameNotFoundException: Error looking up datasources/OracleDS, service service jboss.naming.context.java.jboss.datasources.OracleDS is not started
> Caused by: javax.naming.NameNotFoundException: Error looking up datasources/OracleDS, service service jboss.naming.context.java.jboss.datasources.OracleDS is not started
> xx:xx:xx,860 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-5) JBAS010400: Bound data source [java:jboss/datasources/OracleDS]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 3 months