[JBoss JIRA] (ISPN-3868) Deadlock in RemoteCache getAsync
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3868?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration updated ISPN-3868:
------------------------------------------
Bugzilla Update: Perform
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=1073331
> Deadlock in RemoteCache getAsync
> --------------------------------
>
> Key: ISPN-3868
> URL: https://issues.jboss.org/browse/ISPN-3868
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 6.0.0.Final
> Environment: RemoteCahe component of 6.0.0.Final
> Reporter: Alexander Furer
> Assignee: Mircea Markus
> Labels: remote
> Fix For: 7.0.0.Alpha1, 7.0.0.Final
>
>
> Here is the implementation of remoteCahe.getAsync() :
> {code}
> public NotifyingFuture<V> getAsync(final K key) {
> assertRemoteCacheManagerIsStarted();
> final NotifyingFutureImpl<V> result = new NotifyingFutureImpl<V>();
> Future<V> future = executorService.submit(new Callable<V>() {
> @Override
> public V call() throws Exception {
> V toReturn = get(key);
> result.notifyFutureCompletion();
> return toReturn;
> }
> });
> result.setExecuting(future);
> return result;
> }
> {code}
> 2 problems here :
> 1. Callable's call method might be called BEFORE calling client had a chance to add listener (i.e. getAsync is not returned yet), in this case its' listener futureDone method will never be called.
> 2. Even case #1 has not happened and notifyFutureCompletion is called on listener, but the future is not resolved yet : "call" has not returned, that's why the future that is passed to listener is not resolved, and calling future.get from listener blocks forever.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 2 months
[JBoss JIRA] (ISPN-3803) Stale locks during state transfer in non-tx caches
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3803?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration updated ISPN-3803:
------------------------------------------
Bugzilla Update: Perform
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=1073327
> Stale locks during state transfer in non-tx caches
> --------------------------------------------------
>
> Key: ISPN-3803
> URL: https://issues.jboss.org/browse/ISPN-3803
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer, Transactions
> Affects Versions: 6.0.0.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Labels: testsuite_stability
> Fix For: 7.0.0.Alpha1, 7.0.0.Final
>
>
> The problem is that {{SingleKeyNonTxInvocationContext.addLockedKey()}} only sets a {{isLocked}} flag, the actual key is set when the entry is wrapped and inserted in the context. If the topology changes between the lock acquisition and the entry wrapping, {{SingleKeyNonTxInvocationContext.getLockedKeys()}} will return an empty set and the lock won't be released.
> Future commands won't try to acquire a lock on this node, so the stale lock is harmless most of the time. But if there was already a command waiting for the lock, that command will time out (instead of retrying on the new primary owner).
> This causes random failures in NonTxPutIfAbsentDuringJoinStressTest:
> {noformat}
> 12:34:50,960 TRACE (ForkThread-3,NonTxPutIfAbsentDuringJoinStressTest:___defaultcache) [InvocationContextInterceptor] Invoked with command PutKeyValueCommand{key=key_52, value=value_52_2, flags=null, putIfAbsent=true, valueMatcher=MATCH_EXPECTED, metadata=EmbeddedMetadata{version=null}, successful=true} and InvocationContext [org.infinispan.context.SingleKeyNonTxInvocationContext@478c9796]
> 12:34:50,960 TRACE (ForkThread-3,NonTxPutIfAbsentDuringJoinStressTest:___defaultcache) [NonTransactionalLockingInterceptor] Are (NodeA-19338) we the lock owners for key 'key_52'? true
> 12:34:50,960 TRACE (ForkThread-3,NonTxPutIfAbsentDuringJoinStressTest:___defaultcache) [LockManagerImpl] Attempting to lock key_52 with acquisition timeout of 10000 millis
> 12:34:50,962 TRACE (remote-thread-1,NodeA:___defaultcache) [NonTransactionalLockingInterceptor] Are (NodeA-19338) we the lock owners for key 'key_52'? true
> 12:34:50,962 TRACE (remote-thread-1,NodeA:___defaultcache) [LockManagerImpl] Attempting to lock key_52 with acquisition timeout of 10000 millis
> 12:34:50,965 TRACE (asyncTransportThread-0,NodeA:___defaultcache) [StateConsumerImpl] Received new topology for cache ___defaultcache, isRebalance = false, isMember = true, topology = CacheTopology{id=4, currentCH=DefaultConsistentHash{numSegments=60, numOwners=2, members=[NodeA-19338, NodeB-49967, NodeC-56763]}, pendingCH=null}
> 12:34:50,966 TRACE (ForkThread-3,NonTxPutIfAbsentDuringJoinStressTest:___defaultcache) [LockManagerImpl] Successfully acquired lock key_52!
> *** 12:34:50,966 TRACE (ForkThread-3,NonTxPutIfAbsentDuringJoinStressTest:___defaultcache) [EntryWrappingInterceptor] Wrapping entry 'key_52'? false
> 12:34:50,966 TRACE (ForkThread-3,NonTxPutIfAbsentDuringJoinStressTest:___defaultcache) [StateTransferInterceptor] Retrying command because of topology change: PutKeyValueCommand{key=key_52, value=value_52_2, flags=null, putIfAbsent=true, valueMatcher=MATCH_EXPECTED, metadata=EmbeddedMetadata{version=null}, successful=true}
> 12:34:50,978 TRACE (ForkThread-3,NonTxPutIfAbsentDuringJoinStressTest:___defaultcache) [NonTransactionalLockingInterceptor] Are (NodeA-19338) we the lock owners for key 'key_52'? false
> 12:34:50,978 TRACE (ForkThread-3,NonTxPutIfAbsentDuringJoinStressTest:___defaultcache) [EntryWrappingInterceptor] Wrapping entry 'key_52'? false
> 12:34:50,978 TRACE (ForkThread-3,NonTxPutIfAbsentDuringJoinStressTest:___defaultcache) [BaseDistributionInterceptor] I'm not the primary owner, so sending the command to the primary owner(NodeC-56763) in order to be forwarded
> 12:34:51,007 TRACE (ForkThread-3,NonTxPutIfAbsentDuringJoinStressTest:___defaultcache) [EntryWrappingInterceptor] The return value is value_52_0
> 12:35:00,963 TRACE (remote-thread-1,NodeA:___defaultcache) [ReentrantPerEntryLockContainer] Timed out attempting to acquire lock for key key_52 after 10 seconds
> 12:35:00,963 DEBUG (remote-thread-1,NodeA:___defaultcache) [LockManagerImpl] Failed to acquire lock key_52, owner is Thread[ForkThread-3,NonTxPutIfAbsentDuringJoinStressTest,5,]
> 12:35:00,963 DEBUG (remote-thread-1,NodeA:___defaultcache) [LockManagerImpl] This transaction (Thread[remote-thread-1,NodeA,5,main]) already owned locks []
> 12:35:00,966 ERROR (testng-NonTxPutIfAbsentDuringJoinStressTest:) [UnitTestTestNGListener] Test testNodeJoiningDuringPutIfAbsent(org.infinispan.distribution.rehash.NonTxPutIfAbsentDuringJoinStressTest) failed.
> java.util.concurrent.ExecutionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from NodeA-19338, see cause for remote stack trace
> Caused by: org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [10 seconds] on key [key_52] for requestor [Thread[remote-thread-1,NodeA,5,main]]! Lock held by [Thread[ForkThread-3,NonTxPutIfAbsentDuringJoinStressTest,5,]]{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 2 months
[JBoss JIRA] (ISPN-3936) State transfer should not modify shared cache stores
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-3936?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-3936:
------------------------------------
After a closer look, it turns out the entries that are no longer local are deleted using an InvalidateCommand, which does not write to the cache stores. So the only issue is that iterating over the entire cache store can take a lot of time (and memory, because of ISPN-3937).
> State transfer should not modify shared cache stores
> ----------------------------------------------------
>
> Key: ISPN-3936
> URL: https://issues.jboss.org/browse/ISPN-3936
> Project: Infinispan
> Issue Type: Bug
> Components: Core, State Transfer
> Affects Versions: 6.0.0.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Fix For: 7.0.0.Final, 7.0.0.Alpha2
>
>
> When installing a new topology, we invalidate the cache entries that are no longer mapped to the local node. We also iterate over the entries in the cache stores and delete the ones that are no longer local, but this should only happen if the cache store is not shared.
> We had similar issues in the past, but this seems to be related to the new cache store API introduced in 6.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 2 months
[JBoss JIRA] (ISPN-3672) Remote Query test failure in case of clustered hotrod server configured with Infinispan directory_provider
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-3672?page=com.atlassian.jira.plugin.... ]
Adrian Nistor updated ISPN-3672:
--------------------------------
Fix Version/s: 7.0.0.Final
> Remote Query test failure in case of clustered hotrod server configured with Infinispan directory_provider
> ----------------------------------------------------------------------------------------------------------
>
> Key: ISPN-3672
> URL: https://issues.jboss.org/browse/ISPN-3672
> Project: Infinispan
> Issue Type: Bug
> Components: Embedded Querying
> Reporter: Anna Manukyan
> Assignee: Adrian Nistor
> Fix For: 7.0.0.Final
>
> Attachments: MultiHotRodServerIspnDirQueryTest.java, MultiHotRodServerIspnDirReplQueryTest.java, MultiHotRodServerQueryTest.java
>
>
> I've added new tests, where the cache configuration is following:
> 2 clustered HotRod Servers configured with indexing enabled caches and using Infinispan as a directory_provider.
> When the test is putting some data into the caches and then perform query on it, the query doesn't return any results. Even sometimes when getting the entry using key from the first cache, the entry is null.
> The failing tests are MultiHotRodServerIspnDirReplQueryTest and MultiHotRodServerIspnDirQueryTest which are extended from MultiHotRodServerQueryTest.
> In the first case the ISPN directory related caches are replicated, in the second case the caches are local.
> You can find the test attached.
> Also, if I'm trying to add the following line to the cache configuration:
> {code}
> .addProperty("default.indexmanager", "org.infinispan.query.indexmanager.InfinispanIndexManager")
> {code}
> then, I'm getting the following exception:
> {code}
> org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for message id[2] returned server error (status=0x85): org.infinispan.commons.CacheException: java.lang.NoSuchMethodError: org.apache.avro.io.BinaryEncoder: method <init>()V not found
> at org.infinispan.client.hotrod.impl.protocol.Codec10.checkForErrorsInResponseStatus(Codec10.java:143)
> at org.infinispan.client.hotrod.impl.protocol.Codec10.readHeader(Codec10.java:99)
> at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:56)
> at org.infinispan.client.hotrod.impl.operations.AbstractKeyValueOperation.sendPutOperation(AbstractKeyValueOperation.java:50)
> at org.infinispan.client.hotrod.impl.operations.PutOperation.executeOperation(PutOperation.java:30)
> at org.infinispan.client.hotrod.impl.operations.PutOperation.executeOperation(PutOperation.java:19)
> at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:46)
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:213)
> at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:79)
> at org.infinispan.client.hotrod.query.MultiHotRodServerQueryTest.testAttributeQuery(MultiHotRodServerQueryTest.java:68)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 2 months
[JBoss JIRA] (ISPN-4062) Infinispan directory server module is missing some dependecies
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-4062?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration updated ISPN-4062:
------------------------------------------
Bugzilla Update: Perform
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=1073086
> Infinispan directory server module is missing some dependecies
> --------------------------------------------------------------
>
> Key: ISPN-4062
> URL: https://issues.jboss.org/browse/ISPN-4062
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Affects Versions: 6.0.0.Final
> Reporter: Adrian Nistor
> Assignee: Adrian Nistor
> Labels: 6.3remotequeries, 621
> Fix For: 7.0.0.Final, 7.0.0.Alpha2
>
>
> Configuring an indexed cache with 'infinispan' provider will result in an exception at server startup due to missing dependencies from module.xml:
> {code}
> 19:01:59,862 INFO [org.jboss.as.clustering.infinispan] (CacheStartThread,null,LuceneIndexesData) JBAS010281: Started LuceneIndexesData cache from local container
> 19:01:59,868 INFO [org.infinispan.jmx.CacheJmxRegistration] (CacheStartThread,null,LuceneIndexesLocking) ISPN000031: MBeans were successfully registered to the platform MBean server.
> 19:01:59,868 INFO [org.jboss.as.clustering.infinispan] (CacheStartThread,null,LuceneIndexesLocking) JBAS010281: Started LuceneIndexesLocking cache from local container
> 19:01:59,883 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-6) MSC00001: Failed to start service jboss.infinispan.local.addressbook: org.jboss.msc.service.StartException in service jboss.infinispan.local.addressbook: Failed to start service
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1767) [jboss-msc-1.0.4.GA.jar:1.0.4.GA]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_40]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_40]
> at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_40]
> Caused by: java.lang.NoClassDefFoundError: javax/management/MBeanRegistrationException
> at java.lang.Class.forName0(Native Method) [rt.jar:1.7.0_40]
> at java.lang.Class.forName(Class.java:270) [rt.jar:1.7.0_40]
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2247)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2211)
> at org.infinispan.util.logging.LogFactory.getLog(LogFactory.java:19)
> at org.infinispan.lucene.impl.DirectoryBuilderImpl.<clinit>(DirectoryBuilderImpl.java:23)
> at org.infinispan.lucene.directory.DirectoryBuilder.newDirectoryInstance(DirectoryBuilder.java:25)
> at org.hibernate.search.infinispan.impl.InfinispanDirectoryProvider.start(InfinispanDirectoryProvider.java:103)
> at org.hibernate.search.indexes.impl.DirectoryBasedIndexManager.initialize(DirectoryBasedIndexManager.java:103)
> at org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManager(IndexManagerHolder.java:261)
> at org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManager(IndexManagerHolder.java:528)
> at org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManagers(IndexManagerHolder.java:495)
> at org.hibernate.search.indexes.impl.IndexManagerHolder.buildEntityIndexBinding(IndexManagerHolder.java:104)
> at org.hibernate.search.spi.SearchFactoryBuilder.initDocumentBuilders(SearchFactoryBuilder.java:363)
> at org.hibernate.search.spi.SearchFactoryBuilder.buildNewSearchFactory(SearchFactoryBuilder.java:219)
> at org.hibernate.search.spi.SearchFactoryBuilder.buildSearchFactory(SearchFactoryBuilder.java:143)
> at org.infinispan.query.impl.LifecycleManager.getSearchFactory(LifecycleManager.java:213)
> at org.infinispan.query.impl.LifecycleManager.cacheStarting(LifecycleManager.java:73)
> at org.infinispan.factories.ComponentRegistry.notifyCacheStarting(ComponentRegistry.java:228)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:214)
> at org.infinispan.CacheImpl.start(CacheImpl.java:674)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:553)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:516)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:398)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:412)
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:89)
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:80)
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.4.GA.jar:1.0.4.GA]
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.4.GA.jar:1.0.4.GA]
> ... 3 more
> Caused by: java.lang.ClassNotFoundException: javax.management.MBeanRegistrationException from [Module "org.infinispan.lucene:main" from local module loader @136d1b4 (finder: local module finder @19dc4 (roots: /home/adrian/work/cpp-client/infinispan-server-7.0.0-SNAPSHOT/modules,/home/adrian/work/cpp-client/infinispan-server-7.0.0-SNAPSHOT/modules/system/layers/base))]
> at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:190) [jboss-modules.jar:1.2.0.CR1]
> at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:468) [jboss-modules.jar:1.2.0.CR1]
> at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:456) [jboss-modules.jar:1.2.0.CR1]
> at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398) [jboss-modules.jar:1.2.0.CR1]
> at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:120) [jboss-modules.jar:1.2.0.CR1]
> ... 33 more
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 2 months