[infinispan-dev] Infinipedia: a stress test for InfinispanDirectory and CassandraCacheStore

Manik Surtani manik at jboss.org
Mon Oct 25 07:43:55 EDT 2010 

Guys

Do you still have this issue with the latest ALPHAs?

Cheers
Manik

On 6 Oct 2010, at 14:24, Galder Zamarreño wrote:

> Tristan, 
> 
> Not sure what the cause of the TimeoutException is but if you the test again, can you generate a thread dump when the first TE is reported so that we can see what the other threads are doing?
> 
> We might be able to see what the thread holding the lock is doing.
> 
> Cheers, 
> 
> On Oct 1, 2010, at 3:00 PM, Tristan Tarrant wrote:
> 
>> Dear all,
>> 
>> I have written a small application for stress testing my CassandraCacheStore in conjunction with the InfinispanDirectory.
>> 
>> Purpose of the test, inspired by a blog I've seen on Lucene performance, is to index a dump of English Wikipedia (see http://download.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2).
>> The dump is 27GB of XML text for 3.5M articles. I read all the pages and for each one I create a Lucene Document with three fields: the ID, the Title and the Text of each (ID and Title are also Stored Fields). I open one IndexWriter for the entire session and commit every 100 documents. I am using the following infinispan.xml:
>> 
>> 	<namedCache name="cassandraCache">	
>> 		<eviction strategy="LIRS" maxEntries="10000" />
>> 		<invocationBatching enabled="true" />
>> 		<loaders passivation="false" shared="true" preload="false">
>> 			<loader	class="net.dataforte.infinispan.loaders.cassandra.CassandraCacheStore"
>> 				fetchPersistentState="false" ignoreModifications="false"
>> 				purgeOnStartup="false">
>> 				<properties>
>> 					<property name="host" value="localhost" />
>> 					<property name="keySpace" value="Infinispan" />
>> 				</properties>
>> 				<async />
>> 			</loader>
>> 		</loaders>
>> 	</namedCache>
>> 
>> Everything goes smoothly for a while, until I get a TimeoutException when attempting to acquire the lock. This is during the IndexWriter commit() and, depending on the run, it is caused either within the InfinispanDirectory locking or within the Eviction thread. I have tried both with 4.1.0.FINAL and with 4.2.0.ALPHA2, and also replacing my CacheStore with the basic FileCacheStore but the result is the same. 
>> Basically, at some point the commit() hangs indefinitely, and after a while it breaks. Here is the stack trace:
>> 
>> 47927> Job
>> 47928> Battle of Naissus
>> 47929> Strange loop
>> 47930> Gloucester (disambiguation)
>> Committing...2010-10-01 14:48:55 EvictionManagerImpl [WARN] Could not acquire lock for eviction of _56.cfs|1518|wikiIndex
>> 2010-10-01 14:48:55 EvictionManagerImpl [WARN] Could not acquire lock for eviction of _78.cfs|6|wikiIndex
>> 2010-10-01 14:48:56 CommitLogSegment [INFO] Creating new commitlog segment /tmp/infiniwiki/commitlog/CommitLog-1285937336075.log
>> 2010-10-01 14:49:05 InvocationContextInterceptor [ERROR] Execution error: 
>> org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [10 seconds] on key [_7w.tis|RL|wikiIndex] for requestor [Thread[main,5,main]]! Lock held by [GlobalTransaction:<null>:66883:local]
>> 	at org.infinispan.container.EntryFactoryImpl.acquireLock(EntryFactoryImpl.java:218)
>> 	at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:153)
>> 	at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:106)
>> 	at org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:234)
>> 	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
>> 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>> 	at org.infinispan.interceptors.CacheStoreInterceptor.visitPutKeyValueCommand(CacheStoreInterceptor.java:194)
>> 	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
>> 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>> 	at org.infinispan.interceptors.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java:79)
>> 	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
>> 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>> 	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
>> 	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
>> 	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
>> 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>> 	at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:172)
>> 	at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:120)
>> 	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
>> 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>> 	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:79)
>> 	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:56)
>> 	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
>> 	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
>> 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
>> 	at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:76)
>> 	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
>> 	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
>> 	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273)
>> 	at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:436)
>> 	at org.infinispan.CacheSupport.putIfAbsent(CacheSupport.java:40)
>> 	at org.infinispan.lucene.readlocks.DistributedSegmentReadLocker.aquireReadLock(DistributedSegmentReadLocker.java:130)
>> 	at org.infinispan.lucene.InfinispanDirectory.openInput(InfinispanDirectory.java:269)
>> 	at org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter.java:221)
>> 	at org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java:184)
>> 	at org.apache.lucene.index.DocumentsWriter.createCompoundFile(DocumentsWriter.java:672)
>> 	at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4418)
>> 	at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4264)
>> 	at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4255)
>> 	at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:4133)
>> 	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4206)
>> 	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4179)
>> 	at net.dataforte.infiniwiki.WikiHandler.endElement(WikiHandler.java:90)
>> 	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:601)
>> 	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1782)
>> 	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2938)
>> 	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
>> 	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:511)
>> 	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:808)
>> 	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
>> 	at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119)
>> 	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205)
>> 	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:522)
>> 	at javax.xml.parsers.SAXParser.parse(SAXParser.java:395)
>> 	at javax.xml.parsers.SAXParser.parse(SAXParser.java:198)
>> 	at net.dataforte.infiniwiki.InfiniWiki.doIndex(InfiniWiki.java:126)
>> 	at net.dataforte.infiniwiki.InfiniWiki.main(InfiniWiki.java:152)
>> 
>> Tristan
>> 
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> --
> Galder Zamarreño
> Sr. Software Engineer
> Infinispan, JBoss Cache
> 
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Manik Surtani
manik at jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org

More information about the infinispan-dev mailing list