[infinispan-dev] Infinipedia: a stress test for InfinispanDirectory and CassandraCacheStore

[Tristan Tarrant]

Dear all,

I have written a small application for stress testing my CassandraCacheStore
in conjunction with the InfinispanDirectory.

Purpose of the test, inspired by a blog I've seen on Lucene performance, is
to index a dump of English Wikipedia (see
http://download.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2
).
The dump is 27GB of XML text for 3.5M articles. I read all the pages and for
each one I create a Lucene Document with three fields: the ID, the Title and
the Text of each (ID and Title are also Stored Fields). I open one
IndexWriter for the entire session and commit every 100 documents. I am
using the following infinispan.xml:

<namedCache name="cassandraCache">
<eviction strategy="LIRS" maxEntries="10000" />
<invocationBatching enabled="true" />
<loaders passivation="false" shared="true" preload="false">
<loader
class="net.dataforte.infinispan.loaders.cassandra.CassandraCacheStore"
fetchPersistentState="false" ignoreModifications="false"
purgeOnStartup="false">
<properties>
<property name="host" value="localhost" />
<property name="keySpace" value="Infinispan" />
</properties>
<async />
</loader>
</loaders>
</namedCache>

Everything goes smoothly for a while, until I get a TimeoutException when
attempting to acquire the lock. This is during the IndexWriter commit() and,
depending on the run, it is caused either within the InfinispanDirectory
locking or within the Eviction thread. I have tried both with 4.1.0.FINAL
and with 4.2.0.ALPHA2, and also replacing my CacheStore with the basic
FileCacheStore but the result is the same.
Basically, at some point the commit() hangs indefinitely, and after a while
it breaks. Here is the stack trace:

47927> Job
47928> Battle of Naissus
47929> Strange loop
47930> Gloucester (disambiguation)
Committing...2010-10-01 14:48:55 EvictionManagerImpl [WARN] Could not
acquire lock for eviction of _56.cfs|1518|wikiIndex
2010-10-01 14:48:55 EvictionManagerImpl [WARN] Could not acquire lock for
eviction of _78.cfs|6|wikiIndex
2010-10-01 14:48:56 CommitLogSegment [INFO] Creating new commitlog segment
/tmp/infiniwiki/commitlog/CommitLog-1285937336075.log
2010-10-01 14:49:05 InvocationContextInterceptor [ERROR] Execution error:
org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock
after [10 seconds] on key [_7w.tis|RL|wikiIndex] for requestor
[Thread[main,5,main]]! Lock held by [GlobalTransaction:<null>:66883:local]
at
org.infinispan.container.EntryFactoryImpl.acquireLock(EntryFactoryImpl.java:218)
at
org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:153)
at
org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:106)
at
org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:234)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.CacheStoreInterceptor.visitPutKeyValueCommand(CacheStoreInterceptor.java:194)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java:79)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:172)
at
org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:120)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:79)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:56)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:76)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273)
at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:436)
at org.infinispan.CacheSupport.putIfAbsent(CacheSupport.java:40)
at
org.infinispan.lucene.readlocks.DistributedSegmentReadLocker.aquireReadLock(DistributedSegmentReadLocker.java:130)
at
org.infinispan.lucene.InfinispanDirectory.openInput(InfinispanDirectory.java:269)
at
org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter.java:221)
at
org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java:184)
at
org.apache.lucene.index.DocumentsWriter.createCompoundFile(DocumentsWriter.java:672)
at
org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4418)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4264)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4255)
at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:4133)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4206)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4179)
at net.dataforte.infiniwiki.WikiHandler.endElement(WikiHandler.java:90)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:601)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1782)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2938)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:511)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:808)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
at
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205)
at
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:522)
at javax.xml.parsers.SAXParser.parse(SAXParser.java:395)
at javax.xml.parsers.SAXParser.parse(SAXParser.java:198)
at net.dataforte.infiniwiki.InfiniWiki.doIndex(InfiniWiki.java:126)
at net.dataforte.infiniwiki.InfiniWiki.main(InfiniWiki.java:152)

Tristan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20101001/98bb8aef/attachment-0001.html