[infinispan-dev] Infinipedia: a stress test for InfinispanDirectory and CassandraCacheStore

Fri Oct 1 10:26:34 EDT 2010

Hi Tristan,
two quick notes:

1) Until I fix ISPN-680 you shouldn't use eviction on a cache backing
the Lucene Directory, I'll fix it this weekend, maybe today.

2) you seem to have a lot of chunks, from your log
"_56.cfs|1518|wikiIndex" this means that the _56.cfs file is
fragmented in at least 1518 pieces, likely more.
Please set a quite-higher chunk value using the alternative constructor
InfinispanDirectory(Cache cache, String indexName, int chunkSize)
The default for chunkSize is quite small (16K) to optimize network I/O
during distribution/replication, but with such large indexes you
likely want higher chunks.
For Scarlet I'm defaulting to 7MB - be aware that you might need to
tune network/IO, in my case I had to rise the MySQL allowed max packet
size too, no idea if you have to configure something on Cassandra.

About the TimeoutException, I don't know, I hope someone from
Infinispan core could provide some hints;
if you can share this test I'll be glad to look into it, assuming it's
not complex to setup Cassandra, or a patch for an automated unit test
would be better.

Cheers,
Sanne

2010/10/1 Tristan Tarrant <tristan.tarrant at gmail.com>:
> Dear all,
> I have written a small application for stress testing my CassandraCacheStore
> in conjunction with the InfinispanDirectory.
> Purpose of the test, inspired by a blog I've seen on Lucene performance, is
> to index a dump of English Wikipedia
> (see http://download.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2).
> The dump is 27GB of XML text for 3.5M articles. I read all the pages and for
> each one I create a Lucene Document with three fields: the ID, the Title and
> the Text of each (ID and Title are also Stored Fields). I open one
> IndexWriter for the entire session and commit every 100 documents. I am
> using the following infinispan.xml:
> <namedCache name="cassandraCache">
> <eviction strategy="LIRS" maxEntries="10000" />
> <invocationBatching enabled="true" />
> <loaders passivation="false" shared="true" preload="false">
> <loader
> class="net.dataforte.infinispan.loaders.cassandra.CassandraCacheStore"
> fetchPersistentState="false" ignoreModifications="false"
> purgeOnStartup="false">
> <properties>
> <property name="host" value="localhost" />
> <property name="keySpace" value="Infinispan" />
> </properties>
> <async />
> </loader>
> </loaders>
> </namedCache>
> Everything goes smoothly for a while, until I get a TimeoutException when
> attempting to acquire the lock. This is during the IndexWriter commit() and,
> depending on the run, it is caused either within the InfinispanDirectory
> locking or within the Eviction thread. I have tried both with 4.1.0.FINAL
> and with 4.2.0.ALPHA2, and also replacing my CacheStore with the basic
> FileCacheStore but the result is the same.
> Basically, at some point the commit() hangs indefinitely, and after a while
> it breaks. Here is the stack trace:
> 47927> Job
> 47928> Battle of Naissus
> 47929> Strange loop
> 47930> Gloucester (disambiguation)
> Committing...2010-10-01 14:48:55 EvictionManagerImpl [WARN] Could not
> acquire lock for eviction of _56.cfs|1518|wikiIndex
> 2010-10-01 14:48:55 EvictionManagerImpl [WARN] Could not acquire lock for
> eviction of _78.cfs|6|wikiIndex
> 2010-10-01 14:48:56 CommitLogSegment [INFO] Creating new commitlog segment
> /tmp/infiniwiki/commitlog/CommitLog-1285937336075.log
> 2010-10-01 14:49:05 InvocationContextInterceptor [ERROR] Execution error:
> org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock
> after [10 seconds] on key [_7w.tis|RL|wikiIndex] for requestor
> [Thread[main,5,main]]! Lock held by [GlobalTransaction:<null>:66883:local]
> at
> org.infinispan.container.EntryFactoryImpl.acquireLock(EntryFactoryImpl.java:218)
> at
> org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:153)
> at
> org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:106)
> at
> org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:234)
> at
> org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
> at
> org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at
> org.infinispan.interceptors.CacheStoreInterceptor.visitPutKeyValueCommand(CacheStoreInterceptor.java:194)
> at
> org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
> at
> org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at
> org.infinispan.interceptors.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java:79)
> at
> org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
> at
> org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at
> org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at
> org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
> at
> org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
> at
> org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at
> org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:172)
> at
> org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:120)
> at
> org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
> at
> org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at
> org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:79)
> at
> org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:56)
> at
> org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
> at
> org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
> at
> org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at
> org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:76)
> at
> org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
> at
> org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
> at
> org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273)
> at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:436)
> at org.infinispan.CacheSupport.putIfAbsent(CacheSupport.java:40)
> at
> org.infinispan.lucene.readlocks.DistributedSegmentReadLocker.aquireReadLock(DistributedSegmentReadLocker.java:130)
> at
> org.infinispan.lucene.InfinispanDirectory.openInput(InfinispanDirectory.java:269)
> at
> org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter.java:221)
> at
> org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java:184)
> at
> org.apache.lucene.index.DocumentsWriter.createCompoundFile(DocumentsWriter.java:672)
> at
> org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4418)
> at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4264)
> at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4255)
> at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:4133)
> at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4206)
> at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4179)
> at net.dataforte.infiniwiki.WikiHandler.endElement(WikiHandler.java:90)
> at
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:601)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1782)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2938)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:511)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:808)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
> at
> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119)
> at
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205)
> at
> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:522)
> at javax.xml.parsers.SAXParser.parse(SAXParser.java:395)
> at javax.xml.parsers.SAXParser.parse(SAXParser.java:198)
> at net.dataforte.infiniwiki.InfiniWiki.doIndex(InfiniWiki.java:126)
> at net.dataforte.infiniwiki.InfiniWiki.main(InfiniWiki.java:152)
> Tristan
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>