Re: [infinispan-dev] Infinipedia: a stress test for InfinispanDirectory and CassandraCacheStore

Dear all, I have written a small application for stress testing my CassandraCacheStore in conjunction with the InfinispanDirectory. Purpose of the test, inspired by a blog I've seen on Lucene performance, is to index a dump of English Wikipedia (see http://download.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles....). The dump is 27GB of XML text for 3.5M articles. I read all the pages and for each one I create a Lucene Document with three fields: the ID, the Title and the Text of each (ID and Title are also Stored Fields). I open one IndexWriter for the entire session and commit every 100 documents. I am using the following infinispan.xml: <namedCache name="cassandraCache"> <eviction strategy="LIRS" maxEntries="10000" /> <invocationBatching enabled="true" /> <loaders passivation="false" shared="true" preload="false"> <loader class="net.dataforte.infinispan.loaders.cassandra.CassandraCacheStore" fetchPersistentState="false" ignoreModifications="false" purgeOnStartup="false"> <properties> <property name="host" value="localhost" /> <property name="keySpace" value="Infinispan" /> </properties> <async /> </loader> </loaders> </namedCache> Everything goes smoothly for a while, until I get a TimeoutException when attempting to acquire the lock. This is during the IndexWriter commit() and, depending on the run, it is caused either within the InfinispanDirectory locking or within the Eviction thread. I have tried both with 4.1.0.FINAL and with 4.2.0.ALPHA2, and also replacing my CacheStore with the basic FileCacheStore but the result is the same. Basically, at some point the commit() hangs indefinitely, and after a while it breaks. Here is the stack trace: 47927> Job 47928> Battle of Naissus 47929> Strange loop 47930> Gloucester (disambiguation) Committing...2010-10-01 14:48:55 EvictionManagerImpl [WARN] Could not acquire lock for eviction of _56.cfs|1518|wikiIndex 2010-10-01 14:48:55 EvictionManagerImpl [WARN] Could not acquire lock for eviction of _78.cfs|6|wikiIndex 2010-10-01 14:48:56 CommitLogSegment [INFO] Creating new commitlog segment /tmp/infiniwiki/commitlog/CommitLog-1285937336075.log 2010-10-01 14:49:05 InvocationContextInterceptor [ERROR] Execution error: org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [10 seconds] on key [_7w.tis|RL|wikiIndex] for requestor [Thread[main,5,main]]! Lock held by [GlobalTransaction:<null>:66883:local] at org.infinispan.container.EntryFactoryImpl.acquireLock(EntryFactoryImpl.java:218) at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:153) at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:106) at org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:234) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.CacheStoreInterceptor.visitPutKeyValueCommand(CacheStoreInterceptor.java:194) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java:79) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:172) at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:120) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:79) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:56) at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:76) at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273) at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:436) at org.infinispan.CacheSupport.putIfAbsent(CacheSupport.java:40) at org.infinispan.lucene.readlocks.DistributedSegmentReadLocker.aquireReadLock(DistributedSegmentReadLocker.java:130) at org.infinispan.lucene.InfinispanDirectory.openInput(InfinispanDirectory.java:269) at org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter.java:221) at org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java:184) at org.apache.lucene.index.DocumentsWriter.createCompoundFile(DocumentsWriter.java:672) at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4418) at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4264) at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4255) at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:4133) at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4206) at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4179) at net.dataforte.infiniwiki.WikiHandler.endElement(WikiHandler.java:90) at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:601) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1782) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2938) at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:511) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:808) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737) at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119) at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:522) at javax.xml.parsers.SAXParser.parse(SAXParser.java:395) at javax.xml.parsers.SAXParser.parse(SAXParser.java:198) at net.dataforte.infiniwiki.InfiniWiki.doIndex(InfiniWiki.java:126) at net.dataforte.infiniwiki.InfiniWiki.main(InfiniWiki.java:152) Tristan _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev