Dear all,<div><br></div><div>I have written a small application for stress testing my CassandraCacheStore in conjunction with the InfinispanDirectory.</div><div><br></div><div>Purpose of the test, inspired by a blog I've seen on Lucene performance, is to index a dump of English Wikipedia (see <a href="http://download.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2">http://download.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2</a>).</div>
<div>The dump is 27GB of XML text for 3.5M articles. I read all the pages and for each one I create a Lucene Document with three fields: the ID, the Title and the Text of each (ID and Title are also Stored Fields). I open one IndexWriter for the entire session and commit every 100 documents. I am using the following infinispan.xml:</div>
<div><br></div><div><meta http-equiv="content-type" content="text/html; charset=utf-8"><div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-style-span" style="white-space: pre; "> </span><namedCache name="cassandraCache"><span class="Apple-tab-span" style="white-space:pre"> </span></font></div>
<div><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'courier new', monospace"> </font></span><font class="Apple-style-span" face="'courier new', monospace"><eviction strategy="LIRS" maxEntries="10000" /></font></div>
<div><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'courier new', monospace"> </font></span><font class="Apple-style-span" face="'courier new', monospace"><invocationBatching enabled="true" /></font></div>
<div><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'courier new', monospace"> </font></span><font class="Apple-style-span" face="'courier new', monospace"><loaders passivation="false" shared="true" preload="false"></font></div>
<div><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'courier new', monospace"> </font></span><font class="Apple-style-span" face="'courier new', monospace"><loader</font><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'courier new', monospace"> </font></span><font class="Apple-style-span" face="'courier new', monospace">class="net.dataforte.infinispan.loaders.cassandra.CassandraCacheStore"</font></div>
<div><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'courier new', monospace"> </font></span><font class="Apple-style-span" face="'courier new', monospace">fetchPersistentState="false" ignoreModifications="false"</font></div>
<div><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'courier new', monospace"> </font></span><font class="Apple-style-span" face="'courier new', monospace">purgeOnStartup="false"></font></div>
<div><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'courier new', monospace"> </font></span><font class="Apple-style-span" face="'courier new', monospace"><properties></font></div>
<div><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'courier new', monospace"> </font></span><font class="Apple-style-span" face="'courier new', monospace"><property name="host" value="localhost" /></font></div>
<div><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'courier new', monospace"> </font></span><font class="Apple-style-span" face="'courier new', monospace"><property name="keySpace" value="Infinispan" /></font></div>
<div><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'courier new', monospace"> </font></span><font class="Apple-style-span" face="'courier new', monospace"></properties></font></div>
<div><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'courier new', monospace"> </font></span><font class="Apple-style-span" face="'courier new', monospace"><async /></font></div>
<div><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'courier new', monospace"> </font></span><font class="Apple-style-span" face="'courier new', monospace"></loader></font></div>
<div><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'courier new', monospace"> </font></span><font class="Apple-style-span" face="'courier new', monospace"></loaders></font></div>
<div><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'courier new', monospace"> </font></span><font class="Apple-style-span" face="'courier new', monospace"></namedCache></font></div>
</div><div><br></div><div>Everything goes smoothly for a while, until I get a TimeoutException when attempting to acquire the lock. This is during the IndexWriter commit() and, depending on the run, it is caused either within the InfinispanDirectory locking or within the Eviction thread. I have tried both with 4.1.0.FINAL and with 4.2.0.ALPHA2, and also replacing my CacheStore with the basic FileCacheStore but the result is the same. </div>
<div>Basically, at some point the commit() hangs indefinitely, and after a while it breaks. Here is the stack trace:</div><div><br></div><div><div>47927> Job</div><div>47928> Battle of Naissus</div><div>47929> Strange loop</div>
<div>47930> Gloucester (disambiguation)</div><div>Committing...2010-10-01 14:48:55 EvictionManagerImpl [WARN] Could not acquire lock for eviction of _56.cfs|1518|wikiIndex</div><div>2010-10-01 14:48:55 EvictionManagerImpl [WARN] Could not acquire lock for eviction of _78.cfs|6|wikiIndex</div>
<div>2010-10-01 14:48:56 CommitLogSegment [INFO] Creating new commitlog segment /tmp/infiniwiki/commitlog/CommitLog-1285937336075.log</div><div>2010-10-01 14:49:05 InvocationContextInterceptor [ERROR] Execution error: </div>
<div>org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [10 seconds] on key [_7w.tis|RL|wikiIndex] for requestor [Thread[main,5,main]]! Lock held by [GlobalTransaction:<null>:66883:local]</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.container.EntryFactoryImpl.acquireLock(EntryFactoryImpl.java:218)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:153)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:106)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:234)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.interceptors.CacheStoreInterceptor.visitPutKeyValueCommand(CacheStoreInterceptor.java:194)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.interceptors.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java:79)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:172)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:120)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:79)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:56)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:76)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:436)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.CacheSupport.putIfAbsent(CacheSupport.java:40)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.lucene.readlocks.DistributedSegmentReadLocker.aquireReadLock(DistributedSegmentReadLocker.java:130)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.lucene.InfinispanDirectory.openInput(InfinispanDirectory.java:269)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter.java:221)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java:184)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.apache.lucene.index.DocumentsWriter.createCompoundFile(DocumentsWriter.java:672)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4418)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4264)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4255)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:4133)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4206)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4179)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at net.dataforte.infiniwiki.WikiHandler.endElement(WikiHandler.java:90)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:601)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1782)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2938)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:511)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:808)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:522)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at javax.xml.parsers.SAXParser.parse(SAXParser.java:395)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at javax.xml.parsers.SAXParser.parse(SAXParser.java:198)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at net.dataforte.infiniwiki.InfiniWiki.doIndex(InfiniWiki.java:126)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>at net.dataforte.infiniwiki.InfiniWiki.main(InfiniWiki.java:152)</div></div><div><br></div><div>Tristan</div><div><br></div>