9:26 a.m.
Hi Tristan,
two quick notes:
1) Until I fix ISPN-680 you shouldn't use eviction on a cache backing
the Lucene Directory, I'll fix it this weekend, maybe today.
2) you seem to have a lot of chunks, from your log
"_56.cfs|1518|wikiIndex" this means that the _56.cfs file is
fragmented in at least 1518 pieces, likely more.
Please set a quite-higher chunk value using the alternative constructor
InfinispanDirectory(Cache cache, String indexName, int chunkSize)
The default for chunkSize is quite small (16K) to optimize network I/O
during distribution/replication, but with such large indexes you
likely want higher chunks.
For Scarlet I'm defaulting to 7MB - be aware that you might need to
tune network/IO, in my case I had to rise the MySQL allowed max packet
size too, no idea if you have to configure something on Cassandra.
About the TimeoutException, I don't know, I hope someone from
Infinispan core could provide some hints;
if you can share this test I'll be glad to look into it, assuming it's
not complex to setup Cassandra, or a patch for an automated unit test
would be better.
Cheers,
Sanne
2010/10/1 Tristan Tarrant <tristan.tarrant(a)gmail.com>:
Dear all,
I have written a small application for stress testing my CassandraCacheStore
in conjunction with the InfinispanDirectory.
Purpose of the test, inspired by a blog I've seen on Lucene performance, is
to index a dump of English Wikipedia
(see http://download.wikimedia.org/enwiki/latest/enwiki-latest-pages-artic...).
The dump is 27GB of XML text for 3.5M articles. I read all the pages and for
each one I create a Lucene Document with three fields: the ID, the Title and
the Text of each (ID and Title are also Stored Fields). I open one
IndexWriter for the entire session and commit every 100 documents. I am
using the following infinispan.xml:
<namedCache name="cassandraCache">
<eviction strategy="LIRS" maxEntries="10000" />
<invocationBatching enabled="true" />
<loaders passivation="false" shared="true"
preload="false">
<loader
class="net.dataforte.infinispan.loaders.cassandra.CassandraCacheStore"
fetchPersistentState="false" ignoreModifications="false"
purgeOnStartup="false">
<properties>
<property name="host" value="localhost" />
<property name="keySpace" value="Infinispan" />
</properties>
<async />
</loader>
</loaders>
</namedCache>
Everything goes smoothly for a while, until I get a TimeoutException when
attempting to acquire the lock. This is during the IndexWriter commit() and,
depending on the run, it is caused either within the InfinispanDirectory
locking or within the Eviction thread. I have tried both with 4.1.0.FINAL
and with 4.2.0.ALPHA2, and also replacing my CacheStore with the basic
FileCacheStore but the result is the same.
Basically, at some point the commit() hangs indefinitely, and after a while
it breaks. Here is the stack trace:
47927> Job
47928> Battle of Naissus
47929> Strange loop
47930> Gloucester (disambiguation)
Committing...2010-10-01 14:48:55 EvictionManagerImpl [WARN] Could not
acquire lock for eviction of _56.cfs|1518|wikiIndex
2010-10-01 14:48:55 EvictionManagerImpl [WARN] Could not acquire lock for
eviction of _78.cfs|6|wikiIndex
2010-10-01 14:48:56 CommitLogSegment [INFO] Creating new commitlog segment
/tmp/infiniwiki/commitlog/CommitLog-1285937336075.log
2010-10-01 14:49:05 InvocationContextInterceptor [ERROR] Execution error:
org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock
after [10 seconds] on key [_7w.tis|RL|wikiIndex] for requestor
[Thread[main,5,main]]! Lock held by [GlobalTransaction:<null>:66883:local]
at
org.infinispan.container.EntryFactoryImpl.acquireLock(EntryFactoryImpl.java:218)
at
org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:153)
at
org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:106)
at
org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:234)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.CacheStoreInterceptor.visitPutKeyValueCommand(CacheStoreInterceptor.java:194)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java:79)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:172)
at
org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:120)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:79)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:56)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:76)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273)
at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:436)
at org.infinispan.CacheSupport.putIfAbsent(CacheSupport.java:40)
at
org.infinispan.lucene.readlocks.DistributedSegmentReadLocker.aquireReadLock(DistributedSegmentReadLocker.java:130)
at
org.infinispan.lucene.InfinispanDirectory.openInput(InfinispanDirectory.java:269)
at
org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter.java:221)
at
org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java:184)
at
org.apache.lucene.index.DocumentsWriter.createCompoundFile(DocumentsWriter.java:672)
at
org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4418)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4264)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4255)
at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:4133)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4206)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4179)
at net.dataforte.infiniwiki.WikiHandler.endElement(WikiHandler.java:90)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:601)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1782)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2938)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:511)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:808)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
at
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205)
at
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:522)
at javax.xml.parsers.SAXParser.parse(SAXParser.java:395)
at javax.xml.parsers.SAXParser.parse(SAXParser.java:198)
at net.dataforte.infiniwiki.InfiniWiki.doIndex(InfiniWiki.java:126)
at net.dataforte.infiniwiki.InfiniWiki.main(InfiniWiki.java:152)
Tristan
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev