Infinipedia: a stress test for InfinispanDirectory and CassandraCacheStore
by Tristan Tarrant
Dear all,
I have written a small application for stress testing my CassandraCacheStore
in conjunction with the InfinispanDirectory.
Purpose of the test, inspired by a blog I've seen on Lucene performance, is
to index a dump of English Wikipedia (see
http://download.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles....
).
The dump is 27GB of XML text for 3.5M articles. I read all the pages and for
each one I create a Lucene Document with three fields: the ID, the Title and
the Text of each (ID and Title are also Stored Fields). I open one
IndexWriter for the entire session and commit every 100 documents. I am
using the following infinispan.xml:
<namedCache name="cassandraCache">
<eviction strategy="LIRS" maxEntries="10000" />
<invocationBatching enabled="true" />
<loaders passivation="false" shared="true" preload="false">
<loader
class="net.dataforte.infinispan.loaders.cassandra.CassandraCacheStore"
fetchPersistentState="false" ignoreModifications="false"
purgeOnStartup="false">
<properties>
<property name="host" value="localhost" />
<property name="keySpace" value="Infinispan" />
</properties>
<async />
</loader>
</loaders>
</namedCache>
Everything goes smoothly for a while, until I get a TimeoutException when
attempting to acquire the lock. This is during the IndexWriter commit() and,
depending on the run, it is caused either within the InfinispanDirectory
locking or within the Eviction thread. I have tried both with 4.1.0.FINAL
and with 4.2.0.ALPHA2, and also replacing my CacheStore with the basic
FileCacheStore but the result is the same.
Basically, at some point the commit() hangs indefinitely, and after a while
it breaks. Here is the stack trace:
47927> Job
47928> Battle of Naissus
47929> Strange loop
47930> Gloucester (disambiguation)
Committing...2010-10-01 14:48:55 EvictionManagerImpl [WARN] Could not
acquire lock for eviction of _56.cfs|1518|wikiIndex
2010-10-01 14:48:55 EvictionManagerImpl [WARN] Could not acquire lock for
eviction of _78.cfs|6|wikiIndex
2010-10-01 14:48:56 CommitLogSegment [INFO] Creating new commitlog segment
/tmp/infiniwiki/commitlog/CommitLog-1285937336075.log
2010-10-01 14:49:05 InvocationContextInterceptor [ERROR] Execution error:
org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock
after [10 seconds] on key [_7w.tis|RL|wikiIndex] for requestor
[Thread[main,5,main]]! Lock held by [GlobalTransaction:<null>:66883:local]
at
org.infinispan.container.EntryFactoryImpl.acquireLock(EntryFactoryImpl.java:218)
at
org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:153)
at
org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:106)
at
org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:234)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.CacheStoreInterceptor.visitPutKeyValueCommand(CacheStoreInterceptor.java:194)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java:79)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:172)
at
org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:120)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:79)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:56)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:76)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273)
at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:436)
at org.infinispan.CacheSupport.putIfAbsent(CacheSupport.java:40)
at
org.infinispan.lucene.readlocks.DistributedSegmentReadLocker.aquireReadLock(DistributedSegmentReadLocker.java:130)
at
org.infinispan.lucene.InfinispanDirectory.openInput(InfinispanDirectory.java:269)
at
org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter.java:221)
at
org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java:184)
at
org.apache.lucene.index.DocumentsWriter.createCompoundFile(DocumentsWriter.java:672)
at
org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4418)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4264)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4255)
at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:4133)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4206)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4179)
at net.dataforte.infiniwiki.WikiHandler.endElement(WikiHandler.java:90)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:601)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1782)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2938)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:511)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:808)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
at
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205)
at
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:522)
at javax.xml.parsers.SAXParser.parse(SAXParser.java:395)
at javax.xml.parsers.SAXParser.parse(SAXParser.java:198)
at net.dataforte.infiniwiki.InfiniWiki.doIndex(InfiniWiki.java:126)
at net.dataforte.infiniwiki.InfiniWiki.main(InfiniWiki.java:152)
Tristan
13 years, 6 months
Native CacheStore implementation?
by "이희승 (Trustin Lee)"
Hi,
Currently, we have two local CacheStore implementations - BDBJE and JDBM.
If someone distributes an application that uses BDBJE, that application
must be under an open source license. It's not a problem to Infinispan,
but it is to our users who want to distribute Infinispan with their
applications and want to use BDBJE as a backend.
JDBM has one of the most permissive license, but the recent
investigation revealed a significant performance issue:
https://jira.jboss.org/browse/ISPN-663
Moreover, JDBM hasn't been maintained actively for a long time.
If we write our own native CacheStore implementation that is optimized
for our CacheStore API, we will be able to present the whole new
performance numbers to wider audience.
WDYT?
--
http://gleamynode.net/
13 years, 6 months
NotifyingFuture vs Deferred
by "이희승 (Trustin Lee)"
Deferred is not different from NotifyingFuture fundamentally. The only
difference is how it handles an exceptional case and callbacks can
return another Deffered, which makes chained asynchronous operations
possible.
This difference opens a lot of possibilities for performing a series of
asynchronous operations. It can be compared to dynamic construction of
interceptor chains.
So, I'd like to propose to replacing the current NotifyingFuture
implementation with Deferred. I think the name 'Deferred' is somewhat
less intuitive, so retaining the name might be a good idea.
If you are not familiar with Deferreds, check this out:
http://twistedmatrix.com/documents/current/core/howto/deferredindepth.html
--
Trustin Lee - http://gleamynode.net/
13 years, 6 months
Clustering without mode
by Galder Zamarreño
The following named cache configuration:
<namedCache name="withReplicatinQueue">
<clustering>
<async useReplQueue="true" replQueueInterval="100" replQueueMaxElements="200"/>
</clustering>
</namedCache>
Is considered to be a local cache. Should that really be the case? This is taken from all.xml.
The result is that replication queue obviously won't be used.
Cheers,
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
13 years, 6 months
Key set, value set and size within transaction
by Manik Surtani
Guys
I just noticed this in the codebase:
http://fisheye.jboss.org/browse/Infinispan/branches/4.2.x/core/src/main/j...
This seems to have come in as a part of changeset 2518 - http://fisheye.jboss.org/changelog/Infinispan/?cs=2518 - for ISPN-679 (Cache.values does not work correctly).
Please be careful when doing stuff like this - the new method added to AbstractLocalCommand effectively creates a new collection the size of the entire data container and loads it with (almost) all keys. An Infinispan node running close to capacity will almost certainly die with an OOM if you do this. It will even OOM if you are not necessarily close to capacity (under half your heap used) and you have > 1 thread concurrently invoking this method.
I presume this is to make Cache.keySet(), Cache.values() and Cache.size() transactionally accurate, but remember that this comes at a huge cost. Which is why we chose to go with a best-effort approach. See Javadocs on Cache [1]. The relevant bit states:
"Other methods such as Map.size() provide an approximation-only, and should not be relied on for an accurate picture as to the size of the entire, distributed cache. Remote nodes are not queried and in-fly transactions are not taken into account, even if Map.size() is invoked from within such a transaction."
Could we please discuss why we have added this?
Cheers
Manik
[1] http://docs.jboss.org/infinispan/4.2/apidocs/org/infinispan/Cache.html
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
13 years, 6 months
IntelliJ OSGI facet is rather annoying
by Galder Zamarreño
Hi,
For those using IntelliJ, are you also encountering issues with building since OSGI was introduced? It seems like with each build, it attempts to build OSGI bundles.
I've tried to disable OSGI facet for all modules, but whenever I have to reimport maven projects, it creates the facets and I have the same issue again.
I've looked around and I haven't seen a way to fully disable the OSGI facet, any ideas anyone?
Cheers,
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
13 years, 6 months
Configuration reference generation - rework
by Vladimir Blagojevic
Hey all,
Manik and I talked privately regarding this topic through https://jira.jboss.org/browse/ISPN-626 and we concluded that we could use some input from people that have used configuration reference markers within Infinispan code etc. Manik basically had a few ideas where he wanted to polish/expand the final look of configuration reference (summed up in ISPN-626) and upon closer inspection I think that we need to rework configuration markers to make things simpler and easier to use.
One of the things I wanted to do is replace javadoc tag @configRef with annotation(s) and use pure comment text of javadoc elements as input for both javadoc and configuration reference. So lets gather some more ideas, and I'll work on this on a side whenever I need the break from hard core stuff :)
Cheers,
Vladimir
--
Vladimir Blagojevic
JBoss Clustering Team
JBoss, by Red Hat
13 years, 6 months
ISPN-699 - proper cancellation of cache store operations
by Galder Zamarreño
Hi,
Re: https://jira.jboss.org/browse/ISPN-699
I'm trying to figure out what the best way to solve this issue is. Basically, the problem is that when cache manager is stopped, EvictionManagerImpl cancels with interruption the evictionTask and I'm seeing issues with cacheStore.purgeExpired() not responding to cancellation properly. This results in Marshaller being stopped and then eviction thread trying to purge the cache store. Obviously, once the marshaller is stopped, nothing can be read any more.
I've tried to simply protect cacheStore.purgeExpired() call around a Thread.currentThread().isInterrupted() call but this is not enough because we could have hundreds of buckets to check for purging, and the interruption could happen while looping through them. Now, I don't see the point of plaguing the code with Thread.currentThread().isInterrupted() checks, it'd be pointless. Instead, I wanted to share other ideas to solve this issue:
1. EvictionManagerImpl could wait for any ongoing eviction task to finished. This could be potentially lengthy if for example the cache store has hundreds or thousands of buckets, and we don't want for stop requests to block at all.
2. The main problem comes from the fact that the marshaller is being requested to read something but it can't do it anymore since it's shutting down. An alternative would be for the ConstantObjectTable to return null under the situation that is stopped and the thread is interrupted. This might work fine since if the bucket read returns null, it skips the bucket, but it's not optimal. If you have 1000s of buckets, it is going to continue looping through them, so you'd need something else, and this logic would need to be replicated in all cache stores.
3. COT returning null might not be a good idea. Instead, ObjectTable readObject() method could be changed to declare that it can throw an InterruptedException. This would force any caller to deal with this situation, including all current cache stores. I think this is the only way we can enforce cache stores to behave properly to interruption situations like the one mentioned above. Otherwise, we have to start doing esoteric things in all cache stores to check what a null return from unmarshalling means, or try to double guess that an IOException might be wrapping an IE.
I'm currently leaning towards 3.
Thoughts?
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
13 years, 6 months
