[infinispan-dev] Cachestores performance

Radim Vansa rvansa at redhat.com
Thu Jun 27 05:33:53 EDT 2013 

Oops, by the cache store I mean the previously-superfast KarstenFileCacheStore implementation.

----- Original Message -----
| From: "Radim Vansa" <rvansa at redhat.com>
| To: "infinispan -Dev List" <infinispan-dev at lists.jboss.org>
| Sent: Thursday, June 27, 2013 11:30:53 AM
| Subject: Re: [infinispan-dev] Cachestores performance
| 
| I have added FileChannel.force(false) flushes after all write operations in
| the cache store, and now the comparison is also updated with these values.
| 
| Radim
| 
| ----- Original Message -----
| | From: "Radim Vansa" <rvansa at redhat.com>
| | To: "infinispan -Dev List" <infinispan-dev at lists.jboss.org>
| | Sent: Thursday, June 27, 2013 8:54:25 AM
| | Subject: Re: [infinispan-dev] Cachestores performance
| | 
| | Yep, write-through. LevelDB JAVA used FileChannelTable implementation
| | (-Dleveldb.mmap), because Mmaping is not implemented very well and causes
| | JVM crashes (I believe it's because of calling non-public API via
| | reflection
| | - I've found post from the Oracle JVM guys discouraging the particular
| | trick
| | it uses). After writing the record to the log, it calls
| | FileChannel.force(true), therefore, it should be really on the disc by that
| | moment.
| | I have not looked into the JNI implementation but I expect the same.
| | 
| | By the way, I have updated [1] with numbers when running on more data (2 GB
| | instead of 100 MB). I won't retype it here, so look there. The performance
| | is much lower.
| | I may try also increase JVM heap size and try with a bit more data yet.
| | 
| | Radim
| | 
| | [1] https://community.jboss.org/wiki/FileCacheStoreRedesign
| | 
| | ----- Original Message -----
| | | From: "Erik Salter" <an1310 at hotmail.com>
| | | To: "infinispan -Dev List" <infinispan-dev at lists.jboss.org>
| | | Sent: Wednesday, June 26, 2013 7:40:19 PM
| | | Subject: Re: [infinispan-dev] Cachestores performance
| | | 
| | | These were write-through cache stores, right?  And with LevelDB, this was
| | | through to the database file itself?
| | | 
| | | Erik
| | | 
| | | -----Original Message-----
| | | From: infinispan-dev-bounces at lists.jboss.org
| | | [mailto:infinispan-dev-bounces at lists.jboss.org] On Behalf Of Radim Vansa
| | | Sent: Wednesday, June 26, 2013 11:24 AM
| | | To: infinispan -Dev List
| | | Subject: [infinispan-dev] Cachestores performance
| | | 
| | | Hi all,
| | | 
| | | according to [1] I've created the comparison of performance in
| | | stress-tests.
| | | 
| | | All setups used local-cache, benchmark was executed via Radargun
| | | (actually
| | | version not merged into master yet [2]). I've used 4 nodes just to get
| | | more
| | | data - each slave was absolutely independent of the others.
| | | 
| | | First test was preloading performance - the cache started and tried to
| | | load
| | | 1GB of data from harddrive. Without cachestore the startup takes about 2
| | | -
| | | 4
| | | seconds, average numbers for the cachestores are below:
| | | 
| | | FileCacheStore:        9.8 s
| | | KarstenFileCacheStore:  14 s
| | | LevelDB-JAVA impl.:   12.3 s
| | | LevelDB-JNI impl.:    12.9 s
| | | 
| | | IMO nothing special, all times seem affordable. We don't benchmark
| | | exactly
| | | storing the data into the cachestore, here FileCacheStore took about 44
| | | minutes, while Karsten about 38 seconds, LevelDB-JAVA 4 minutes and
| | | LevelDB-JNI 96 seconds. The units are right, it's minutes compared to
| | | seconds. But we all know that FileCacheStore is bloody slow.
| | | 
| | | Second test is stress test (5 minutes, preceded by 2 minute warmup) where
| | | each of 10 threads works on 10k entries with 1kB values (~100 MB in
| | | total).
| | | 20 % writes, 80 % reads, as usual. No eviction is configured, therefore
| | | the
| | | cache-store works as a persistent storage only for case of crash.
| | | 
| | | FileCacheStore:         3.1M reads/s   112 writes/s  // on one node the
| | | performance was only 2.96M reads/s 75 writes/s
| | | KarstenFileCacheStore:  9.2M reads/s  226k writes/s  // yikes!
| | | LevelDB-JAVA impl.:     3.9M reads/s  5100 writes/s
| | | LevelDB-JNI impl.:      6.6M reads/s   14k writes/s  // on one node the
| | | performance was 3.9M/8.3k - about half of the others
| | | Without cache store:   15.5M reads/s  4.4M writes/s
| | | 
| | | Karsten implementation pretty rules here for two reasons. First of all,
| | | it
| | | does not flush the data (it calls only RandomAccessFile.write()). Other
| | | cheat is that it stores in-memory the keys and offsets of data values in
| | | the
| | | database file. Therefore, it's definitely the best choice for this
| | | scenario,
| | | but it does not allow to scale the cache-store, especially in cases where
| | | the keys are big and values small. However, this performance boost is
| | | definitely worth checking - I could think of caching the disk offsets in
| | | memory and querying persistent index only in case of missing record, with
| | | part of the persistent index flushed asynchronously (the index can be
| | | always
| | | rebuilt during the preloading for case of crash).
| | | 
| | | The third test should have tested the scenario with more data to be
| | | stored
| | | than memory - therefore, the stressors operated on 100k entries (~100 MB
| | | of
| | | data) but eviction was set to 10k entries (9216 entries ended up in
| | | memory
| | | after the test has ended).
| | | 
| | | FileCacheStore:            750 reads/s         285 writes/s  // one node
| | | had
| | | only 524 reads and 213 writes per second
| | | KarstenFileCacheStore:    458k reads/s        137k writes/s
| | | LevelDB-JAVA impl.:        21k reads/s          9k writes/s  // a bit
| | | varying
| | | performance
| | | LevelDB-JNI impl.:     13k-46k reads/s  6.6k-15.2k writes/s  // the
| | | performance varied a lot!
| | | 
| | | 100 MB of data is not much, but it takes so long to push it into
| | | FileCacheStore that I won't use more unless we exclude this loser from
| | | the
| | | comparison :)
| | | 
| | | Radim
| | | 
| | | [1] https://community.jboss.org/wiki/FileCacheStoreRedesign
| | | [2] https://github.com/rvansa/radargun/tree/t_keygen
| | | 
| | | -----------------------------------------------------------
| | | Radim Vansa
| | | Quality Assurance Engineer
| | | JBoss Datagrid
| | | tel. +420532294559 ext. 62559
| | | 
| | | Red Hat Czech, s.r.o.
| | | Brno, Purkyňova 99/71, PSČ 612 45
| | | Czech Republic
| | | 
| | | 
| | | _______________________________________________
| | | infinispan-dev mailing list
| | | infinispan-dev at lists.jboss.org
| | | https://lists.jboss.org/mailman/listinfo/infinispan-dev
| | | 
| | | 
| | | _______________________________________________
| | | infinispan-dev mailing list
| | | infinispan-dev at lists.jboss.org
| | | https://lists.jboss.org/mailman/listinfo/infinispan-dev
| | 
| | _______________________________________________
| | infinispan-dev mailing list
| | infinispan-dev at lists.jboss.org
| | https://lists.jboss.org/mailman/listinfo/infinispan-dev
| 
| _______________________________________________
| infinispan-dev mailing list
| infinispan-dev at lists.jboss.org
| https://lists.jboss.org/mailman/listinfo/infinispan-dev

More information about the infinispan-dev mailing list