Yep, write-through. LevelDB JAVA used FileChannelTable implementation (-Dleveldb.mmap), because Mmaping is not implemented very well and causes JVM crashes (I believe it's because of calling non-public API via reflection - I've found post from the Oracle JVM guys discouraging the particular trick it uses). After writing the record to the log, it calls FileChannel.force(true), therefore, it should be really on the disc by that moment. I have not looked into the JNI implementation but I expect the same. By the way, I have updated [1] with numbers when running on more data (2 GB instead of 100 MB). I won't retype it here, so look there. The performance is much lower. I may try also increase JVM heap size and try with a bit more data yet. Radim [1] https://community.jboss.org/wiki/FileCacheStoreRedesign ----- Original Message ----- | From: "Erik Salter" <an1310(a)hotmail.com&gt; | To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; | Sent: Wednesday, June 26, 2013 7:40:19 PM | Subject: Re: [infinispan-dev] Cachestores performance | | These were write-through cache stores, right? And with LevelDB, this was | through to the database file itself? | | Erik | | -----Original Message----- | From: infinispan-dev-bounces(a)lists.jboss.org | [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Radim Vansa | Sent: Wednesday, June 26, 2013 11:24 AM | To: infinispan -Dev List | Subject: [infinispan-dev] Cachestores performance | | Hi all, | | according to [1] I've created the comparison of performance in stress-tests. | | All setups used local-cache, benchmark was executed via Radargun (actually | version not merged into master yet [2]). I've used 4 nodes just to get more | data - each slave was absolutely independent of the others. | | First test was preloading performance - the cache started and tried to load | 1GB of data from harddrive. Without cachestore the startup takes about 2 - 4 | seconds, average numbers for the cachestores are below: | | FileCacheStore: 9.8 s | KarstenFileCacheStore: 14 s | LevelDB-JAVA impl.: 12.3 s | LevelDB-JNI impl.: 12.9 s | | IMO nothing special, all times seem affordable. We don't benchmark exactly | storing the data into the cachestore, here FileCacheStore took about 44 | minutes, while Karsten about 38 seconds, LevelDB-JAVA 4 minutes and | LevelDB-JNI 96 seconds. The units are right, it's minutes compared to | seconds. But we all know that FileCacheStore is bloody slow. | | Second test is stress test (5 minutes, preceded by 2 minute warmup) where | each of 10 threads works on 10k entries with 1kB values (~100 MB in total). | 20 % writes, 80 % reads, as usual. No eviction is configured, therefore the | cache-store works as a persistent storage only for case of crash. | | FileCacheStore: 3.1M reads/s 112 writes/s // on one node the | performance was only 2.96M reads/s 75 writes/s | KarstenFileCacheStore: 9.2M reads/s 226k writes/s // yikes! | LevelDB-JAVA impl.: 3.9M reads/s 5100 writes/s | LevelDB-JNI impl.: 6.6M reads/s 14k writes/s // on one node the | performance was 3.9M/8.3k - about half of the others | Without cache store: 15.5M reads/s 4.4M writes/s | | Karsten implementation pretty rules here for two reasons. First of all, it | does not flush the data (it calls only RandomAccessFile.write()). Other | cheat is that it stores in-memory the keys and offsets of data values in the | database file. Therefore, it's definitely the best choice for this scenario, | but it does not allow to scale the cache-store, especially in cases where | the keys are big and values small. However, this performance boost is | definitely worth checking - I could think of caching the disk offsets in | memory and querying persistent index only in case of missing record, with | part of the persistent index flushed asynchronously (the index can be always | rebuilt during the preloading for case of crash). | | The third test should have tested the scenario with more data to be stored | than memory - therefore, the stressors operated on 100k entries (~100 MB of | data) but eviction was set to 10k entries (9216 entries ended up in memory | after the test has ended). | | FileCacheStore: 750 reads/s 285 writes/s // one node had | only 524 reads and 213 writes per second | KarstenFileCacheStore: 458k reads/s 137k writes/s | LevelDB-JAVA impl.: 21k reads/s 9k writes/s // a bit varying | performance | LevelDB-JNI impl.: 13k-46k reads/s 6.6k-15.2k writes/s // the | performance varied a lot! | | 100 MB of data is not much, but it takes so long to push it into | FileCacheStore that I won't use more unless we exclude this loser from the | comparison :) | | Radim | | [1] https://community.jboss.org/wiki/FileCacheStoreRedesign | [2] https://github.com/rvansa/radargun/tree/t_keygen | | ----------------------------------------------------------- | Radim Vansa | Quality Assurance Engineer | JBoss Datagrid | tel. +420532294559 ext. 62559 | | Red Hat Czech, s.r.o. | Brno, Purkyňova 99/71, PSČ 612 45 | Czech Republic | | | _______________________________________________ | infinispan-dev mailing list | infinispan-dev(a)lists.jboss.org | https://lists.jboss.org/mailman/listinfo/infinispan-dev | | | _______________________________________________ | infinispan-dev mailing list | infinispan-dev(a)lists.jboss.org | https://lists.jboss.org/mailman/listinfo/infinispan-dev

Oops, by the cache store I mean the previously-superfast KarstenFileCacheStore implementation. ----- Original Message ----- | From: "Radim Vansa" <rvansa(a)redhat.com&gt; | To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; | Sent: Thursday, June 27, 2013 11:30:53 AM | Subject: Re: [infinispan-dev] Cachestores performance | | I have added FileChannel.force(false) flushes after all write operations in | the cache store, and now the comparison is also updated with these values. | | Radim | | ----- Original Message ----- | | From: "Radim Vansa" <rvansa(a)redhat.com&gt; | | To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; | | Sent: Thursday, June 27, 2013 8:54:25 AM | | Subject: Re: [infinispan-dev] Cachestores performance | | | | Yep, write-through. LevelDB JAVA used FileChannelTable implementation | | (-Dleveldb.mmap), because Mmaping is not implemented very well and causes | | JVM crashes (I believe it's because of calling non-public API via | | reflection | | - I've found post from the Oracle JVM guys discouraging the particular | | trick | | it uses). After writing the record to the log, it calls | | FileChannel.force(true), therefore, it should be really on the disc by that | | moment. | | I have not looked into the JNI implementation but I expect the same. | | | | By the way, I have updated [1] with numbers when running on more data (2 GB | | instead of 100 MB). I won't retype it here, so look there. The performance | | is much lower. | | I may try also increase JVM heap size and try with a bit more data yet. | | | | Radim | | | | [1] https://community.jboss.org/wiki/FileCacheStoreRedesign | | | | ----- Original Message ----- | | | From: "Erik Salter" <an1310(a)hotmail.com&gt; | | | To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; | | | Sent: Wednesday, June 26, 2013 7:40:19 PM | | | Subject: Re: [infinispan-dev] Cachestores performance | | | | | | These were write-through cache stores, right? And with LevelDB, this was | | | through to the database file itself? | | | | | | Erik | | | | | | -----Original Message----- | | | From: infinispan-dev-bounces(a)lists.jboss.org | | | [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Radim Vansa | | | Sent: Wednesday, June 26, 2013 11:24 AM | | | To: infinispan -Dev List | | | Subject: [infinispan-dev] Cachestores performance | | | | | | Hi all, | | | | | | according to [1] I've created the comparison of performance in | | | stress-tests. | | | | | | All setups used local-cache, benchmark was executed via Radargun | | | (actually | | | version not merged into master yet [2]). I've used 4 nodes just to get | | | more | | | data - each slave was absolutely independent of the others. | | | | | | First test was preloading performance - the cache started and tried to | | | load | | | 1GB of data from harddrive. Without cachestore the startup takes about 2 | | | - | | | 4 | | | seconds, average numbers for the cachestores are below: | | | | | | FileCacheStore: 9.8 s | | | KarstenFileCacheStore: 14 s | | | LevelDB-JAVA impl.: 12.3 s | | | LevelDB-JNI impl.: 12.9 s | | | | | | IMO nothing special, all times seem affordable. We don't benchmark | | | exactly | | | storing the data into the cachestore, here FileCacheStore took about 44 | | | minutes, while Karsten about 38 seconds, LevelDB-JAVA 4 minutes and | | | LevelDB-JNI 96 seconds. The units are right, it's minutes compared to | | | seconds. But we all know that FileCacheStore is bloody slow. | | | | | | Second test is stress test (5 minutes, preceded by 2 minute warmup) where | | | each of 10 threads works on 10k entries with 1kB values (~100 MB in | | | total). | | | 20 % writes, 80 % reads, as usual. No eviction is configured, therefore | | | the | | | cache-store works as a persistent storage only for case of crash. | | | | | | FileCacheStore: 3.1M reads/s 112 writes/s // on one node the | | | performance was only 2.96M reads/s 75 writes/s | | | KarstenFileCacheStore: 9.2M reads/s 226k writes/s // yikes! | | | LevelDB-JAVA impl.: 3.9M reads/s 5100 writes/s | | | LevelDB-JNI impl.: 6.6M reads/s 14k writes/s // on one node the | | | performance was 3.9M/8.3k - about half of the others | | | Without cache store: 15.5M reads/s 4.4M writes/s | | | | | | Karsten implementation pretty rules here for two reasons. First of all, | | | it | | | does not flush the data (it calls only RandomAccessFile.write()). Other | | | cheat is that it stores in-memory the keys and offsets of data values in | | | the | | | database file. Therefore, it's definitely the best choice for this | | | scenario, | | | but it does not allow to scale the cache-store, especially in cases where | | | the keys are big and values small. However, this performance boost is | | | definitely worth checking - I could think of caching the disk offsets in | | | memory and querying persistent index only in case of missing record, with | | | part of the persistent index flushed asynchronously (the index can be | | | always | | | rebuilt during the preloading for case of crash). | | | | | | The third test should have tested the scenario with more data to be | | | stored | | | than memory - therefore, the stressors operated on 100k entries (~100 MB | | | of | | | data) but eviction was set to 10k entries (9216 entries ended up in | | | memory | | | after the test has ended). | | | | | | FileCacheStore: 750 reads/s 285 writes/s // one node | | | had | | | only 524 reads and 213 writes per second | | | KarstenFileCacheStore: 458k reads/s 137k writes/s | | | LevelDB-JAVA impl.: 21k reads/s 9k writes/s // a bit | | | varying | | | performance | | | LevelDB-JNI impl.: 13k-46k reads/s 6.6k-15.2k writes/s // the | | | performance varied a lot! | | | | | | 100 MB of data is not much, but it takes so long to push it into | | | FileCacheStore that I won't use more unless we exclude this loser from | | | the | | | comparison :) | | | | | | Radim | | | | | | [1] https://community.jboss.org/wiki/FileCacheStoreRedesign | | | [2] https://github.com/rvansa/radargun/tree/t_keygen | | | | | | ----------------------------------------------------------- | | | Radim Vansa | | | Quality Assurance Engineer | | | JBoss Datagrid | | | tel. +420532294559 ext. 62559 | | | | | | Red Hat Czech, s.r.o. | | | Brno, Purkyňova 99/71, PSČ 612 45 | | | Czech Republic | | | | | | | | | _______________________________________________ | | | infinispan-dev mailing list | | | infinispan-dev(a)lists.jboss.org | | | https://lists.jboss.org/mailman/listinfo/infinispan-dev | | | | | | | | | _______________________________________________ | | | infinispan-dev mailing list | | | infinispan-dev(a)lists.jboss.org | | | https://lists.jboss.org/mailman/listinfo/infinispan-dev | | | | _______________________________________________ | | infinispan-dev mailing list | | infinispan-dev(a)lists.jboss.org | | https://lists.jboss.org/mailman/listinfo/infinispan-dev | | _______________________________________________ | infinispan-dev mailing list | infinispan-dev(a)lists.jboss.org | https://lists.jboss.org/mailman/listinfo/infinispan-dev

On Jun 27, 2013, at 1:25 PM, Manik Surtani <msurtani(a)redhat.com&gt; wrote: Yeah, we're collaborating. We came up with the test plan and cache stores to test together :). ^ Karsten's implementation makes this relatively easy to achieve because it already keeps this mapping in a LinkedHashMap (with a given max entries limit [1]) assuming removeEldestEntry() is overriden to flush to disk older entries. Some extra logic would be needed to bring back data from the disk too… but your suggestion below is also quite interesting... ^ In-memory, the structure can optionally be ordered if it's bound [1], otherwise it's just a normal map. How would be store it at the disk level? B+ tree with hashes of keys and then linked lists? ^ Hmmm, interesting idea… has the potential to safe the memory space by not having to keep that extra data structure in the cache store. My worry about Karsten's impl is writing actually. If you look at the last performance numbers in [2], where we see the performance difference of force=true and force=false in Karsten's cache store compared with LevelDB JNI, you see that force=false is fastest, then JNI LevelDB, and the force=true. Me wonders what kind of write guarantees LevelDB JNI provides (and the JAVA version)... [1] http://goo.gl/rPYp2 -- Galder Zamarreño galder(a)redhat.com twitter.com/galderz Project Lead, Escalante http://escalante.io Engineer, Infinispan http://infinispan.org

On Jun 27, 2013, at 1:58 PM, Galder Zamarreño <galder(a)redhat.com&gt; wrote: Actually Radim, maybe I misunderstood your comments earlier, but what is this mmap/FileChannelTable stuff. Here's what I'm seeing: LevelDBCacheStore implementation calls DB.put without any options [1]. Underneath, iq80's DbImpl calls the other put with a newly constructed WriteOptions instance, whose sync parameter is set to false. The default in Fuse's JNI implementation seems to be sync=false too [2]. So, it might be interesting to re-run the test after changing line in [1] to pass a new instance of org.iq80.leveldb.WriteOptions instance with sync=true… WDYT? Btw, at least in the LevelDB Java implementation, FileChannelLogWriter is the one that is using this sync parameter, and not FileChannelTable. I'm not sure about the relationship of these two yet, but from I see in the code, DbImpl uses FileChannelTable as a tableCache (what the heck is that? no docu) and that doesn't seem to be touched when an entry is put. What seems to be touched when putting an entry is the log, and the memTable, which is a in-memoyr cache of keys to slices, and the log is where you write the slice. Cheers, [1] http://goo.gl/w6hMw [2] http://goo.gl/h2OxG -- Galder Zamarreño galder(a)redhat.com twitter.com/galderz Project Lead, Escalante http://escalante.io Engineer, Infinispan http://infinispan.org

| >>> | >>> My worry about Karsten's impl is writing actually. If you look at the | >>> last performance numbers in [2], where we see the performance difference | >>> of force=true and force=false in Karsten's cache store compared with | >>> LevelDB JNI, you see that force=false is fastest, then JNI LevelDB, and | >>> the force=true. Me wonders what kind of write guarantees LevelDB JNI | >>> provides (and the JAVA version)… | >> | >> ^ Oh, Radim mentioned this topic already in a previous post. LevelDB JAVA | >> library seems to provide force=true equivalent logic. | > | > Actually Radim, maybe I misunderstood your comments earlier, but what is | > this mmap/FileChannelTable stuff. Here's what I'm seeing: Ok, the leveldb.mmap flag was about using (FileChannelTable + FileChannelLogWriter) OR (MMapTable + MMapLogWriter). And we stick to the FileChannel* stuff. | > | > LevelDBCacheStore implementation calls DB.put without any options [1]. | > Underneath, iq80's DbImpl calls the other put with a newly constructed | > WriteOptions instance, whose sync parameter is set to false. The default | > in Fuse's JNI implementation seems to be sync=false too [2]. Sorry, I've really mixed things here, FileChannelLogWriter really uses force(false) only if it is explicitely requested, therefore, in my tests it does not. | > | > So, it might be interesting to re-run the test after changing line in [1] | > to pass a new instance of org.iq80.leveldb.WriteOptions instance with | > sync=true… WDYT? | > | > Btw, at least in the LevelDB Java implementation, FileChannelLogWriter is | > the one that is using this sync parameter, and not | > FileChannelTable. I'm not sure about the relationship of these two yet, but | > from I see in the code, DbImpl uses FileChannelTable as a tableCache (what | > the heck is that? no docu) and that doesn't seem to be touched when an | > entry is put. | > | > What seems to be touched when putting an entry is the log, and the | > memTable, which is a in-memoyr cache of keys to slices, and the log is | > where you write the slice. | | ^ If my suspicions are correct, then none of the LevelDB implementations | (neither Java nor JNI) are forcing writes, in which case Karsten's cache | store could be a winner even in the write area (really??). Just looked at | Google's WriteOptions struct, and sync default value is false too [1]. | | So, we definitely need to construct an instance of WriteOptions and pass true | in the sync parameter, and do the put calls with that. Then compare the | results… | Aye aye, will do. Radim

Hi, I've written down this proposal for the implementation of new cache store. https://community.jboss.org/wiki/BrnoCacheStoreDesignProposal WDYT? Radim ----- Original Message ----- | From: "Radim Vansa" <rvansa(a)redhat.com&gt; | To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; | Sent: Thursday, June 27, 2013 2:37:43 PM | Subject: Re: [infinispan-dev] Cachestores performance | | | | ----- Original Message ----- | | From: "Galder Zamarreño" <galder(a)redhat.com&gt; | | To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; | | Sent: Thursday, June 27, 2013 1:52:11 PM | | Subject: Re: [infinispan-dev] Cachestores performance | | | | > As for Karsten's FCS implementation, I too have issues with the key set | | > and | | > value offsets being solely in memory. However I think that could be | | > improved by storing only a certain number of keys/offsets in memory, and | | > flushing the rest to disk again into an index file. | | | | ^ Karsten's implementation makes this relatively easy to achieve because it | | already keeps this mapping in a LinkedHashMap (with a given max entries | | limit [1]) assuming removeEldestEntry() is overriden to flush to disk older | | entries. Some extra logic would be needed to bring back data from the disk | | too… but your suggestion below is also quite interesting... | | I certainly wouldn't call this easy task, because the most problematic part | is what we will do when the whole entry (both key and value) are gone from | memory and we want to read them - that requires keeping some searchable | structure on-disk. And that's the hard stuff. | | | | | > I believe LevelDB follows a similar design, but I think Karsten's FCS | | > will | | > perform better than LevelDB since it doesn't attempt to maintain a sorted | | > structure on disk. | | | | ^ In-memory, the structure can optionally be ordered if it's bound [1], | | otherwise it's just a normal map. How would be store it at the disk level? | | B+ tree with hashes of keys and then linked lists? | | Before choosing "I love B#@& trees, let's use B#@& trees!", I'd find out what | requirements do we have for the structure. I believe that the index itself | should not be considered persistent, as it can be rebuilt when preloading | the data (sequentially reading the data is fast, therefore we can afford do | this indexing preload), the reason of the index being on-disk is that we | don't have enough memory to store all keys, or even key hashes. Therefore it | does not have to be updated synchronously with the writes. It should be | mostly read-optimized then, because that's the thing where we need | synchronous access to this structure. | | | | | > One approach to maintaining keys and offsets in memory could be a | | > WeakReference that points to the key stored in the in-memory | | > DataContainer. Once evicted from the DC, then the CacheStore impl would | | > need to fetch the key again from the index file before looking up the | | > value in the actual store. | | | | ^ Hmmm, interesting idea… has the potential to safe the memory space by not | | having to keep that extra data structure in the cache store. | | You mean to mix the DataContainer with xCacheEntry implementation and the | cache store implementation? Is that possible from design perspective? | Speaking about different kind of references, we may even optimize | not-well-tuned eviction by SoftReferences, so that even if the entry was | evicted from main DataContainer, we'd keep the value referenced from the | cache-store (and this does not have to be loaded from disk if referenced | before garbage collection). But such thought may be premature optimization. | For having eviction managed in relation with GC we should rather combine | this with PhantomReferences, where entries would be written to cache upon | finalization. | | | | | > This way we have hot items always in memory, semi-hot items with offsets | | > in | | > memory and values on disk, and cold items needing to be read off disk | | > entirely (both offset and value). Also for write-through and | | > write-behind, as long as the item is hot or warm (key and offset in | | > memory), writing will be pretty fast. | | | | My worry about Karsten's impl is writing actually. If you look at the last | | performance numbers in [2], where we see the performance difference of | | force=true and force=false in Karsten's cache store compared with LevelDB | | JNI, you see that force=false is fastest, then JNI LevelDB, and the | | force=true. Me wonders what kind of write guarantees LevelDB JNI provides | | (and the JAVA version)... | | Just for clarification: the fast implementation is without force at all, the | slower is with force(false). Force(true) means updating metadata (such as | access times?) which is not required for cache-store. | But the numbers suggest that the random access with syncing is really not a | good option, and that we should rather use the temporary append-only log, | which would be persisted into structured DB by different thread (as LevelDB | does, I suppose). | | Thinking about all the levels and cache structures optimizing the read | access, I can see four levels of search structures: key + value (usual | DataContainer), key + offset, hash + offset, all on disk. The "hash + | offset" may seem superflous but for some use-cases with big keys it may be | worth sparing a few disk look-ups. | | Radim | | | > | | > On 27 Jun 2013, at 10:33, Radim Vansa <rvansa(a)redhat.com&gt; wrote: | | > | | >> Oops, by the cache store I mean the previously-superfast | | >> KarstenFileCacheStore implementation. | | >> | | >> ----- Original Message ----- | | >> | From: "Radim Vansa" <rvansa(a)redhat.com&gt; | | >> | To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; | | >> | Sent: Thursday, June 27, 2013 11:30:53 AM | | >> | Subject: Re: [infinispan-dev] Cachestores performance | | >> | | | >> | I have added FileChannel.force(false) flushes after all write | | >> | operations | | >> | in | | >> | the cache store, and now the comparison is also updated with these | | >> | values. | | >> | | | >> | Radim | | >> | | | >> | ----- Original Message ----- | | >> | | From: "Radim Vansa" <rvansa(a)redhat.com&gt; | | >> | | To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; | | >> | | Sent: Thursday, June 27, 2013 8:54:25 AM | | >> | | Subject: Re: [infinispan-dev] Cachestores performance | | >> | | | | >> | | Yep, write-through. LevelDB JAVA used FileChannelTable | | >> | | implementation | | >> | | (-Dleveldb.mmap), because Mmaping is not implemented very well and | | >> | | causes | | >> | | JVM crashes (I believe it's because of calling non-public API via | | >> | | reflection | | >> | | - I've found post from the Oracle JVM guys discouraging the | | >> | | particular | | >> | | trick | | >> | | it uses). After writing the record to the log, it calls | | >> | | FileChannel.force(true), therefore, it should be really on the disc | | >> | | by | | >> | | that | | >> | | moment. | | >> | | I have not looked into the JNI implementation but I expect the same. | | >> | | | | >> | | By the way, I have updated [1] with numbers when running on more | | >> | | data | | >> | | (2 GB | | >> | | instead of 100 MB). I won't retype it here, so look there. The | | >> | | performance | | >> | | is much lower. | | >> | | I may try also increase JVM heap size and try with a bit more data | | >> | | yet. | | >> | | | | >> | | Radim | | >> | | | | >> | | [1] https://community.jboss.org/wiki/FileCacheStoreRedesign | | >> | | | | >> | | ----- Original Message ----- | | >> | | | From: "Erik Salter" <an1310(a)hotmail.com&gt; | | >> | | | To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; | | >> | | | Sent: Wednesday, June 26, 2013 7:40:19 PM | | >> | | | Subject: Re: [infinispan-dev] Cachestores performance | | >> | | | | | >> | | | These were write-through cache stores, right? And with LevelDB, | | >> | | | this was | | >> | | | through to the database file itself? | | >> | | | | | >> | | | Erik | | >> | | | | | >> | | | -----Original Message----- | | >> | | | From: infinispan-dev-bounces(a)lists.jboss.org | | >> | | | [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Radim | | >> | | | Vansa | | >> | | | Sent: Wednesday, June 26, 2013 11:24 AM | | >> | | | To: infinispan -Dev List | | >> | | | Subject: [infinispan-dev] Cachestores performance | | >> | | | | | >> | | | Hi all, | | >> | | | | | >> | | | according to [1] I've created the comparison of performance in | | >> | | | stress-tests. | | >> | | | | | >> | | | All setups used local-cache, benchmark was executed via Radargun | | >> | | | (actually | | >> | | | version not merged into master yet [2]). I've used 4 nodes just to | | >> | | | get | | >> | | | more | | >> | | | data - each slave was absolutely independent of the others. | | >> | | | | | >> | | | First test was preloading performance - the cache started and | | >> | | | tried | | >> | | | to | | >> | | | load | | >> | | | 1GB of data from harddrive. Without cachestore the startup takes | | >> | | | about 2 | | >> | | | - | | >> | | | 4 | | >> | | | seconds, average numbers for the cachestores are below: | | >> | | | | | >> | | | FileCacheStore: 9.8 s | | >> | | | KarstenFileCacheStore: 14 s | | >> | | | LevelDB-JAVA impl.: 12.3 s | | >> | | | LevelDB-JNI impl.: 12.9 s | | >> | | | | | >> | | | IMO nothing special, all times seem affordable. We don't benchmark | | >> | | | exactly | | >> | | | storing the data into the cachestore, here FileCacheStore took | | >> | | | about | | >> | | | 44 | | >> | | | minutes, while Karsten about 38 seconds, LevelDB-JAVA 4 minutes | | >> | | | and | | >> | | | LevelDB-JNI 96 seconds. The units are right, it's minutes compared | | >> | | | to | | >> | | | seconds. But we all know that FileCacheStore is bloody slow. | | >> | | | | | >> | | | Second test is stress test (5 minutes, preceded by 2 minute | | >> | | | warmup) | | >> | | | where | | >> | | | each of 10 threads works on 10k entries with 1kB values (~100 MB | | >> | | | in | | >> | | | total). | | >> | | | 20 % writes, 80 % reads, as usual. No eviction is configured, | | >> | | | therefore | | >> | | | the | | >> | | | cache-store works as a persistent storage only for case of crash. | | >> | | | | | >> | | | FileCacheStore: 3.1M reads/s 112 writes/s // on one | | >> | | | node | | >> | | | the | | >> | | | performance was only 2.96M reads/s 75 writes/s | | >> | | | KarstenFileCacheStore: 9.2M reads/s 226k writes/s // yikes! | | >> | | | LevelDB-JAVA impl.: 3.9M reads/s 5100 writes/s | | >> | | | LevelDB-JNI impl.: 6.6M reads/s 14k writes/s // on one | | >> | | | node | | >> | | | the | | >> | | | performance was 3.9M/8.3k - about half of the others | | >> | | | Without cache store: 15.5M reads/s 4.4M writes/s | | >> | | | | | >> | | | Karsten implementation pretty rules here for two reasons. First of | | >> | | | all, | | >> | | | it | | >> | | | does not flush the data (it calls only RandomAccessFile.write()). | | >> | | | Other | | >> | | | cheat is that it stores in-memory the keys and offsets of data | | >> | | | values in | | >> | | | the | | >> | | | database file. Therefore, it's definitely the best choice for this | | >> | | | scenario, | | >> | | | but it does not allow to scale the cache-store, especially in | | >> | | | cases | | >> | | | where | | >> | | | the keys are big and values small. However, this performance boost | | >> | | | is | | >> | | | definitely worth checking - I could think of caching the disk | | >> | | | offsets in | | >> | | | memory and querying persistent index only in case of missing | | >> | | | record, | | >> | | | with | | >> | | | part of the persistent index flushed asynchronously (the index can | | >> | | | be | | >> | | | always | | >> | | | rebuilt during the preloading for case of crash). | | >> | | | | | >> | | | The third test should have tested the scenario with more data to | | >> | | | be | | >> | | | stored | | >> | | | than memory - therefore, the stressors operated on 100k entries | | >> | | | (~100 MB | | >> | | | of | | >> | | | data) but eviction was set to 10k entries (9216 entries ended up | | >> | | | in | | >> | | | memory | | >> | | | after the test has ended). | | >> | | | | | >> | | | FileCacheStore: 750 reads/s 285 writes/s // | | >> | | | one | | >> | | | node | | >> | | | had | | >> | | | only 524 reads and 213 writes per second | | >> | | | KarstenFileCacheStore: 458k reads/s 137k writes/s | | >> | | | LevelDB-JAVA impl.: 21k reads/s 9k writes/s // a | | >> | | | bit | | >> | | | varying | | >> | | | performance | | >> | | | LevelDB-JNI impl.: 13k-46k reads/s 6.6k-15.2k writes/s // | | >> | | | the | | >> | | | performance varied a lot! | | >> | | | | | >> | | | 100 MB of data is not much, but it takes so long to push it into | | >> | | | FileCacheStore that I won't use more unless we exclude this loser | | >> | | | from | | >> | | | the | | >> | | | comparison :) | | >> | | | | | >> | | | Radim | | >> | | | | | >> | | | [1] https://community.jboss.org/wiki/FileCacheStoreRedesign | | >> | | | [2] https://github.com/rvansa/radargun/tree/t_keygen | | >> | | | | | >> | | | ----------------------------------------------------------- | | >> | | | Radim Vansa | | >> | | | Quality Assurance Engineer | | >> | | | JBoss Datagrid | | >> | | | tel. +420532294559 ext. 62559 | | >> | | | | | >> | | | Red Hat Czech, s.r.o. | | >> | | | Brno, Purkyňova 99/71, PSČ 612 45 | | >> | | | Czech Republic | | >> | | | | | >> | | | | | >> | | | _______________________________________________ | | >> | | | infinispan-dev mailing list | | >> | | | infinispan-dev(a)lists.jboss.org | | >> | | | https://lists.jboss.org/mailman/listinfo/infinispan-dev | | >> | | | | | >> | | | | | >> | | | _______________________________________________ | | >> | | | infinispan-dev mailing list | | >> | | | infinispan-dev(a)lists.jboss.org | | >> | | | https://lists.jboss.org/mailman/listinfo/infinispan-dev | | >> | | | | >> | | _______________________________________________ | | >> | | infinispan-dev mailing list | | >> | | infinispan-dev(a)lists.jboss.org | | >> | | https://lists.jboss.org/mailman/listinfo/infinispan-dev | | >> | | | >> | _______________________________________________ | | >> | infinispan-dev mailing list | | >> | infinispan-dev(a)lists.jboss.org | | >> | https://lists.jboss.org/mailman/listinfo/infinispan-dev | | >> | | >> _______________________________________________ | | >> infinispan-dev mailing list | | >> infinispan-dev(a)lists.jboss.org | | >> https://lists.jboss.org/mailman/listinfo/infinispan-dev | | > | | > -- | | > Manik Surtani | | > manik(a)jboss.org | | > twitter.com/maniksurtani | | > | | > Platform Architect, JBoss Data Grid | | > http://red.ht/data-grid | | > | | > | | > _______________________________________________ | | > infinispan-dev mailing list | | > infinispan-dev(a)lists.jboss.org | | > https://lists.jboss.org/mailman/listinfo/infinispan-dev | | | | | | -- | | Galder Zamarreño | | galder(a)redhat.com | | twitter.com/galderz | | | | Project Lead, Escalante | | http://escalante.io | | | | Engineer, Infinispan | | http://infinispan.org | | | | | | _______________________________________________ | | infinispan-dev mailing list | | infinispan-dev(a)lists.jboss.org | | https://lists.jboss.org/mailman/listinfo/infinispan-dev | | _______________________________________________ | infinispan-dev mailing list | infinispan-dev(a)lists.jboss.org | https://lists.jboss.org/mailman/listinfo/infinispan-dev

| Given such a scenario, I am not interested at all in synchronous | storage. Before we commit into a design which is basically assuming | the need for synchronous storage guarantees, I'd like to understand | what kind of use case it's aiming to solve. You're right that I may miss the wider perspective and that I am too much focused on storing stuff reliably when we have reliability assured by multiple owners. | | Only then we would be able to pick a design (or multiple ones); for my | use case the proposal from Karsten seems excellent, so I'm wondering | why I should be looking for alternatives, and wondering why everyone | is still wasting time on different discussions :-D Karsten implementation has one major drawback: all the keys are stored in-memory, and there's no search structure for overflow. For use case we've had with bigger keys and very small values this is simply not an option. However, we have the LevelDB JNI implementation for this purpose and the numbers are not bad in non-syncing mode. Do we want to have some efficient no-dependency cache-store? If the question is no, okay, let use recommend LevelDB JNI for this purposes. | | I'm pretty sure there is people looking forward for a synch-CacheStore | too: if you could nail down such a scenario however I'm pretty sure | that some other considerations would not be taken into account (like | consistency of data when reactivating a dormant node), so I suspect | that just implementing such a component would actually not make any | new architecture possible, as you would get blocked by other problems | which need to be solved too.. better define all expectations asap! | | To me this thread smells of needing the off-heap Direct Memory buffers | which I suggested [long time ago] to efficiently offload internal | buffers, Yes, when it comes to reducing GC pauses, that is the best option. However, you can't have enough RAM to handle everything in-memory (off-heap), persistent storage for overflow is a must-have. | but failing to recognise this we're pushing responsibility to | an epic level complex CacheStore.. guys let's not forget that a mayor | bottleneck of CacheStores today is the SPI it has to implement, we | identified several limitations in the contract in the past which | prevent a superior efficiency: we're working towards a mayor release | now so I'd rather focus on the API changes which will make it possible | to get decent performance even without changing any storage engine.. | I'm pretty sure Cassandra (to pick one) doesn't scale too bad. The issues common to all local cache-stores are definitely something to do. I just don't feel enough into the problematics to handle them myself - that's why I jumped onto implementing just another cache store. Radim | | Cheers, | Sanne | | | | On 2 July 2013 10:09, Radim Vansa <rvansa(a)redhat.com&gt; wrote: | > Hi, | > | > I've written down this proposal for the implementation of new cache store. | > | > https://community.jboss.org/wiki/BrnoCacheStoreDesignProposal | > | > WDYT? | > | > Radim | > | > ----- Original Message ----- | > | From: "Radim Vansa" <rvansa(a)redhat.com&gt; | > | To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; | > | Sent: Thursday, June 27, 2013 2:37:43 PM | > | Subject: Re: [infinispan-dev] Cachestores performance | > | | > | | > | | > | ----- Original Message ----- | > | | From: "Galder Zamarreño" <galder(a)redhat.com&gt; | > | | To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; | > | | Sent: Thursday, June 27, 2013 1:52:11 PM | > | | Subject: Re: [infinispan-dev] Cachestores performance | > | | | > | | > As for Karsten's FCS implementation, I too have issues with the key | > | | > set | > | | > and | > | | > value offsets being solely in memory. However I think that could be | > | | > improved by storing only a certain number of keys/offsets in memory, | > | | > and | > | | > flushing the rest to disk again into an index file. | > | | | > | | ^ Karsten's implementation makes this relatively easy to achieve | > | | because it | > | | already keeps this mapping in a LinkedHashMap (with a given max entries | > | | limit [1]) assuming removeEldestEntry() is overriden to flush to disk | > | | older | > | | entries. Some extra logic would be needed to bring back data from the | > | | disk | > | | too… but your suggestion below is also quite interesting... | > | | > | I certainly wouldn't call this easy task, because the most problematic | > | part | > | is what we will do when the whole entry (both key and value) are gone | > | from | > | memory and we want to read them - that requires keeping some searchable | > | structure on-disk. And that's the hard stuff. | > | | > | | | > | | > I believe LevelDB follows a similar design, but I think Karsten's FCS | > | | > will | > | | > perform better than LevelDB since it doesn't attempt to maintain a | > | | > sorted | > | | > structure on disk. | > | | | > | | ^ In-memory, the structure can optionally be ordered if it's bound [1], | > | | otherwise it's just a normal map. How would be store it at the disk | > | | level? | > | | B+ tree with hashes of keys and then linked lists? | > | | > | Before choosing "I love B#@& trees, let's use B#@& trees!", I'd find out | > | what | > | requirements do we have for the structure. I believe that the index | > | itself | > | should not be considered persistent, as it can be rebuilt when preloading | > | the data (sequentially reading the data is fast, therefore we can afford | > | do | > | this indexing preload), the reason of the index being on-disk is that we | > | don't have enough memory to store all keys, or even key hashes. Therefore | > | it | > | does not have to be updated synchronously with the writes. It should be | > | mostly read-optimized then, because that's the thing where we need | > | synchronous access to this structure. | > | | > | | | > | | > One approach to maintaining keys and offsets in memory could be a | > | | > WeakReference that points to the key stored in the in-memory | > | | > DataContainer. Once evicted from the DC, then the CacheStore impl | > | | > would | > | | > need to fetch the key again from the index file before looking up the | > | | > value in the actual store. | > | | | > | | ^ Hmmm, interesting idea… has the potential to safe the memory space by | > | | not | > | | having to keep that extra data structure in the cache store. | > | | > | You mean to mix the DataContainer with xCacheEntry implementation and the | > | cache store implementation? Is that possible from design perspective? | > | Speaking about different kind of references, we may even optimize | > | not-well-tuned eviction by SoftReferences, so that even if the entry was | > | evicted from main DataContainer, we'd keep the value referenced from the | > | cache-store (and this does not have to be loaded from disk if referenced | > | before garbage collection). But such thought may be premature | > | optimization. | > | For having eviction managed in relation with GC we should rather combine | > | this with PhantomReferences, where entries would be written to cache upon | > | finalization. | > | | > | | | > | | > This way we have hot items always in memory, semi-hot items with | > | | > offsets | > | | > in | > | | > memory and values on disk, and cold items needing to be read off disk | > | | > entirely (both offset and value). Also for write-through and | > | | > write-behind, as long as the item is hot or warm (key and offset in | > | | > memory), writing will be pretty fast. | > | | | > | | My worry about Karsten's impl is writing actually. If you look at the | > | | last | > | | performance numbers in [2], where we see the performance difference of | > | | force=true and force=false in Karsten's cache store compared with | > | | LevelDB | > | | JNI, you see that force=false is fastest, then JNI LevelDB, and the | > | | force=true. Me wonders what kind of write guarantees LevelDB JNI | > | | provides | > | | (and the JAVA version)... | > | | > | Just for clarification: the fast implementation is without force at all, | > | the | > | slower is with force(false). Force(true) means updating metadata (such as | > | access times?) which is not required for cache-store. | > | But the numbers suggest that the random access with syncing is really not | > | a | > | good option, and that we should rather use the temporary append-only log, | > | which would be persisted into structured DB by different thread (as | > | LevelDB | > | does, I suppose). | > | | > | Thinking about all the levels and cache structures optimizing the read | > | access, I can see four levels of search structures: key + value (usual | > | DataContainer), key + offset, hash + offset, all on disk. The "hash + | > | offset" may seem superflous but for some use-cases with big keys it may | > | be | > | worth sparing a few disk look-ups. | > | | > | Radim | > | | > | | > | > | | > On 27 Jun 2013, at 10:33, Radim Vansa <rvansa(a)redhat.com&gt; wrote: | > | | > | > | | >> Oops, by the cache store I mean the previously-superfast | > | | >> KarstenFileCacheStore implementation. | > | | >> | > | | >> ----- Original Message ----- | > | | >> | From: "Radim Vansa" <rvansa(a)redhat.com&gt; | > | | >> | To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; | > | | >> | Sent: Thursday, June 27, 2013 11:30:53 AM | > | | >> | Subject: Re: [infinispan-dev] Cachestores performance | > | | >> | | > | | >> | I have added FileChannel.force(false) flushes after all write | > | | >> | operations | > | | >> | in | > | | >> | the cache store, and now the comparison is also updated with these | > | | >> | values. | > | | >> | | > | | >> | Radim | > | | >> | | > | | >> | ----- Original Message ----- | > | | >> | | From: "Radim Vansa" <rvansa(a)redhat.com&gt; | > | | >> | | To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; | > | | >> | | Sent: Thursday, June 27, 2013 8:54:25 AM | > | | >> | | Subject: Re: [infinispan-dev] Cachestores performance | > | | >> | | | > | | >> | | Yep, write-through. LevelDB JAVA used FileChannelTable | > | | >> | | implementation | > | | >> | | (-Dleveldb.mmap), because Mmaping is not implemented very well | > | | >> | | and | > | | >> | | causes | > | | >> | | JVM crashes (I believe it's because of calling non-public API | > | | >> | | via | > | | >> | | reflection | > | | >> | | - I've found post from the Oracle JVM guys discouraging the | > | | >> | | particular | > | | >> | | trick | > | | >> | | it uses). After writing the record to the log, it calls | > | | >> | | FileChannel.force(true), therefore, it should be really on the | > | | >> | | disc | > | | >> | | by | > | | >> | | that | > | | >> | | moment. | > | | >> | | I have not looked into the JNI implementation but I expect the | > | | >> | | same. | > | | >> | | | > | | >> | | By the way, I have updated [1] with numbers when running on more | > | | >> | | data | > | | >> | | (2 GB | > | | >> | | instead of 100 MB). I won't retype it here, so look there. The | > | | >> | | performance | > | | >> | | is much lower. | > | | >> | | I may try also increase JVM heap size and try with a bit more | > | | >> | | data | > | | >> | | yet. | > | | >> | | | > | | >> | | Radim | > | | >> | | | > | | >> | | [1] https://community.jboss.org/wiki/FileCacheStoreRedesign | > | | >> | | | > | | >> | | ----- Original Message ----- | > | | >> | | | From: "Erik Salter" <an1310(a)hotmail.com&gt; | > | | >> | | | To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; | > | | >> | | | Sent: Wednesday, June 26, 2013 7:40:19 PM | > | | >> | | | Subject: Re: [infinispan-dev] Cachestores performance | > | | >> | | | | > | | >> | | | These were write-through cache stores, right? And with | > | | >> | | | LevelDB, | > | | >> | | | this was | > | | >> | | | through to the database file itself? | > | | >> | | | | > | | >> | | | Erik | > | | >> | | | | > | | >> | | | -----Original Message----- | > | | >> | | | From: infinispan-dev-bounces(a)lists.jboss.org | > | | >> | | | [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of | > | | >> | | | Radim | > | | >> | | | Vansa | > | | >> | | | Sent: Wednesday, June 26, 2013 11:24 AM | > | | >> | | | To: infinispan -Dev List | > | | >> | | | Subject: [infinispan-dev] Cachestores performance | > | | >> | | | | > | | >> | | | Hi all, | > | | >> | | | | > | | >> | | | according to [1] I've created the comparison of performance in | > | | >> | | | stress-tests. | > | | >> | | | | > | | >> | | | All setups used local-cache, benchmark was executed via | > | | >> | | | Radargun | > | | >> | | | (actually | > | | >> | | | version not merged into master yet [2]). I've used 4 nodes | > | | >> | | | just to | > | | >> | | | get | > | | >> | | | more | > | | >> | | | data - each slave was absolutely independent of the others. | > | | >> | | | | > | | >> | | | First test was preloading performance - the cache started and | > | | >> | | | tried | > | | >> | | | to | > | | >> | | | load | > | | >> | | | 1GB of data from harddrive. Without cachestore the startup | > | | >> | | | takes | > | | >> | | | about 2 | > | | >> | | | - | > | | >> | | | 4 | > | | >> | | | seconds, average numbers for the cachestores are below: | > | | >> | | | | > | | >> | | | FileCacheStore: 9.8 s | > | | >> | | | KarstenFileCacheStore: 14 s | > | | >> | | | LevelDB-JAVA impl.: 12.3 s | > | | >> | | | LevelDB-JNI impl.: 12.9 s | > | | >> | | | | > | | >> | | | IMO nothing special, all times seem affordable. We don't | > | | >> | | | benchmark | > | | >> | | | exactly | > | | >> | | | storing the data into the cachestore, here FileCacheStore took | > | | >> | | | about | > | | >> | | | 44 | > | | >> | | | minutes, while Karsten about 38 seconds, LevelDB-JAVA 4 | > | | >> | | | minutes | > | | >> | | | and | > | | >> | | | LevelDB-JNI 96 seconds. The units are right, it's minutes | > | | >> | | | compared | > | | >> | | | to | > | | >> | | | seconds. But we all know that FileCacheStore is bloody slow. | > | | >> | | | | > | | >> | | | Second test is stress test (5 minutes, preceded by 2 minute | > | | >> | | | warmup) | > | | >> | | | where | > | | >> | | | each of 10 threads works on 10k entries with 1kB values (~100 | > | | >> | | | MB | > | | >> | | | in | > | | >> | | | total). | > | | >> | | | 20 % writes, 80 % reads, as usual. No eviction is configured, | > | | >> | | | therefore | > | | >> | | | the | > | | >> | | | cache-store works as a persistent storage only for case of | > | | >> | | | crash. | > | | >> | | | | > | | >> | | | FileCacheStore: 3.1M reads/s 112 writes/s // on one | > | | >> | | | node | > | | >> | | | the | > | | >> | | | performance was only 2.96M reads/s 75 writes/s | > | | >> | | | KarstenFileCacheStore: 9.2M reads/s 226k writes/s // yikes! | > | | >> | | | LevelDB-JAVA impl.: 3.9M reads/s 5100 writes/s | > | | >> | | | LevelDB-JNI impl.: 6.6M reads/s 14k writes/s // on one | > | | >> | | | node | > | | >> | | | the | > | | >> | | | performance was 3.9M/8.3k - about half of the others | > | | >> | | | Without cache store: 15.5M reads/s 4.4M writes/s | > | | >> | | | | > | | >> | | | Karsten implementation pretty rules here for two reasons. | > | | >> | | | First of | > | | >> | | | all, | > | | >> | | | it | > | | >> | | | does not flush the data (it calls only | > | | >> | | | RandomAccessFile.write()). | > | | >> | | | Other | > | | >> | | | cheat is that it stores in-memory the keys and offsets of data | > | | >> | | | values in | > | | >> | | | the | > | | >> | | | database file. Therefore, it's definitely the best choice for | > | | >> | | | this | > | | >> | | | scenario, | > | | >> | | | but it does not allow to scale the cache-store, especially in | > | | >> | | | cases | > | | >> | | | where | > | | >> | | | the keys are big and values small. However, this performance | > | | >> | | | boost | > | | >> | | | is | > | | >> | | | definitely worth checking - I could think of caching the disk | > | | >> | | | offsets in | > | | >> | | | memory and querying persistent index only in case of missing | > | | >> | | | record, | > | | >> | | | with | > | | >> | | | part of the persistent index flushed asynchronously (the index | > | | >> | | | can | > | | >> | | | be | > | | >> | | | always | > | | >> | | | rebuilt during the preloading for case of crash). | > | | >> | | | | > | | >> | | | The third test should have tested the scenario with more data | > | | >> | | | to | > | | >> | | | be | > | | >> | | | stored | > | | >> | | | than memory - therefore, the stressors operated on 100k | > | | >> | | | entries | > | | >> | | | (~100 MB | > | | >> | | | of | > | | >> | | | data) but eviction was set to 10k entries (9216 entries ended | > | | >> | | | up | > | | >> | | | in | > | | >> | | | memory | > | | >> | | | after the test has ended). | > | | >> | | | | > | | >> | | | FileCacheStore: 750 reads/s 285 writes/s | > | | >> | | | // | > | | >> | | | one | > | | >> | | | node | > | | >> | | | had | > | | >> | | | only 524 reads and 213 writes per second | > | | >> | | | KarstenFileCacheStore: 458k reads/s 137k writes/s | > | | >> | | | LevelDB-JAVA impl.: 21k reads/s 9k writes/s | > | | >> | | | // a | > | | >> | | | bit | > | | >> | | | varying | > | | >> | | | performance | > | | >> | | | LevelDB-JNI impl.: 13k-46k reads/s 6.6k-15.2k writes/s | > | | >> | | | // | > | | >> | | | the | > | | >> | | | performance varied a lot! | > | | >> | | | | > | | >> | | | 100 MB of data is not much, but it takes so long to push it | > | | >> | | | into | > | | >> | | | FileCacheStore that I won't use more unless we exclude this | > | | >> | | | loser | > | | >> | | | from | > | | >> | | | the | > | | >> | | | comparison :) | > | | >> | | | | > | | >> | | | Radim | > | | >> | | | | > | | >> | | | [1] https://community.jboss.org/wiki/FileCacheStoreRedesign | > | | >> | | | [2] https://github.com/rvansa/radargun/tree/t_keygen | > | | >> | | | | > | | >> | | | ----------------------------------------------------------- | > | | >> | | | Radim Vansa | > | | >> | | | Quality Assurance Engineer | > | | >> | | | JBoss Datagrid | > | | >> | | | tel. +420532294559 ext. 62559 | > | | >> | | | | > | | >> | | | Red Hat Czech, s.r.o. | > | | >> | | | Brno, Purkyňova 99/71, PSČ 612 45 | > | | >> | | | Czech Republic | > | | >> | | | | > | | >> | | | | > | | >> | | | _______________________________________________ | > | | >> | | | infinispan-dev mailing list | > | | >> | | | infinispan-dev(a)lists.jboss.org | > | | >> | | | https://lists.jboss.org/mailman/listinfo/infinispan-dev | > | | >> | | | | > | | >> | | | | > | | >> | | | _______________________________________________ | > | | >> | | | infinispan-dev mailing list | > | | >> | | | infinispan-dev(a)lists.jboss.org | > | | >> | | | https://lists.jboss.org/mailman/listinfo/infinispan-dev | > | | >> | | | > | | >> | | _______________________________________________ | > | | >> | | infinispan-dev mailing list | > | | >> | | infinispan-dev(a)lists.jboss.org | > | | >> | | https://lists.jboss.org/mailman/listinfo/infinispan-dev | > | | >> | | > | | >> | _______________________________________________ | > | | >> | infinispan-dev mailing list | > | | >> | infinispan-dev(a)lists.jboss.org | > | | >> | https://lists.jboss.org/mailman/listinfo/infinispan-dev | > | | >> | > | | >> _______________________________________________ | > | | >> infinispan-dev mailing list | > | | >> infinispan-dev(a)lists.jboss.org | > | | >> https://lists.jboss.org/mailman/listinfo/infinispan-dev | > | | > | > | | > -- | > | | > Manik Surtani | > | | > manik(a)jboss.org | > | | > twitter.com/maniksurtani | > | | > | > | | > Platform Architect, JBoss Data Grid | > | | > http://red.ht/data-grid | > | | > | > | | > | > | | > _______________________________________________ | > | | > infinispan-dev mailing list | > | | > infinispan-dev(a)lists.jboss.org | > | | > https://lists.jboss.org/mailman/listinfo/infinispan-dev | > | | | > | | | > | | -- | > | | Galder Zamarreño | > | | galder(a)redhat.com | > | | twitter.com/galderz | > | | | > | | Project Lead, Escalante | > | | http://escalante.io | > | | | > | | Engineer, Infinispan | > | | http://infinispan.org | > | | | > | | | > | | _______________________________________________ | > | | infinispan-dev mailing list | > | | infinispan-dev(a)lists.jboss.org | > | | https://lists.jboss.org/mailman/listinfo/infinispan-dev | > | | > | _______________________________________________ | > | infinispan-dev mailing list | > | infinispan-dev(a)lists.jboss.org | > | https://lists.jboss.org/mailman/listinfo/infinispan-dev | > | > _______________________________________________ | > infinispan-dev mailing list | > infinispan-dev(a)lists.jboss.org | > https://lists.jboss.org/mailman/listinfo/infinispan-dev | | _______________________________________________ | infinispan-dev mailing list | infinispan-dev(a)lists.jboss.org | https://lists.jboss.org/mailman/listinfo/infinispan-dev

On Jul 2, 2013, at 3:32 PM, Radim Vansa <rvansa(a)redhat.com&gt; wrote: That's been my feeling as well. Karten's implementation, in spite of its limitations, is a very good base and kicks performance of our current offering. Sure, it has some drawbacks, but as Radim said, when bigger keys are used, or many keys, we could recommend the LevelDB JNI cache store alternative. Instead of writing a brand new cache store from scratch, it might more valuable to figure out what the threshold is after which the the LevelDB option should be used instead of Karsten's cache store. This could be explored by increasily testing keys of different sizes, and/or different amount of keys stored. WDYT? This way, we can provide better guidance on when to use each one. The starting point has been this: https://issues.jboss.org/browse/ISPN-2806 We need a faster FCS. We already have several options that are faster, we all seem to agree that Karten's FCS and LevelDB JNI offer the best solutions, each with their own caveats. The primary difference between them two, as far as I can see from Radim's performance test results, is the read performance in overflow. Karsten's characteristics makes it incredibly fast here, but there's a limit to it. If we can figure out what that limit is, we have all the info that we can provide to the user: if you store keys with size bigger than X, or store more than Y keys in the cache, use LevelDB JNI, otherwise use Karsten's FCS (which would really be the new FCS). ^ Indeed, that's been discussed in [1], and you've contributed to it, but I think having a faster FCS to start with is a bigger priority than this. Our FCS has been bashed pretty much since day one, and we know have faster alternatives. We just need to come up with the right recommendations on how to use them, and then we can focus on [1]. Also, part of me wants JCache to be finaled before tackling [1], so that we have a stable cacheloader/cachewriter interface at the spec level. I think it's pretty stable now, but you never know. [1] https://community.jboss.org/wiki/CacheLoaderAndCacheStoreSPIRedesign -- Galder Zamarreño galder(a)redhat.com twitter.com/galderz Project Lead, Escalante http://escalante.io Engineer, Infinispan http://infinispan.org

I concur with part of the below, but with a few changes: - The cache is the primary storage, similar to Sanne's case. (DIST mode) - My customers are not interested in extra components to the system, like databases or Cassandra nodes. They wonder why they can't simply use the existing file system on the nodes they have. - I'm only going to be using the filesystem to recover in the case of upgrades and catastrophic failures. So during normal operation, flushes to disk cannot impact cluster performance. - Most importantly, there needs to be a way, scripted or otherwise, to recover the keys from local storage in a DIST-mode cache. I cannot guarantee anything regarding node ordering, so anything about persisting segment info/previous CHs are out. If that means copying all LevelDB SST files to all nodes and restarting them, that's fine. At the executive levels of my customer, they don't see (or really care about) the differentiation between data grids and MySQL -- only that one has file-based persistence and the other doesn't. In production, we've already taken a massive outage where a unbelievable series of coincidences occurred to reveal a JBoss AS bug that ended up deadlocking all threads in the cluster and we had to restart all nodes. And I'm sure it'll happen again. Hope this offers some user perspective. Erik -----Original Message----- From: infinispan-dev-bounces(a)lists.jboss.org [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Sanne Grinovero Sent: Tuesday, July 02, 2013 8:47 AM To: infinispan -Dev List Subject: Re: [infinispan-dev] Cachestores performance It would be nice to have a deck of "cheat sheets" on the expected use cases and guarantees: to me it looks like everyone is trying to solve a different problem / having a different problem in mind. My own take on it: Scenario #1 I'll primarily use Infinispan with DIST, and I don't care much for other options. Reliability is guaranteed via numOwners>1, NOT by persisting to disk: if a node fails, I kill the VM (the machine, not the Java process) and start new ones to compensate: I'm assuming cloud nodes, so it's likely that when a failed node is gone, the disk is gone as well, with all the carefully stored data. I will use Infinispan primarily to absorb write spikes - so a "synch flush" is no good for me - and to boost read performance by as much memory I can throw at it. CacheStore is used for two reasons: - overflow (LIRS+passivation) for then the memory is not enough - clean shutdown: you can think of it as a way to be able to upgrade some component in the system (Infinispan or my own); I would expect some kind of "JMX flush" operation to do a clean shutdown without data loss. Given such a scenario, I am not interested at all in synchronous storage. Before we commit into a design which is basically assuming the need for synchronous storage guarantees, I'd like to understand what kind of use case it's aiming to solve. It would be great to document each such use case and put down a table of things which can be expected, which features should not be expected (be very explicit on the limitations), and how basic operations are expected to be performed in the scenario: like how do you do a rolling upgrade in Scenario 1# ? How do you do a backup? And of course some configurations & code examples. Only then we would be able to pick a design (or multiple ones); for my use case the proposal from Karsten seems excellent, so I'm wondering why I should be looking for alternatives, and wondering why everyone is still wasting time on different discussions :-D I'm pretty sure there is people looking forward for a synch-CacheStore too: if you could nail down such a scenario however I'm pretty sure that some other considerations would not be taken into account (like consistency of data when reactivating a dormant node), so I suspect that just implementing such a component would actually not make any new architecture possible, as you would get blocked by other problems which need to be solved too.. better define all expectations asap! To me this thread smells of needing the off-heap Direct Memory buffers which I suggested [long time ago] to efficiently offload internal buffers, but failing to recognise this we're pushing responsibility to an epic level complex CacheStore.. guys let's not forget that a mayor bottleneck of CacheStores today is the SPI it has to implement, we identified several limitations in the contract in the past which prevent a superior efficiency: we're working towards a mayor release now so I'd rather focus on the API changes which will make it possible to get decent performance even without changing any storage engine.. I'm pretty sure Cassandra (to pick one) doesn't scale too bad. Cheers, Sanne On 2 July 2013 10:09, Radim Vansa <rvansa(a)redhat.com&gt; wrote: _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

I went through Loaders and Stores JIRAs and I don't see anything that should handle the overall node-restart/cluster-restart scenario with DIST + local cachestore. We have two situations to happen: single-node failure followed by a restart and cluster-failure/shutdown followed by retrieval from backup/last working state. Speaking about the first situation, I personally thing that the purgeOnStartup option is conceptually wrong. Well, it's great for testing, you don't have to care about the previous state, but it's something you have to change according to the situation, and I wouldn't want to change config files because I need to start the node in a different way. For single node restart this is required (so, if the node joins a running cluster, it should always purge). For first node started I see the use case where the node is started as cache over shared persistent DB, and therefore, no purge should be executed. There may be a case where it would make sense to purge on the first node as well, but currently I don't see it. After all the app can always execute clear manually. For whole cluster restart there should be a way to startup the nodes without accepting request/preloading anything, then figure out the CH function based on the last state - it would help a lot if the store had to keep the last used consistent hash! A small complication is that all machines must be started for such restart to work properly. Currently, the shutdown would have to transfer all data to single node (when removing the nodes from the cluster one after another) in order to be able to restart properly and that can be really too much. As Sanne suggested: now is a good time to change the cache-store API, and some cache-store persistent conciousness about the segments stored would help with the startup. Some ability to perform snapshots would be also great for backups. Radim ----- Original Message ----- | From: "Sanne Grinovero" <sanne(a)infinispan.org&gt; | To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; | Sent: Tuesday, July 2, 2013 8:01:13 PM | Subject: Re: [infinispan-dev] Cachestores performance | | Thanks Erik ! exactly the kind of use-case description I'd like to collect. | | If I could summarise the main differences with my use case, it's that | you are implying a non-shared (local) CacheStore right? | | I wouldn't mind having that on my own wishlist too, but as you also | point out it's unclear how to restore the grid state from the various | pieces of stored data. In my case I'm assuming a single shared | CacheStore as that's the only scenario in which consistency in the | CacheStore isn't puzzling me too much. | | On to your recovery approach: I don't think collecting a copy of all | SST files will save you: many nodes might have older copies of the | same entry which where not cleaned up after they lost the role of | "owners" on that particular entry. | | You could even enter the messy situation of nodes which | - are owners for K1 | - store some value K1->V1 | - is no longer an owner | - K1 is deleted by other activity | - you become an owner again (no state transfer happens as the entry is gone) | - you receive a GET and serve it from your CacheStore | | .. and I could make up more horror stories like this ;-) | Bottom-line: I would not advice DIST + local cachestores as long as | your app has some peculiarity (like never reusing a key) which avoids | such situations. At least, this is why I can't use it. | This is just an example of the kinds of limitations I would like to | collect on a per-configuration basis. | | Sanne | | | On 2 July 2013 18:39, Erik Salter <an1310(a)hotmail.com&gt; wrote: | > I concur with part of the below, but with a few changes: | > | > - The cache is the primary storage, similar to Sanne's case. (DIST mode) | > - My customers are not interested in extra components to the system, like | > databases or Cassandra nodes. They wonder why they can't simply use the | > existing file system on the nodes they have. | > - I'm only going to be using the filesystem to recover in the case of | > upgrades and catastrophic failures. So during normal operation, flushes | > to disk cannot impact cluster performance. | > - Most importantly, there needs to be a way, scripted or otherwise, to | > recover the keys from local storage in a DIST-mode cache. I cannot | > guarantee anything regarding node ordering, so anything about persisting | > segment info/previous CHs are out. If that means copying all LevelDB SST | > files to all nodes and restarting them, that's fine. | > | > At the executive levels of my customer, they don't see (or really care | > about) the differentiation between data grids and MySQL -- only that one | > has file-based persistence and the other doesn't. | > | > In production, we've already taken a massive outage where a unbelievable | > series of coincidences occurred to reveal a JBoss AS bug that ended up | > deadlocking all threads in the cluster and we had to restart all nodes. | > And I'm sure it'll happen again. | > | > Hope this offers some user perspective. | > | > Erik | > | > -----Original Message----- | > From: infinispan-dev-bounces(a)lists.jboss.org | > [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Sanne | > Grinovero | > Sent: Tuesday, July 02, 2013 8:47 AM | > To: infinispan -Dev List | > Subject: Re: [infinispan-dev] Cachestores performance | > | > It would be nice to have a deck of "cheat sheets" on the expected use cases | > and guarantees: to me it looks like everyone is trying to solve a | > different problem / having a different problem in mind. | > | > My own take on it: | > | > Scenario #1 | > I'll primarily use Infinispan with DIST, and I don't care much for other | > options. Reliability is guaranteed via numOwners>1, NOT by persisting to | > disk: if a node fails, I kill the VM (the machine, not the Java process) | > and start new ones to compensate: I'm assuming cloud nodes, so it's likely | > that when a failed node is gone, the disk is gone as well, with all the | > carefully stored data. | > I will use Infinispan primarily to absorb write spikes - so a "synch flush" | > is no good for me - and to boost read performance by as much memory I can | > throw at it. | > CacheStore is used for two reasons: | > - overflow (LIRS+passivation) for then the memory is not enough | > - clean shutdown: you can think of it as a way to be able to upgrade some | > component in the system (Infinispan or my own); I would expect some kind | > of "JMX flush" operation to do a clean shutdown without data loss. | > | > Given such a scenario, I am not interested at all in synchronous storage. | > Before we commit into a design which is basically assuming the need for | > synchronous storage guarantees, I'd like to understand what kind of use | > case it's aiming to solve. | > | > It would be great to document each such use case and put down a table of | > things which can be expected, which features should not be expected (be | > very explicit on the limitations), and how basic operations are expected | > to be performed in the scenario: like how do you do a rolling upgrade in | > Scenario 1# ? How do you do a backup? And of course some configurations & | > code examples. | > | > Only then we would be able to pick a design (or multiple ones); for my use | > case the proposal from Karsten seems excellent, so I'm wondering why I | > should be looking for alternatives, and wondering why everyone is still | > wasting time on different discussions :-D | > | > I'm pretty sure there is people looking forward for a synch-CacheStore | > too: if you could nail down such a scenario however I'm pretty sure that | > some other considerations would not be taken into account (like | > consistency of data when reactivating a dormant node), so I suspect that | > just implementing such a component would actually not make any new | > architecture possible, as you would get blocked by other problems which | > need to be solved too.. better define all expectations asap! | > | > To me this thread smells of needing the off-heap Direct Memory buffers | > which I suggested [long time ago] to efficiently offload internal buffers, | > but failing to recognise this we're pushing responsibility to an epic | > level complex CacheStore.. guys let's not forget that a mayor bottleneck | > of CacheStores today is the SPI it has to implement, we identified several | > limitations in the contract in the past which prevent a superior | > efficiency: we're working towards a mayor release now so I'd rather focus | > on the API changes which will make it possible to get decent performance | > even without changing any storage engine.. | > I'm pretty sure Cassandra (to pick one) doesn't scale too bad. | > | > Cheers, | > Sanne | > | > | > | > On 2 July 2013 10:09, Radim Vansa <rvansa(a)redhat.com&gt; wrote: | >> Hi, | >> | >> I've written down this proposal for the implementation of new cache store. | >> | >> https://community.jboss.org/wiki/BrnoCacheStoreDesignProposal | >> | >> WDYT? | >> | >> Radim | >> | >> ----- Original Message ----- | >> | From: "Radim Vansa" <rvansa(a)redhat.com&gt; | >> | To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; | >> | Sent: Thursday, June 27, 2013 2:37:43 PM | >> | Subject: Re: [infinispan-dev] Cachestores performance | >> | | >> | | >> | | >> | ----- Original Message ----- | >> | | From: "Galder Zamarreño" <galder(a)redhat.com&gt; | >> | | To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; | >> | | Sent: Thursday, June 27, 2013 1:52:11 PM | >> | | Subject: Re: [infinispan-dev] Cachestores performance | >> | | | >> | | > As for Karsten's FCS implementation, I too have issues with the | >> | | > key set and value offsets being solely in memory. However I | >> | | > think that could be improved by storing only a certain number of | >> | | > keys/offsets in memory, and flushing the rest to disk again into | >> | | > an index file. | >> | | | >> | | ^ Karsten's implementation makes this relatively easy to achieve | >> | | because it already keeps this mapping in a LinkedHashMap (with a | >> | | given max entries limit [1]) assuming removeEldestEntry() is | >> | | overriden to flush to disk older entries. Some extra logic would | >> | | be needed to bring back data from the disk too… but your suggestion | >> | | below is also quite interesting... | >> | | >> | I certainly wouldn't call this easy task, because the most | >> | problematic part is what we will do when the whole entry (both key | >> | and value) are gone from memory and we want to read them - that | >> | requires keeping some searchable structure on-disk. And that's the hard | >> | stuff. | >> | | >> | | | >> | | > I believe LevelDB follows a similar design, but I think | >> | | > Karsten's FCS will perform better than LevelDB since it doesn't | >> | | > attempt to maintain a sorted structure on disk. | >> | | | >> | | ^ In-memory, the structure can optionally be ordered if it's bound | >> | | [1], otherwise it's just a normal map. How would be store it at the | >> | | disk level? | >> | | B+ tree with hashes of keys and then linked lists? | >> | | >> | Before choosing "I love B#@& trees, let's use B#@& trees!", I'd find | >> | out what requirements do we have for the structure. I believe that | >> | the index itself should not be considered persistent, as it can be | >> | rebuilt when preloading the data (sequentially reading the data is | >> | fast, therefore we can afford do this indexing preload), the reason | >> | of the index being on-disk is that we don't have enough memory to | >> | store all keys, or even key hashes. Therefore it does not have to be | >> | updated synchronously with the writes. It should be mostly | >> | read-optimized then, because that's the thing where we need synchronous | >> | access to this structure. | >> | | >> | | | >> | | > One approach to maintaining keys and offsets in memory could be | >> | | > a WeakReference that points to the key stored in the in-memory | >> | | > DataContainer. Once evicted from the DC, then the CacheStore | >> | | > impl would need to fetch the key again from the index file | >> | | > before looking up the value in the actual store. | >> | | | >> | | ^ Hmmm, interesting idea… has the potential to safe the memory | >> | | space by not having to keep that extra data structure in the cache | >> | | store. | >> | | >> | You mean to mix the DataContainer with xCacheEntry implementation | >> | and the cache store implementation? Is that possible from design | >> | perspective? | >> | Speaking about different kind of references, we may even optimize | >> | not-well-tuned eviction by SoftReferences, so that even if the entry | >> | was evicted from main DataContainer, we'd keep the value referenced | >> | from the cache-store (and this does not have to be loaded from disk | >> | if referenced before garbage collection). But such thought may be | >> | premature optimization. | >> | For having eviction managed in relation with GC we should rather | >> | combine this with PhantomReferences, where entries would be written | >> | to cache upon finalization. | >> | | >> | | | >> | | > This way we have hot items always in memory, semi-hot items with | >> | | > offsets in memory and values on disk, and cold items needing to | >> | | > be read off disk entirely (both offset and value). Also for | >> | | > write-through and write-behind, as long as the item is hot or | >> | | > warm (key and offset in memory), writing will be pretty fast. | >> | | | >> | | My worry about Karsten's impl is writing actually. If you look at | >> | | the last performance numbers in [2], where we see the performance | >> | | difference of force=true and force=false in Karsten's cache store | >> | | compared with LevelDB JNI, you see that force=false is fastest, | >> | | then JNI LevelDB, and the force=true. Me wonders what kind of | >> | | write guarantees LevelDB JNI provides (and the JAVA version)... | >> | | >> | Just for clarification: the fast implementation is without force at | >> | all, the slower is with force(false). Force(true) means updating | >> | metadata (such as access times?) which is not required for cache-store. | >> | But the numbers suggest that the random access with syncing is | >> | really not a good option, and that we should rather use the | >> | temporary append-only log, which would be persisted into structured | >> | DB by different thread (as LevelDB does, I suppose). | >> | | >> | Thinking about all the levels and cache structures optimizing the | >> | read access, I can see four levels of search structures: key + value | >> | (usual DataContainer), key + offset, hash + offset, all on disk. The | >> | "hash + offset" may seem superflous but for some use-cases with big | >> | keys it may be worth sparing a few disk look-ups. | >> | | >> | Radim | >> | | >> | | > | >> | | > On 27 Jun 2013, at 10:33, Radim Vansa <rvansa(a)redhat.com&gt; wrote: | >> | | > | >> | | >> Oops, by the cache store I mean the previously-superfast | >> | | >> KarstenFileCacheStore implementation. | >> | | >> | >> | | >> ----- Original Message ----- | >> | | >> | From: "Radim Vansa" <rvansa(a)redhat.com&gt; | >> | | >> | To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; | >> | | >> | Sent: Thursday, June 27, 2013 11:30:53 AM | >> | | >> | Subject: Re: [infinispan-dev] Cachestores performance | >> | | >> | | >> | | >> | I have added FileChannel.force(false) flushes after all write | >> | | >> | operations in the cache store, and now the comparison is also | >> | | >> | updated with these values. | >> | | >> | | >> | | >> | Radim | >> | | >> | | >> | | >> | ----- Original Message ----- | >> | | >> | | From: "Radim Vansa" <rvansa(a)redhat.com&gt; | >> | | >> | | To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; | >> | | >> | | Sent: Thursday, June 27, 2013 8:54:25 AM | >> | | >> | | Subject: Re: [infinispan-dev] Cachestores performance | >> | | >> | | | >> | | >> | | Yep, write-through. LevelDB JAVA used FileChannelTable | >> | | >> | | implementation (-Dleveldb.mmap), because Mmaping is not | >> | | >> | | implemented very well and causes JVM crashes (I believe | >> | | >> | | it's because of calling non-public API via reflection | >> | | >> | | - I've found post from the Oracle JVM guys discouraging the | >> | | >> | | particular trick it uses). After writing the record to the | >> | | >> | | log, it calls FileChannel.force(true), therefore, it should | >> | | >> | | be really on the disc by that moment. | >> | | >> | | I have not looked into the JNI implementation but I expect the | >> | | >> | | same. | >> | | >> | | | >> | | >> | | By the way, I have updated [1] with numbers when running on | >> | | >> | | more data | >> | | >> | | (2 GB | >> | | >> | | instead of 100 MB). I won't retype it here, so look there. | >> | | >> | | The performance is much lower. | >> | | >> | | I may try also increase JVM heap size and try with a bit | >> | | >> | | more data yet. | >> | | >> | | | >> | | >> | | Radim | >> | | >> | | | >> | | >> | | [1] https://community.jboss.org/wiki/FileCacheStoreRedesign | >> | | >> | | | >> | | >> | | ----- Original Message ----- | >> | | >> | | | From: "Erik Salter" <an1310(a)hotmail.com&gt; | >> | | >> | | | To: "infinispan -Dev List" | >> | | >> | | | <infinispan-dev(a)lists.jboss.org&gt; | >> | | >> | | | Sent: Wednesday, June 26, 2013 7:40:19 PM | >> | | >> | | | Subject: Re: [infinispan-dev] Cachestores performance | >> | | >> | | | | >> | | >> | | | These were write-through cache stores, right? And with | >> | | >> | | | LevelDB, this was through to the database file itself? | >> | | >> | | | | >> | | >> | | | Erik | >> | | >> | | | | >> | | >> | | | -----Original Message----- | >> | | >> | | | From: infinispan-dev-bounces(a)lists.jboss.org | >> | | >> | | | [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf | >> | | >> | | | Of Radim Vansa | >> | | >> | | | Sent: Wednesday, June 26, 2013 11:24 AM | >> | | >> | | | To: infinispan -Dev List | >> | | >> | | | Subject: [infinispan-dev] Cachestores performance | >> | | >> | | | | >> | | >> | | | Hi all, | >> | | >> | | | | >> | | >> | | | according to [1] I've created the comparison of | >> | | >> | | | performance in stress-tests. | >> | | >> | | | | >> | | >> | | | All setups used local-cache, benchmark was executed via | >> | | >> | | | Radargun (actually version not merged into master yet | >> | | >> | | | [2]). I've used 4 nodes just to get more data - each | >> | | >> | | | slave was absolutely independent of the others. | >> | | >> | | | | >> | | >> | | | First test was preloading performance - the cache started | >> | | >> | | | and tried to load 1GB of data from harddrive. Without | >> | | >> | | | cachestore the startup takes about 2 | >> | | >> | | | - | >> | | >> | | | 4 | >> | | >> | | | seconds, average numbers for the cachestores are below: | >> | | >> | | | | >> | | >> | | | FileCacheStore: 9.8 s | >> | | >> | | | KarstenFileCacheStore: 14 s | >> | | >> | | | LevelDB-JAVA impl.: 12.3 s | >> | | >> | | | LevelDB-JNI impl.: 12.9 s | >> | | >> | | | | >> | | >> | | | IMO nothing special, all times seem affordable. We don't | >> | | >> | | | benchmark exactly storing the data into the cachestore, | >> | | >> | | | here FileCacheStore took about | >> | | >> | | | 44 | >> | | >> | | | minutes, while Karsten about 38 seconds, LevelDB-JAVA 4 | >> | | >> | | | minutes and LevelDB-JNI 96 seconds. The units are right, | >> | | >> | | | it's minutes compared to seconds. But we all know that | >> | | >> | | | FileCacheStore is bloody slow. | >> | | >> | | | | >> | | >> | | | Second test is stress test (5 minutes, preceded by 2 | >> | | >> | | | minute | >> | | >> | | | warmup) | >> | | >> | | | where | >> | | >> | | | each of 10 threads works on 10k entries with 1kB values | >> | | >> | | | (~100 MB in total). | >> | | >> | | | 20 % writes, 80 % reads, as usual. No eviction is | >> | | >> | | | configured, therefore the cache-store works as a | >> | | >> | | | persistent storage only for case of crash. | >> | | >> | | | | >> | | >> | | | FileCacheStore: 3.1M reads/s 112 writes/s // on | >> | | >> | | | one | >> | | >> | | | node | >> | | >> | | | the | >> | | >> | | | performance was only 2.96M reads/s 75 writes/s | >> | | >> | | | KarstenFileCacheStore: 9.2M reads/s 226k writes/s // | >> | | >> | | | yikes! | >> | | >> | | | LevelDB-JAVA impl.: 3.9M reads/s 5100 writes/s | >> | | >> | | | LevelDB-JNI impl.: 6.6M reads/s 14k writes/s // on | >> | | >> | | | one | >> | | >> | | | node | >> | | >> | | | the | >> | | >> | | | performance was 3.9M/8.3k - about half of the others | >> | | >> | | | Without cache store: 15.5M reads/s 4.4M writes/s | >> | | >> | | | | >> | | >> | | | Karsten implementation pretty rules here for two reasons. | >> | | >> | | | First of all, it does not flush the data (it calls only | >> | | >> | | | RandomAccessFile.write()). | >> | | >> | | | Other | >> | | >> | | | cheat is that it stores in-memory the keys and offsets of | >> | | >> | | | data values in the database file. Therefore, it's | >> | | >> | | | definitely the best choice for this scenario, but it does | >> | | >> | | | not allow to scale the cache-store, especially in cases | >> | | >> | | | where the keys are big and values small. However, this | >> | | >> | | | performance boost is definitely worth checking - I could | >> | | >> | | | think of caching the disk offsets in memory and querying | >> | | >> | | | persistent index only in case of missing record, with | >> | | >> | | | part of the persistent index flushed asynchronously (the | >> | | >> | | | index can be always rebuilt during the preloading for | >> | | >> | | | case of crash). | >> | | >> | | | | >> | | >> | | | The third test should have tested the scenario with more | >> | | >> | | | data to be stored than memory - therefore, the stressors | >> | | >> | | | operated on 100k entries | >> | | >> | | | (~100 MB | >> | | >> | | | of | >> | | >> | | | data) but eviction was set to 10k entries (9216 entries | >> | | >> | | | ended up in memory after the test has ended). | >> | | >> | | | | >> | | >> | | | FileCacheStore: 750 reads/s 285 writes/s | >> | | >> | | | // | >> | | >> | | | one | >> | | >> | | | node | >> | | >> | | | had | >> | | >> | | | only 524 reads and 213 writes per second | >> | | >> | | | KarstenFileCacheStore: 458k reads/s 137k writes/s | >> | | >> | | | LevelDB-JAVA impl.: 21k reads/s 9k writes/s | >> | | >> | | | // a | >> | | >> | | | bit | >> | | >> | | | varying | >> | | >> | | | performance | >> | | >> | | | LevelDB-JNI impl.: 13k-46k reads/s 6.6k-15.2k writes/s | >> | | >> | | | // | >> | | >> | | | the | >> | | >> | | | performance varied a lot! | >> | | >> | | | | >> | | >> | | | 100 MB of data is not much, but it takes so long to push | >> | | >> | | | it into FileCacheStore that I won't use more unless we | >> | | >> | | | exclude this loser from the comparison :) | >> | | >> | | | | >> | | >> | | | Radim | >> | | >> | | | | >> | | >> | | | [1] | >> | | >> | | | https://community.jboss.org/wiki/FileCacheStoreRedesign | >> | | >> | | | [2] https://github.com/rvansa/radargun/tree/t_keygen | >> | | >> | | | | >> | | >> | | | --------------------------------------------------------- | >> | | >> | | | -- | >> | | >> | | | Radim Vansa | >> | | >> | | | Quality Assurance Engineer JBoss Datagrid tel. | >> | | >> | | | +420532294559 ext. 62559 | >> | | >> | | | | >> | | >> | | | Red Hat Czech, s.r.o. | >> | | >> | | | Brno, Purkyňova 99/71, PSČ 612 45 Czech Republic | >> | | >> | | | | >> | | >> | | | | >> | | >> | | | _______________________________________________ | >> | | >> | | | infinispan-dev mailing list | >> | | >> | | | infinispan-dev(a)lists.jboss.org | >> | | >> | | | https://lists.jboss.org/mailman/listinfo/infinispan-dev | >> | | >> | | | | >> | | >> | | | | >> | | >> | | | _______________________________________________ | >> | | >> | | | infinispan-dev mailing list | >> | | >> | | | infinispan-dev(a)lists.jboss.org | >> | | >> | | | https://lists.jboss.org/mailman/listinfo/infinispan-dev | >> | | >> | | | >> | | >> | | _______________________________________________ | >> | | >> | | infinispan-dev mailing list | >> | | >> | | infinispan-dev(a)lists.jboss.org | >> | | >> | | https://lists.jboss.org/mailman/listinfo/infinispan-dev | >> | | >> | | >> | | >> | _______________________________________________ | >> | | >> | infinispan-dev mailing list | >> | | >> | infinispan-dev(a)lists.jboss.org | >> | | >> | https://lists.jboss.org/mailman/listinfo/infinispan-dev | >> | | >> | >> | | >> _______________________________________________ | >> | | >> infinispan-dev mailing list | >> | | >> infinispan-dev(a)lists.jboss.org | >> | | >> https://lists.jboss.org/mailman/listinfo/infinispan-dev | >> | | > | >> | | > -- | >> | | > Manik Surtani | >> | | > manik(a)jboss.org | >> | | > twitter.com/maniksurtani | >> | | > | >> | | > Platform Architect, JBoss Data Grid | >> | | > http://red.ht/data-grid | >> | | > | >> | | > | >> | | > _______________________________________________ | >> | | > infinispan-dev mailing list | >> | | > infinispan-dev(a)lists.jboss.org | >> | | > https://lists.jboss.org/mailman/listinfo/infinispan-dev | >> | | | >> | | | >> | | -- | >> | | Galder Zamarreño | >> | | galder(a)redhat.com | >> | | twitter.com/galderz | >> | | | >> | | Project Lead, Escalante | >> | | http://escalante.io | >> | | | >> | | Engineer, Infinispan | >> | | http://infinispan.org | >> | | | >> | | | >> | | _______________________________________________ | >> | | infinispan-dev mailing list | >> | | infinispan-dev(a)lists.jboss.org | >> | | https://lists.jboss.org/mailman/listinfo/infinispan-dev | >> | | >> | _______________________________________________ | >> | infinispan-dev mailing list | >> | infinispan-dev(a)lists.jboss.org | >> | https://lists.jboss.org/mailman/listinfo/infinispan-dev | >> | >> _______________________________________________ | >> infinispan-dev mailing list | >> infinispan-dev(a)lists.jboss.org | >> https://lists.jboss.org/mailman/listinfo/infinispan-dev | > | > _______________________________________________ | > infinispan-dev mailing list | > infinispan-dev(a)lists.jboss.org | > https://lists.jboss.org/mailman/listinfo/infinispan-dev | > | > | > _______________________________________________ | > infinispan-dev mailing list | > infinispan-dev(a)lists.jboss.org | > https://lists.jboss.org/mailman/listinfo/infinispan-dev | | _______________________________________________ | infinispan-dev mailing list | infinispan-dev(a)lists.jboss.org | https://lists.jboss.org/mailman/listinfo/infinispan-dev

On Jul 8, 2013, at 3:03 PM, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: +1000

Hi Galder, you make many interesting points but I am not interested in discussing my specific design ideas in detail, I just sketched that quickly as an example of requirements description. My intervention in this thread is all about demanding what the use case is for such a synchronous cachestore. It is my understanding that lots of people are currently discussing how to best make a synchronous cachestore mostly efficient, but I yet have to see what the requirements are. My use case - although expecting an asynchronous one - is just an example of how I would like to see the general architecture described first before we waste time on a useless component. I am not stating that nobody wants a strictly synchronous CacheStore, but I would like to challenge the arguments who made someone (who?) ask for this, as I believe there are many other things that would need to be addressed. Therefore, my suggestion for describing the main use cases we expect to support is not off-topic at all, it's the first thing any engineer would have requested, and I would not spend a minute more of our engineers in coding without a clear description of the expected architecture, expected reliability, expected operations to be used. Sanne On 3 July 2013 14:59, Galder Zamarreño <galder(a)redhat.com&gt; wrote:

On 4 July 2013 13:56, Manik Surtani <msurtani(a)redhat.com&gt; wrote: +1 and sorry for the rant. But still even after all my whining nobody explained why we want to prioritise synchronous cachestore :-) My point is not being against doing it, but to identify eagerly what else is needed to accomodate such a use case: having the CacheStore implementation is not enough to make such a configuration usable. I wasn't serious on Cassandra being the ultimate solution, as I said I just sketched an example use case. I thought we had agreed to hold back until the mayor version change.. !? That's the better solution, would be sad to waste time on temporary compromises. Sanne

On 4 Jul 2013, at 14:21, Sanne Grinovero <sanne(a)infinispan.org&gt; wrote: Again, start a wiki page with your use case. We'll add more use cases and flesh out details. This is useful information for the community anyway, pointing to valid configuration, architecture, workarounds, etc. Sadly, not with this major version change. The roadmap of Infinispan 6.0 is far too tight to accommodate re-achitecting lots of core bits. Also baselining on JDK 7 is out for now. (Before that last comment sparks off a tangent, see [1] and [2]). - M [1] http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-Base... [2] http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-Zero... -- Manik Surtani manik(a)jboss.org twitter.com/maniksurtani Platform Architect, JBoss Data Grid http://red.ht/data-grid