[infinispan-dev] [infinispan-internal] LevelDB performance testing

[Radim Vansa]

----- Original Message -----
| From: "Galder Zamarreño" <galder at redhat.com>
| To: "Radim Vansa" <rvansa at redhat.com>
| Cc: "infinispan -Dev List" <infinispan-dev at lists.jboss.org>
| Sent: Monday, June 24, 2013 12:46:52 PM
| Subject: Re: [infinispan-internal] LevelDB performance testing
| 
| Putting Infinispan-Development list again to get others' thoughts...
| 
| On Jun 24, 2013, at 11:57 AM, Radim Vansa <rvansa at redhat.com> wrote:
| 
| > 
| > 
| > ----- Original Message -----
| > | From: "Galder Zamarreño" <galder at redhat.com>
| > | To: "Radim Vansa" <rvansa at redhat.com>
| > | Sent: Monday, June 24, 2013 10:45:32 AM
| > | Subject: Re: [infinispan-internal] LevelDB performance testing
| > | 
| > | Hey Radim,
| > | 
| > | Thanks a lot for running these tests. Comments inline...
| > | 
| > | On Jun 21, 2013, at 3:33 PM, Radim Vansa <rvansa at redhat.com> wrote:
| > | 
| > | > Hi all,
| > | > 
| > | > I've got some results from LevelDB performance testing.
| > | > 
| > | > Use case description:
| > | > - writes with unique keys about 100 bytes in size, no value.
| > | > - reads rare
| > | > 
| > | > Infinispan configuration:
| > | > - eviction enabled (no passivation), 10000 entries is memory allowed
| > | > (we
| > | > don't need to occupy the memory if almost all operations are writes to
| > | > unique keys)
| > | > - synchronous mode (test have not revealed any significant performance
| > | > gain
| > | > using async mode) - distributed cache with 2 owners and 40 segments
| > | > - write-behind cache store with 4 threads
| > | 
| > | ^ Hmmmm, why write-behind? If you really wanna test the performance of
| > | LevelDB, and you want to check how fast it writes or deletes, how are you
| > | gonna measure that if you're using an async store? Read operations,
| > | assuming
| > | that data is evicted, will go all the way to the cache store, and that's
| > | really the only numbers you can get out this test…
| > 
| > Because I want to handle stuff ASAP. I don't really care how long each
| > operation takes, I am just interested in the throughput I can reach. And
| > this should be best with write-behind, because we're not waiting for the
| > data to be written down, right?
| 
| ^ Indeed, but then not sure you're really getting throughput numbers based on
| what the cache store implementation. In other words, with this test you're
| testing the async store logic, to see how much throughput you can handle
| queuing up those modifications… unless you have a validation phase that
| times how long it takes for the cache store to contain all the data that is
| supposed to have. Then yes, you'd be measuring how long it takes for all
| that queued up modifications to be applied, which is a factor or many
| things, one of which, is the speed of the cache store implementation to
| apply those changes at the persistence layer.

I may be wrong, but isn't the async-processing queue somehow bounded? I've assumed that if we write too fast, the requests start to be discarded eventually, showing some errors in the error log. In a long test, this would prove as the validation that the data have really been written to the persistence layer. Or is this queue simply unbounded?

Radim