[infinispan-dev] [infinispan-internal] LevelDB performance testing

Mon Jun 24 06:46:52 EDT 2013

Putting Infinispan-Development list again to get others' thoughts...

On Jun 24, 2013, at 11:57 AM, Radim Vansa <rvansa at redhat.com> wrote:

> 
> 
> ----- Original Message -----
> | From: "Galder Zamarreño" <galder at redhat.com>
> | To: "Radim Vansa" <rvansa at redhat.com>
> | Sent: Monday, June 24, 2013 10:45:32 AM
> | Subject: Re: [infinispan-internal] LevelDB performance testing
> | 
> | Hey Radim,
> | 
> | Thanks a lot for running these tests. Comments inline...
> | 
> | On Jun 21, 2013, at 3:33 PM, Radim Vansa <rvansa at redhat.com> wrote:
> | 
> | > Hi all,
> | > 
> | > I've got some results from LevelDB performance testing.
> | > 
> | > Use case description:
> | > - writes with unique keys about 100 bytes in size, no value.
> | > - reads rare
> | > 
> | > Infinispan configuration:
> | > - eviction enabled (no passivation), 10000 entries is memory allowed (we
> | > don't need to occupy the memory if almost all operations are writes to
> | > unique keys)
> | > - synchronous mode (test have not revealed any significant performance gain
> | > using async mode) - distributed cache with 2 owners and 40 segments
> | > - write-behind cache store with 4 threads
> | 
> | ^ Hmmmm, why write-behind? If you really wanna test the performance of
> | LevelDB, and you want to check how fast it writes or deletes, how are you
> | gonna measure that if you're using an async store? Read operations, assuming
> | that data is evicted, will go all the way to the cache store, and that's
> | really the only numbers you can get out this test…
> 
> Because I want to handle stuff ASAP. I don't really care how long each operation takes, I am just interested in the throughput I can reach. And this should be best with write-behind, because we're not waiting for the data to be written down, right?

^ Indeed, but then not sure you're really getting throughput numbers based on what the cache store implementation. In other words, with this test you're testing the async store logic, to see how much throughput you can handle queuing up those modifications… unless you have a validation phase that times how long it takes for the cache store to contain all the data that is supposed to have. Then yes, you'd be measuring how long it takes for all that queued up modifications to be applied, which is a factor or many things, one of which, is the speed of the cache store implementation to apply those changes at the persistence layer.

> 
> | 
> | Assuming the levelDB cache store is not shared, why test a 4-node cluster? If
> | what you're trying to figure out is how fast LevelDB is as a unshared cache
> | store (assuming this is going to be used instead of the stock FCS…), then
> | you could have just the tests against a local cache, right?
> | 
> | > Test description:
> | > - we use 100-byte keys with key-id encoded in the first 8 bytes and the
> | > rest is filled with random bytes (equal for all keys).
> | > - value is an empty byte array
> | 
> | ^ Empty? That's not very reallistic, and you'd hardly stress how the cache
> | store deals with differently sized data. I'd say at least 1kb would be
> | minimum?
> 
> That's exactly what the customer asked for - usually we really use this 1kB, but his scenario probably uses just some dummy tombstones for most of data.

Fair enough.

> 
> | 
> | > - 99 % of requests are PUTs (without any flags such as
> | > IGNORE_RETURN_VALUES), 1 % are GETs
> | > - we do not pre-load any keys to the cache, everything starts empty and
> | > each PUT operation adds a new key
> | > - GET operation reads one of the keys that is already in the cache, with
> | > uniform distribution of probability
> | > - 4-node JDG cluster stressed from 3 machines
> | > 
> | >> From client-stress test (200 new clients added every 3 minutes) we have
> | >> found out optimal value of 3500 clients (in total) which led to about 33k
> | >> operations per second.
> | > Soak test was executed to test higher amount of entries - the test was
> | > running with all 3500 clients right after the startup, executing requests
> | > and therefore increasing the size of DB.
> | > 
> | > During first 15 minutes a stable throughput of 33k operations per second
> | > was experienced, adding 2M new keys (200 MB) every minute. iostat shows
> | > about 22 MB/s reads and 11 MB/s writes, 550 read and 130 write (2600
> | > merged) requests to disk.
> | > Then the throughput started decreasing, during following 35 minutes it
> | > dropped to 25k operations per second. No exceptions have occurred in this
> | > period. By the end iostat shows 29 MB/s reads, 10 MB/s writes (up to 750
> | > w/s (2400 wrqm/s), 115 r/s.
> | > 
> | > 
> | > After these 50 minutes of working test error messages started to appear on
> | > one of the nodes:
> | > 
> | > 08:20:07,009 WARN  [org.infinispan.loaders.decorators.AsyncStore]
> | > (AsyncStoreProcessor-testCache-0) ISPN000053: Unable to process some async
> | > modifications after 3 retries!
> | > and
> | > 08:20:09,720 ERROR
> | > [org.infinispan.interceptors.InvocationContextInterceptor]
> | > (HotRodServerWorker-84) ISPN000136: Execution error:
> | > org.infinispan.loaders.CacheLoaderException:
> | > org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException:
> | > java.io.FileNotFoundException: /tmp/leveldb/testCache/data/026662.sst (No
> | > space left on device)
> | > 
> | > At this moment LevelDB was occupying about 2 GB of 16 GB available space,
> | > there's plenty of free inodes and db directory has about 4000 files inside
> | > (1100 with some content, rest of them empty).
> | > 2 minutes later JVM with one of the nodes crashed (see
> | > https://bugzilla.redhat.com/show_bug.cgi?id=976664). The test was
> | > terminated as the cluster was practically not responding and did not met
> | > performance criteria.
> | 
> | ^ What was the performance criteria?
> 
> Sorry for bottling this up - 90 % of requests should have response time < 500 ms. This is a very liberal setting, stopping the test only when things really screw up.
> 
> Radim
> 
> | 
> | > 
> | > I believe that exact performance results after things got wrong are not
> | > necessary to be presented.
> | > 
> | > Radim
> | > 
> | > -----------------------------------------------------------
> | > Radim Vansa
> | > Quality Assurance Engineer
> | > JBoss Datagrid
> | > tel. +420532294559 ext. 62559
> | > 
> | > Red Hat Czech, s.r.o.
> | > Brno, Purkyňova 99/71, PSČ 612 45
> | > Czech Republic
> | > 
> | > 
> | 
> | 
> | --
> | Galder Zamarreño
> | galder at redhat.com
> | twitter.com/galderz
> | 
> | Project Lead, Escalante
> | http://escalante.io
> | 
> | Engineer, Infinispan
> | http://infinispan.org
> | 
> | 

--
Galder Zamarreño
galder at redhat.com
twitter.com/galderz

Project Lead, Escalante
http://escalante.io

Engineer, Infinispan
http://infinispan.org