5:16 p.m.
Hi,
I was playing with efficient ways to write to append-only log and I have some results
below.
I've used three implementations: one simply synchronizing the access and calling
force(false) after each write (by default 1kB). Second with threads cooperating - every
thread puts its data into queue and waits for a short period of time - if then its data
are still in the queue, it writes whole queue to disk, flushes it and wakes up other
waiting threads. Third implementation (actually three flavours) used one spooler thread
which polls the queue, writes as much as it can to disk, flushes and notifies waiting
threads. The flavours are in the type of waiting for the spooler to write their data -
either spinning on volatile counter, checking this counter with yielding and waiting via
Object.wait()/Object.notify().
According to the results, the spooler variant with yielding proved as the best, with 20
threads even 10x faster than the basic variant (which simply does not scale) - as I've
experimented more, it scales even more - with 80 threads I have achieved 435k appends -
that's 14.16 MB/s. For better context, the basic variant without flushing on 80
threads would handle 1.67M appends = 54.26 MB/s, the spooler 660k appends = 21.48 MB/s
(curiously, non-flushing spooler with wait/notify would do 1.19M appends). I do not
present the whole table for higher numbers as I don't think that much threads would
really access the disk concurrently.
Note: the limit in WaitingAppendOnly says with how big queue the thread won't wait but
begins immediately writing the data. Setting this to thread count provides obviously the
best performance but in practice the thread number cannot be as easily guessed -
therefore, I provide results for exact guess, half the number and twice the number of
threads.
Note2: in SingleWriterAppendLog the limit is not that important - it just limits the
number of data written in one row before flush.
Note3: on my local machine the spinning version was slowest of the three ones, but still
faster than the simple variant.
Source code of the test is in attachment.
Radim
Executed on 8-core AMD Opteron(tm) Processor 6128
4 threads, 16988 appends ( 16988 flushes) in 30006.30 ms to SimpleAppendLog
8 threads, 17108 appends ( 17108 flushes) in 30018.71 ms to SimpleAppendLog
12 threads, 17139 appends ( 17139 flushes) in 30029.29 ms to SimpleAppendLog
16 threads, 17280 appends ( 17280 flushes) in 30033.49 ms to SimpleAppendLog
20 threads, 17511 appends ( 17511 flushes) in 30043.29 ms to SimpleAppendLog
4 threads, 37956 appends ( 9489 flushes) in 30002.46 ms to
WaitingAppendLog{limit=4, waitTime=1}
4 threads, 29536 appends ( 7384 flushes) in 30001.88 ms to
WaitingAppendLog{limit=8, waitTime=1}
4 threads, 29530 appends ( 14765 flushes) in 30001.27 ms to
WaitingAppendLog{limit=2, waitTime=1}
4 threads, 39280 appends ( 9820 flushes) in 30001.71 ms to
WaitingAppendLog{limit=4, waitTime=2}
4 threads, 23720 appends ( 5930 flushes) in 30002.85 ms to
WaitingAppendLog{limit=8, waitTime=2}
4 threads, 29302 appends ( 14651 flushes) in 30001.53 ms to
WaitingAppendLog{limit=2, waitTime=2}
4 threads, 39620 appends ( 9905 flushes) in 30004.34 ms to
WaitingAppendLog{limit=4, waitTime=4}
4 threads, 16600 appends ( 4150 flushes) in 30002.58 ms to
WaitingAppendLog{limit=8, waitTime=4}
4 threads, 29234 appends ( 14617 flushes) in 30002.93 ms to
WaitingAppendLog{limit=2, waitTime=4}
8 threads, 71457 appends ( 8933 flushes) in 30004.31 ms to
WaitingAppendLog{limit=8, waitTime=1}
8 threads, 54152 appends ( 6769 flushes) in 30003.77 ms to
WaitingAppendLog{limit=16, waitTime=1}
8 threads, 38104 appends ( 9526 flushes) in 30003.66 ms to
WaitingAppendLog{limit=4, waitTime=1}
8 threads, 70248 appends ( 8781 flushes) in 30002.34 ms to
WaitingAppendLog{limit=8, waitTime=2}
8 threads, 44240 appends ( 5530 flushes) in 30003.16 ms to
WaitingAppendLog{limit=16, waitTime=2}
8 threads, 38788 appends ( 9697 flushes) in 30002.56 ms to
WaitingAppendLog{limit=4, waitTime=2}
8 threads, 68456 appends ( 8557 flushes) in 30002.56 ms to
WaitingAppendLog{limit=8, waitTime=4}
8 threads, 32432 appends ( 4054 flushes) in 30008.19 ms to
WaitingAppendLog{limit=16, waitTime=4}
8 threads, 38164 appends ( 9541 flushes) in 30003.94 ms to
WaitingAppendLog{limit=4, waitTime=4}
12 threads, 97084 appends ( 8091 flushes) in 30005.87 ms to
WaitingAppendLog{limit=12, waitTime=1}
12 threads, 78684 appends ( 6557 flushes) in 30005.24 ms to
WaitingAppendLog{limit=24, waitTime=1}
12 threads, 51462 appends ( 8577 flushes) in 30002.70 ms to
WaitingAppendLog{limit=6, waitTime=1}
12 threads, 100200 appends ( 8350 flushes) in 30004.20 ms to
WaitingAppendLog{limit=12, waitTime=2}
12 threads, 66283 appends ( 5524 flushes) in 30005.82 ms to
WaitingAppendLog{limit=24, waitTime=2}
12 threads, 52134 appends ( 8689 flushes) in 30003.80 ms to
WaitingAppendLog{limit=6, waitTime=2}
12 threads, 95885 appends ( 7991 flushes) in 30007.94 ms to
WaitingAppendLog{limit=12, waitTime=4}
12 threads, 47700 appends ( 3975 flushes) in 30009.75 ms to
WaitingAppendLog{limit=24, waitTime=4}
12 threads, 51822 appends ( 8637 flushes) in 30003.71 ms to
WaitingAppendLog{limit=6, waitTime=4}
16 threads, 126192 appends ( 7887 flushes) in 30005.76 ms to
WaitingAppendLog{limit=16, waitTime=1}
16 threads, 104800 appends ( 6550 flushes) in 30002.69 ms to
WaitingAppendLog{limit=32, waitTime=1}
16 threads, 68168 appends ( 8521 flushes) in 30008.17 ms to
WaitingAppendLog{limit=8, waitTime=1}
16 threads, 119643 appends ( 7478 flushes) in 30005.76 ms to
WaitingAppendLog{limit=16, waitTime=2}
16 threads, 84576 appends ( 5286 flushes) in 30004.32 ms to
WaitingAppendLog{limit=32, waitTime=2}
16 threads, 70368 appends ( 8796 flushes) in 30003.51 ms to
WaitingAppendLog{limit=8, waitTime=2}
16 threads, 119486 appends ( 7468 flushes) in 30005.89 ms to
WaitingAppendLog{limit=16, waitTime=4}
16 threads, 62452 appends ( 3904 flushes) in 30007.32 ms to
WaitingAppendLog{limit=32, waitTime=4}
16 threads, 67912 appends ( 8489 flushes) in 30004.71 ms to
WaitingAppendLog{limit=8, waitTime=4}
20 threads, 139788 appends ( 6990 flushes) in 30005.29 ms to
WaitingAppendLog{limit=20, waitTime=1}
20 threads, 120999 appends ( 6050 flushes) in 30007.57 ms to
WaitingAppendLog{limit=40, waitTime=1}
20 threads, 80130 appends ( 8013 flushes) in 30004.85 ms to
WaitingAppendLog{limit=10, waitTime=1}
20 threads, 140020 appends ( 7001 flushes) in 30008.80 ms to
WaitingAppendLog{limit=20, waitTime=2}
20 threads, 101877 appends ( 5094 flushes) in 30006.94 ms to
WaitingAppendLog{limit=40, waitTime=2}
20 threads, 78710 appends ( 7871 flushes) in 30005.05 ms to
WaitingAppendLog{limit=10, waitTime=2}
20 threads, 150128 appends ( 7507 flushes) in 30009.89 ms to
WaitingAppendLog{limit=20, waitTime=4}
20 threads, 77120 appends ( 3856 flushes) in 30009.32 ms to
WaitingAppendLog{limit=40, waitTime=4}
20 threads, 78450 appends ( 7845 flushes) in 30007.46 ms to
WaitingAppendLog{limit=10, waitTime=4}
4 threads, 40707 appends ( 17046 flushes) in 30002.08 ms to
SingleWriterAppendLog.Spinning{limit=4}
4 threads, 42595 appends ( 20804 flushes) in 30000.96 ms to
SingleWriterAppendLog.Yielding{limit=4}
4 threads, 28979 appends ( 14492 flushes) in 30001.37 ms to
SingleWriterAppendLog.Waiting{limit=4}
8 threads, 6252 appends ( 1255 flushes) in 30011.34 ms to
SingleWriterAppendLog.Spinning{limit=8}
8 threads, 85144 appends ( 17859 flushes) in 30002.90 ms to
SingleWriterAppendLog.Yielding{limit=8}
8 threads, 38323 appends ( 9583 flushes) in 30006.94 ms to
SingleWriterAppendLog.Waiting{limit=8}
12 threads, 7241 appends ( 880 flushes) in 30026.81 ms to
SingleWriterAppendLog.Spinning{limit=12}
12 threads, 126233 appends ( 17027 flushes) in 30003.59 ms to
SingleWriterAppendLog.Yielding{limit=12}
12 threads, 55867 appends ( 9314 flushes) in 30004.56 ms to
SingleWriterAppendLog.Waiting{limit=12}
16 threads, 8956 appends ( 895 flushes) in 30021.69 ms to
SingleWriterAppendLog.Spinning{limit=16}
16 threads, 158579 appends ( 15380 flushes) in 30002.46 ms to
SingleWriterAppendLog.Yielding{limit=16}
16 threads, 82333 appends ( 10293 flushes) in 30007.82 ms to
SingleWriterAppendLog.Waiting{limit=16}
20 threads, 10037 appends ( 862 flushes) in 30008.67 ms to
SingleWriterAppendLog.Spinning{limit=20}
20 threads, 187211 appends ( 14644 flushes) in 30006.72 ms to
SingleWriterAppendLog.Yielding{limit=20}
20 threads, 116620 appends ( 11664 flushes) in 30007.56 ms to
SingleWriterAppendLog.Waiting{limit=20}
-----------------------------------------------------------
Radim Vansa
Quality Assurance Engineer
JBoss Datagrid
tel. +420532294559 ext. 62559
Red Hat Czech, s.r.o.
Brno, Purkyňova 99/71, PSČ 612 45
Czech Republic