[infinispan-dev] Performance gap between different value sizes and between key loactions

[Radim Vansa]

Hi JR,

thanks for those findings! I was benchmarking the dependency of achieved 
throughput based on entry size in the past, and I found the sweet spot 
on 8k values (likely because our machines had 9k MTU). Regrettably, we 
were focusing on throughput rather than on latency.

I think that the increased latency could be on the account of:
a) marshalling - this is the top suspect
b) when receiving the data from network (in JGroups), those are copied 
from the socket to buffer
c) general GC activity - with larger data flow you're about to trigger 
GC sooner

Though, I am quite surprised by such linear scaling, usually RPC latency 
or waiting for locks is the villain. Unless you set in cache 
configuration to storeAsBinary, Infinispan treats values as references 
and there should be no overhead involved.

Could you set up sampling mode profiler and check what it reports? All 
the above are just slightly educated guesses.

Radim

On 12/15/2014 01:54 PM, linjunru wrote:
>
> Hi, all:
>
> I have tested infinispan in distributed mode in terms of latency of 
> put(k,v) operation. The own_num is 1 and the key we put/write locates 
> in the same node as the put operation occurs(In the table,“1+0” 
> represents this scenario), the results indicates that the latency 
> increases as the size of the value increases. However the increments 
> seem to be a little “unreasonable” to me, because the bandwidth of the 
> memory system is quite huge, and the number of keys (10000) remains 
> the same during the experiment. So, here is the questions: which 
> operations inside infinspan have strong relatives with the size of 
> value, and why they costs so much as the size increases?
>
> We have also tested infinispan in the scenario which the key and the 
> put/write(key,value) operation reside in different nodes(we noted it 
> as “0+1”). Compared with “1+0”, “0+1” triggers network communications, 
> however, the network latency is much smaller compared to the 
> performance gas between the two scenarios. Why this situation happens? 
> For example, with a 25K bytes ping packet, the RTT is about 0.713ms 
> while performance gas between the two scenarios is about 8.4ms，what 
> operations inside infinispan used the other 7.6ms?
>
> UDP is utilized as the transport protocol, the infinispan version we 
> used is 7.0 and there are 4 nodes in the cluster, each has 10000 keys, 
> all of them have memory bigger than 32G, and all of them have xeon cpu 
> e5-2407 x2.
>
> Value size
>
> 	
>
> 250B( us)
>
> 	
>
> 2.5K( us)
>
> 	
>
> 25k(us)
>
> 	
>
> 250k(us)
>
> 	
>
> 2.5M(us)
>
> 	
>
> 25M(us)
>
> 1+0
>
> 	
>
> 463
>
> 	
>
> 726
>
> 	
>
> 3 236
>
> 	
>
> 26 560
>
> 	
>
> 354 454
>
> 	
>
> 3 979 830
>
> 0+1
>
> 	
>
> 1 807
>
> 	
>
> 2 829
>
> 	
>
> 11 635
>
> 	
>
> 87 540
>
> 	
>
> 1 035 133
>
> 	
>
> 11 653 389
>
> Thanks!
>
> Best Regards,
>
> JR
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


-- 
Radim Vansa <rvansa at redhat.com>
JBoss DataGrid QA