[infinispan-dev] Improvements for SessionSimulatorTest in benchmark framework

[Mircea Markus]

Galder Zamarreno wrote:
>
>
> Manik Surtani wrote:
>>
>> On 2 Jun 2009, at 11:40, Galder Zamarreno wrote:
>>
>>> Hi all,
>>>
>>> Over the weekend, I've been reading the JCIP and I've come up with a 
>>> list of improvements that I'm planning to implement for 
>>> SessionSimulatorTest:
>>>
>>> - Use a CyclicBarrier to control read/write tasks start when all 
>>> threads are already up and running. Otherwise some taks could have 
>>> head start and begin before other tasks due to scheduling and this 
>>> could affect the results that we see.
>>
>> +1
+1
>>
>>> - When measuring throughpuyt, we get more accurate measures by 
>>> timing the entire run and then dividing it by the number of 
>>> operations to get a per operation time. So, I'd suggest doing this 
>>> rather than measuring individual ops time. We can do this very 
>>> easily using CyclicBarriers as shown in JCIP.
>>
>> True, but the problem with this is that all fwk overhead will be 
>> included in the results - such as time to generate random data, etc.
>
> No, it shouldn't. If generating random data is put before 
> barrier.await() in the task, the start time will only be computed 
> after the barrier.await() releases, so generating random data won't be 
> included. See
>
> http://book.javanb.com/java-concurrency-in-Practice/ch12lev1sec2.html
>
> and Producer/Consumer classes in 
> http://book.javanb.com/java-concurrency-in-Practice/ch12lev1sec1.html 
> (they're like our Read and Write tasks,)
>
>>
>>> - Currently, SessionSimulatorTest does not use the result of 
>>> cache.get() which is a bit unrealistic and could lead to the VM 
>>> optimizing the return path of the value. A recommended technique for 
>>> getting around this is recommended in the book.
>>
>> +1
>>
>>> - Finally, before warmup begins, we should add a check that the view 
>>> of the cluster is formed of the number of nodes that we're testing.
>>
>> Isn't this already there?  Mircea?
>
> It checks that the number of nodes is there connecting to them to 
> colate the results but does not check the cluster view itself. That's 
> how I was able to run the tests even though each node was a single 
> node on its own.
Hmm, it does a put on each node and expects other caches to replicate on 
the same node(at least one, if partialReplication is true). Each node 
running on itself scenario should be catch by this test, as it only 
counts the number of different nodes that replicated here, excluding 
self. Only scenario it does not catch is when two partitions are created 
- so that some replication exists. I haven't run through such a problem, 
though. Re:checking against cluster view itslef, that is possible for 
solutions build on top of jgroups (JBC, infinispan) but it would be more 
difficult to implement for coherene, teracotta etc
>
>>
>>> - Add an option to indicate the stress test timeout (currently 
>>> defaulting to 10 mins). This could be handy when profiling which 
>>> leads to longer execution times.
>>
>> +1
>>
>> Cheers
>> Manik
>>
>>
>>> Thoughts?
>>> -- 
>>> Galder Zamarreño
>>> Sr. Software Maintenance Engineer
>>> JBoss, a division of Red Hat
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> -- 
>> Manik Surtani
>> manik at jboss.org
>> Lead, Infinispan
>> Lead, JBoss Cache
>> http://www.infinispan.org
>> http://www.jbosscache.org
>>
>>
>>
>>
>