[infinispan-dev] RadargunTPCC benchmark review

[Sebastiano Peluso]

> By the way, one last thing:
>
> On Nov 7, 2011, at 1:14 PM, Sebastiano Peluso wrote:
>
>> Hi Galder,
>>
>> thank you for the review. My answers are below inline.
>>
>> Il 04/11/11 11:02, Galder Zamarreño ha scritto:
>>> Hi all,
>>>
>>> I had the chance to look at the TPCC radargun plug-in you've built (https://github.com/sebastianopeluso/radargun.git  - branch TPCC) and I had some comments to make:
>>>
>>> 1. Instead of using divisions for converting time units, it'd be better to use TimeUnit conversions to make the code more readable.
>> Ok. I will adopt TimeUnit.
>>> 2. To make the test run faster, it'd be interesting to transform Order...etc to use Infinispan Externalizers but that's not portable to other frameworks. So, implementing Externalizable could be a good middle ground.
>> Thank you for the suggestion. I will implement Externalizable interface
>> for the TPC-C domain objects.
>>
>>> 3. Is TpccPopulationStage executed in each slave? Or in single slave? Seems like it's only executed in one of the slaves? If so that's fine cos it'd replicate to all right?
>> This is partially true. Each slave populates a slice of the data
>> container in a such a way that, for each pair of distinct slaves s1, s2,
>> if s1 populates the slice c1 and s2 populates the slice c2, then the
>> intersection between c1 and c2 is empty.
>> In this way we try to parallelize as much as possible cache loading, we
>> don't perform population in a transactional context and if the cache is
>> fully replicated then at the end of the population stage each node will
>> store all loaded data.
>>> 4. Has this TPCC benchmark code been run with Infinispan 4?
>> I ran it also on top of Infinispan 5.
> What kind of results did you get?
>
> Did you try the latest Infinispan 5.1 snapshot too?
We tested only 5.0.0, at the time in which we were working on the Atomic 
Multicast based distribution scheme. At this link 
http://www.gsd.inesc-id.pt/~romanop/files/papers/prdc2011.pdf  the paper 
that reports the results. This has been accepted for publication at the 
IEEE 17th Pacific Rim International Symposium on Dependable Computing 
(PRDC'11), Pasadena, California, Dec. 2011.

Note that the results published in that paper are based on a slightly 
modified version of the benchmark that we are contributing to RG, in 
which we "adapted" the benchmark to reduce contention. Specifically we 
commented out the line 121 of class 
org.radargun.tpcc.transaction.PaymentTransaction [1], which would 
otherwise force each transaction of type Payment (accessing the same 
warehouse) to conflict with each other.

About adherence to the spec, which is something you had previously asked 
me. We do not claim that is compliant to the specification. We simply 
based this implementation on a previously existing open source 
implementation of the benchmark (http://jtpcc.sourceforge.net/), and 
stripped out SQL queries to work directly with ISPN's key/value data 
model. Also, we did not implement transaction Delivery and Stock-Level 
which were quite complex to implement without query support. In future, 
in fact, it would be interesting to extend them using the new ISPN query 
support.

Given all the above premises, I think it is fair to claim that this 
benchmark generates a workload which resembles the one of the official 
TPC-C.

About testing on 5.1: our cluster is currently being used intensively as 
several people in our team (me included!!) are working on a tight 
schedule to finalize works for a close deadline. So I cannot promise 
that we will be able to test it in the next week or soon, but towards 
the end of the month it should be possible!

One last issue that just came to mind. Since we based our implementation 
on a pre-existing one, released using GPL, should we take any particular 
care in the header files (adding references, pointers)? I'm not an 
expert in these legal matters, I'd appreciate your comments on this!


Cheers,

     Sebastiano


[1] 
https://github.com/sebastianopeluso/radargun/blob/tpcc/framework/src/main/java/org/radargun/tpcc/transaction/PaymentTransaction.java 

>
>>> Overall looks good from a RadarGun integration point of view, but I'm not familiar with the specifics of TPCC, so someone else should comment on how well the code adheres to the spec.
>>>
>>> I've noticed that RadarGun is not up to date with the latest 4.2 Infinispan version and does not have a plugin for Infinispan 5, so I'll update these two pieces.
>>>
>>> I plan to only create one plugin for Infinispan 5, which will contain latest Infinispan 5.1 beta.
>>>
>>> Cheers,
>> Thank you again.
>>
>> Cheers,
>>
>>     Sebastiano
>>
>>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20111107/50ef6d6b/attachment.html