[infinispan-dev] RadargunTPCC benchmark review

[Paolo Romano]

On 11/8/11 8:16 AM, Galder Zamarreño wrote:
> On Nov 7, 2011, at 5:27 PM, Sebastiano Peluso wrote:
>
>>> By the way, one last thing:
>>>
>>> On Nov 7, 2011, at 1:14 PM, Sebastiano Peluso wrote:
>>>
>>>
>>>> Hi Galder,
>>>>
>>>> thank you for the review. My answers are below inline.
>>>>
>>>> Il 04/11/11 11:02, Galder Zamarreño ha scritto:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I had the chance to look at the TPCC radargun plug-in you've built (
>>>>> https://github.com/sebastianopeluso/radargun.git
>>>>>    - branch TPCC) and I had some comments to make:
>>>>>
>>>>> 1. Instead of using divisions for converting time units, it'd be better to use TimeUnit conversions to make the code more readable.
>>>>>
>>>> Ok. I will adopt TimeUnit.
>>>>
>>>>> 2. To make the test run faster, it'd be interesting to transform Order...etc to use Infinispan Externalizers but that's not portable to other frameworks. So, implementing Externalizable could be a good middle ground.
>>>>>
>>>> Thank you for the suggestion. I will implement Externalizable interface
>>>> for the TPC-C domain objects.
>>>>
>>>>
>>>>> 3. Is TpccPopulationStage executed in each slave? Or in single slave? Seems like it's only executed in one of the slaves? If so that's fine cos it'd replicate to all right?
>>>>>
>>>> This is partially true. Each slave populates a slice of the data
>>>> container in a such a way that, for each pair of distinct slaves s1, s2,
>>>> if s1 populates the slice c1 and s2 populates the slice c2, then the
>>>> intersection between c1 and c2 is empty.
>>>> In this way we try to parallelize as much as possible cache loading, we
>>>> don't perform population in a transactional context and if the cache is
>>>> fully replicated then at the end of the population stage each node will
>>>> store all loaded data.
>>>>
>>>>> 4. Has this TPCC benchmark code been run with Infinispan 4?
>>>>>
>>>> I ran it also on top of Infinispan 5.
>>>>
>>> What kind of results did you get?
>>>
>>> Did you try the latest Infinispan 5.1 snapshot too?
>>>
>> We tested only 5.0.0, at the time in which we were working on the Atomic Multicast based distribution scheme. At this link http://www.gsd.inesc-id.pt/~romanop/files/papers/prdc2011.pdf  the paper that reports the results. This has been accepted for publication at the IEEE 17th Pacific Rim International Symposium on Dependable Computing (PRDC’11), Pasadena, California, Dec. 2011.
> Ok, i'll look into that.
>
>> Note that the results published in that paper are based on a slightly modified version of the benchmark that we are contributing to RG, in which we "adapted" the benchmark to reduce contention. Specifically we commented out the line 121 of class  org.radargun.tpcc.transaction.PaymentTransaction [1], which would otherwise force each transaction of type Payment (accessing the same warehouse) to conflict with each other.
> Why did you do that?

This was not intentional: we had been playing a couple of months earlier 
with the benchmark's code to understand its behavior and ended up 
commenting out that line... and when, later on, we used it for the 
evaluation of that replication protocol, we simply forgot about it.... 
We've realized this only after the recent code revision of the benchmark.

As a side note, I expect that the speed-up achieved via the Atomic 
Multicast scheme would have been even larger in a higher contention 
scenario.

>
>> About adherence to the spec, which is something you had previously asked me. We do not claim that is compliant to the specification. We simply based this implementation on a previously existing open source implementation of the benchmark (http://jtpcc.sourceforge.net/), and stripped out SQL queries to work directly with ISPN's key/value data model. Also, we did not implement transaction Delivery and Stock-Level which were quite complex to implement without query support. In future, in fact, it would be interesting to extend them using the new ISPN query support.
>>
>> Given all the above premises, I think it is fair to claim that this benchmark generates a workload which resembles the one of the official TPC-C.
> Cool, that's valuable information.

Galder, I am not sure if you have access to the cloudtm-wiki. We have a 
dedicated page on this providing additional info about the benchmark's 
implementation. Let us know if you want an account.
>
>> About testing on 5.1: our cluster is currently being used intensively as several people in our team (me included!!) are working on a tight schedule to finalize works for a close deadline. So I cannot promise that we will be able to test it in the next week or soon, but towards the end of the month it should be possible!
>>
>> One last issue that just came to mind. Since we based our implementation on a pre-existing one, released using GPL, should we take any particular care in the header files (adding references, pointers)? I'm not an expert in these legal matters, I'd appreciate your comments on this!
> Well, depends what you did exactly. If there're classes that are pretty much copies, you should maintain the copyright they had. If there're classes based on other GPL classes, the least you can do is reference and give credit.
>
> RadarGun is not being distributed, so nothing much else to worry about AFAIK (others might be able to comment further)
>
>>
>> Cheers,
>>
>>      Sebastiano
>>
>>
>> [1] https://github.com/sebastianopeluso/radargun/blob/tpcc/framework/src/main/java/org/radargun/tpcc/transaction/PaymentTransaction.java
>>>>> Overall looks good from a RadarGun integration point of view, but I'm not familiar with the specifics of TPCC, so someone else should comment on how well the code adheres to the spec.
>>>>>
>>>>> I've noticed that RadarGun is not up to date with the latest 4.2 Infinispan version and does not have a plugin for Infinispan 5, so I'll update these two pieces.
>>>>>
>>>>> I plan to only create one plugin for Infinispan 5, which will contain latest Infinispan 5.1 beta.
>>>>>
>>>>> Cheers,
>>>>>
>>>> Thank you again.
>>>>
>>>> Cheers,
>>>>
>>>>     Sebastiano
>>>>
>>>>
>>>>
> --
> Galder Zamarreño
> Sr. Software Engineer
> Infinispan, JBoss Cache
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev