<div dir="ltr">I think that's a very good point Radim. But on the other hand I think we may treat it as an implementation detail. The TCK must run without Infinispan server. <div><br></div><div>I also believe it would be beneficial to define as least 2-3 levels of support, e.g. basic (only basic operations, no consistent hash support), advanced (all features). This way we could validate some more exotic clients such as Go (<a href="https://github.com/ugol/infinispan-go">https://github.com/ugol/infinispan-go</a>).</div><div><br></div><div>And in my opinion, a final version of this document go to infinispan design repository: <a href="https://github.com/infinispan/infinispan-designs">https://github.com/infinispan/infinispan-designs</a></div><div><br></div><div>Apart from that, that's huge effort and really great job!</div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Apr 11, 2017 at 5:05 PM Radim Vansa <<a href="mailto:rvansa@redhat.com">rvansa@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Since these tests use real server(s), many of them test not only the<br>
client behaviour (generating correct commands according to the<br>
protocol), but server, too. While this is practical (we need to test<br>
server somehow, too), there's nothing all the tests across languages<br>
will have physically in common and all comparison is prone to human error.<br>
<br>
If we want to test various implementations of the client, maybe it would<br>
make sense to give the clients a fake server that will have just a<br>
scenario of expected commands to receive and pre-defined responses. We<br>
could use audit log to generate such scenario based on the actual Java<br>
tests.<br>
<br>
But then we'd have to test the actual behaviour on server, and we'd need<br>
a way to issue the commands.<br>
<br>
Just my 2c<br>
<br>
Radim<br>
<br>
On 04/11/2017 02:33 PM, Martin Gencur wrote:<br>
> Hello all,<br>
> we have been working on <a href="https://issues.jboss.org/browse/ISPN-7120" rel="noreferrer" target="_blank">https://issues.jboss.org/browse/ISPN-7120</a>.<br>
><br>
> Anna has finished the first step from the JIRA - collecting information<br>
> about tests in the Java HotRod client test suite (including server<br>
> integration tests) and it is now prepared for wider review.<br>
><br>
> She created a spreadsheet [1]. The spread sheet includes for each Java<br>
> test its name, the suggested target package in the TCK, whether to<br>
> include it in the TCK or not, and some other notes. The suggested<br>
> package also poses grouping for the tests (e.g. tck.query, tck.near,<br>
> tck.xsite, ...)<br>
><br>
> Let me add that right now the goal is not to create a true TCK [2]. The<br>
> goal is to make sure that all implementations of the HotRod protocol<br>
> have sufficient test coverage and possibly the same server side of the<br>
> client-server test (including the server version and configuration).<br>
><br>
> What are the next step?<br>
><br>
> * Please review the list (at least a quick look) and see if some of the<br>
> tests which are NOT suggested for the TCK should be added or vice versa.<br>
> * I suppose the next step would then be to check other implementations<br>
> (C#, C++, NodeJS, ..) and identify tests which are missing there (there<br>
> will surely be some).<br>
> * Gradually implement the missing tests in the other implementations<br>
> Note: Here we should ensure that the server is configured in the same<br>
> way for all implementations. One way to achieve this (thanks Anna for<br>
> suggestion!) is to have a shell/batch scripts for CLI which would be<br>
> executed before the tests. This can probably be done for all impls. and<br>
> both UNIX/WINDOWS. I also realize that my PR for ISPN [3] becomes<br>
> useless because it uses Creaper (Java) and we need a language-neutral<br>
> solution for configuring the server.<br>
><br>
> Some other notes:<br>
> * there are some duplicated tests in hotrod-client and server<br>
> integration test suites, in this case it probably makes sense to only<br>
> include in the TCK the server integration test<br>
> * tests from the hotrod-client module which are supposed to be part of<br>
> the TCK should be copied to the server integration test suite one day<br>
> (possibly later)<br>
><br>
> Please let us know what you think.<br>
><br>
> Thanks,<br>
> Martin<br>
><br>
><br>
> [1]<br>
> <a href="https://docs.google.com/spreadsheets/d/1bZBBi5m4oLL4lBTZhdRbIC_EA0giQNDZWzFNPWrF5G4/edit#gid=0" rel="noreferrer" target="_blank">https://docs.google.com/spreadsheets/d/1bZBBi5m4oLL4lBTZhdRbIC_EA0giQNDZWzFNPWrF5G4/edit#gid=0</a><br>
> [2] <a href="https://en.wikipedia.org/wiki/Technology_Compatibility_Kit" rel="noreferrer" target="_blank">https://en.wikipedia.org/wiki/Technology_Compatibility_Kit</a><br>
> [3] <a href="https://github.com/infinispan/infinispan/pull/5012" rel="noreferrer" target="_blank">https://github.com/infinispan/infinispan/pull/5012</a><br>
> _______________________________________________<br>
> infinispan-dev mailing list<br>
> <a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
<br>
<br>
--<br>
Radim Vansa <<a href="mailto:rvansa@redhat.com" target="_blank">rvansa@redhat.com</a>><br>
JBoss Performance Team<br>
<br>
_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
</blockquote></div><div dir="ltr">-- <br></div><div data-smartmail="gmail_signature"><div dir="ltr"><p class="inbox-inbox-fullname-container" style="box-sizing:border-box;color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><span class="inbox-inbox-firstname-container" style="box-sizing:border-box">SEBASTIAN</span><span class="inbox-inbox-Apple-converted-space"> </span><span class="inbox-inbox-lastname-container" style="box-sizing:border-box">ŁASKAWIEC</span></p><p class="inbox-inbox-position-container" style="box-sizing:border-box;color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><span class="inbox-inbox-position" style="box-sizing:border-box">INFINISPAN DEVELOPER</span></p><p class="inbox-inbox-legal-container" style="box-sizing:border-box;font-family:overpass,sans-serif;margin:0px;font-size:10px;color:rgb(153,153,153)"><a class="inbox-inbox-redhat-anchor" href="https://www.redhat.com/" target="_blank" style="box-sizing:border-box;color:rgb(0,136,206);margin:0px;text-decoration:none">Red Hat<span class="inbox-inbox-Apple-converted-space"> </span><span style="box-sizing:border-box">EMEA</span></a></p><table border="0" style="box-sizing:border-box;color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium"><tbody style="box-sizing:border-box"><tr style="box-sizing:border-box"><td width="100px" style="box-sizing:border-box"><a href="https://red.ht/sig" style="box-sizing:border-box"><img width="90" height="auto" style="box-sizing: border-box;" src="https://www.redhat.com/files/brand/email/sig-redhat.png"></a></td></tr></tbody></table></div></div>