<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Dec 11, 2013 at 10:38 AM, Radim Vansa <span dir="ltr"><<a href="mailto:rvansa@redhat.com" target="_blank">rvansa@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Hi Dan<br>
<br>
I am not speaking about changing something for the C++ client, I
understand that the client code cannot be changed in order to keep
the backward compatibility.<br></div></div></blockquote><div><br></div><div>Sure, I was just trying to give some background information on what we discussed and why we still have the wheel-based CH in the client.<br>
</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><div>
<br>
The current hash-wheel approach is working well, but there are few
flaws that could be fixed keeping the client code untouched.
Please, correct me if I am wrong.<br>
<br>
1) The denormalization is executed for every client for every
topology change/client join. I don't have any numbers, but calling
the hashing algorithm million times per every such occasion sounds
as wasting computing power. -> cache the denormalized stuff on
server<br></div></div></blockquote><div><br></div><div>+1, like I said it would be easy to do but it never came up as a problem before.<br></div><div><br> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><div>
<br>
2) The server is sending numOwners hashIds per segment, one for
each owner. What's the reason for that? I think that only primary
owners should be inserted there. This would:<br></div></div></blockquote><div><br></div><div>The main reason is to support clients from Infinispan 5.1, which pick a random owner instead of always choosing the primary (<a href="https://issues.jboss.org/browse/ISPN-2655">https://issues.jboss.org/browse/ISPN-2655</a>).<br>
<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><div>
a) target all PUT requests to primary owner, reducing PUT latency
and lowering the general load in cluster<br></div></div></blockquote><div><br></div><div>Nope, it wouldn't. The same fraction of requests would go to the primary owner as before, because we won't find the exact "denormalized" hash id that maps to the segment border when normalized.<br>
<br><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><div>
b) reduce the routing information<br></div></div></blockquote><div><br></div><div>For 7.0, I guess we could say that 5.1 clients are no longer supported and we could switch to sending only the primary owners to the clients. But I'm not sure whether the loss of backwards compatibility is worth a couple hundred bytes sent once for every client.<br>
<br><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><div>
<br>
And yes, ISPN-3530 and ISPN-3701 are pretty serious, but IMO
rather orthogonal to the segment vs. hash wheel approach and its
details.<span class=""><font color="#888888"><br>
<br></font></span></div></div></blockquote><div><br></div><div>Agree. Could you create issues in JIRA for both your proposals? <br><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><div><span class=""><font color="#888888">
Radim</font></span><div><div class="h5"><br>
<br>
<br>
On 12/11/2013 09:18 AM, Dan Berindei wrote:<br>
</div></div></div><div><div class="h5">
<blockquote type="cite">
<div dir="ltr">
<div>
<div>Hi Radim<br>
<br>
</div>
Actually, it's me that wrote the denormalization code :)<br>
<br>
</div>
It was meant as a stop-gap measure before we upgraded the HotRod
protocol to support the segment-based consistent hash, but the
denormalization worked well enough (or so we thought) that we
didn't get to changing the protocol yet. <br>
<br>
That's not a big change in itself, but we also wanted to make
the consistent hash per-cache on the client (it's now per-cache
manager), and that's a bit more complicated to do. And it's not
like it would have been a good idea to change this before
starting the C++ client, the client would still have to support
the current style of consistent hash.<br>
<br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Dec 10, 2013 at 8:17 PM,
Radim Vansa <span dir="ltr"><<a href="mailto:rvansa@redhat.com" target="_blank">rvansa@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Hi Galder,<br>
<br>
as I am trying to debug some problem in C++ client, I was
looking into<br>
the server code. And I am not sure whether I understand
the code<br>
correctly, but it seems to me that the server denormalizes
the<br>
consistent hash for each client anew (after each topology
change or<br>
client joining). Is this true? Looking into trace logs, I
can see stuff<br>
like<br>
<br>
18:15:17,339 TRACE
[org.infinispan.server.hotrod.Encoders$Encoder12$]<br>
(HotRodServerWorker-12) Writing hash id 639767 for <a href="http://192.168.11.101:11222" target="_blank">192.168.11.101:11222</a><br>
<br>
From denormalizeSegmentHashIds() method I see that this
means that we<br>
have executed the hash function 639768 times just to
notify one client.<br>
Is my understanding correct?<br>
</blockquote>
<div><br>
</div>
<div>Yes, this happens every time a client joins and/or
every time the cache topology changes. <br>
<br>
We could easily cache the result of
denormalizeSegmentHashIds, as it only depends on the
number of segments. It's just that I wasn't expecting it
to take so many iterations.<br>
<br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Also, there is nothing like the concept of primary owner,
is this right?<br>
</blockquote>
<div><br>
</div>
<div>The client CH doesn't have a concept of backup owners.
But for each (hash id, server) pair that gets sent to the
client, it means all the hash codes between the previous
hash id and this hash id have this server as the primary
owner. The server in the next (hash id, server) pair is
the first backup, and so on.<br>
</div>
<div><br>
</div>
<div>For each segment, the server generates numOwners (hash
id, server) pairs. That means, for most of the hash codes
in the segment, the list of owners on the client will be
the same as the list of owners on the server. But for
0.0002 (leewayFraction) of the hash codes, the client
primary owner will be indeed one of the server backup
owners.<br>
<br>
<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
I thought that every first request in HotRod will go to
primary owner,<br>
so that the PUT does not have to do the first hop and is
executed<br>
directly on the primary. But it seems to me that it goes
to any of the<br>
owners (practically random one, as you are only looking
for the numOwner<br>
ids in leeway = on the beginning of the range - then,
99.98% or more<br>
requests should go to the server with last position in the
leeway). This<br>
looks pretty suboptimal for writes, isn't it?<br>
</blockquote>
<div><br>
</div>
<div>I'm not sure what you mean here, but I'm pretty sure
the request goes to the correct server because we have a
test for it: ConsistentHashV1IntegrationTest<br>
<br>
</div>
<div>Cheers<br>
</div>
<div>Dan<br>
<br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Cheers<br>
<br>
Radim<br>
<br>
PS: for every line of code you write in Scala, God kills a
kitten<br>
<span><font color="#888888"><br>
--<br>
Radim Vansa <<a href="mailto:rvansa@redhat.com" target="_blank">rvansa@redhat.com</a>><br>
JBoss DataGrid QA<br>
<br>
_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
</font></span></blockquote>
</div>
<br>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
infinispan-dev mailing list
<a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></pre>
</blockquote>
<br>
<br>
<pre cols="72">--
Radim Vansa <a href="mailto:rvansa@redhat.com" target="_blank"><rvansa@redhat.com></a>
JBoss DataGrid QA
</pre>
</div></div></div>
<br>_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br></blockquote></div><br></div></div>