[infinispan-dev] Shipping logical address String in Hot Rod protocol v1.1?

[Dan Berindei]

On Tue, Nov 29, 2011 at 5:09 PM, Bela Ban <bban at redhat.com> wrote:
>
>
> On 11/29/11 3:50 PM, Galder Zamarreño wrote:
>>
>> On Nov 29, 2011, at 2:23 PM, Bela Ban wrote:
>>
>>>
>>>
>>> On 11/29/11 2:10 PM, Galder Zamarreño wrote:
>>>> Hi,
>>>>
>>>> We've been having a discussion this morning with regards to the Hot Rod changes introduced in 5.1 with regards to hashing.
>>>>
>>>> When Hot Rod server is deployed in AS, in order to start correctly, it requires the Hot Rod server to start before any other (clustered) caches in AS. This is because any hashing can only be done once the hash has been calculated on the hot rod endpoint.
>>>>
>>>> Although this can be fixed in a hacky way (have all caches configured to start lazily and let the Hot Rod server start the topology cache and then all defined caches, ugh), we're considering a slight change to Hot Rod protocol v 1.1 (https://docs.jboss.org/author/display/ISPN/Hot+Rod+Protocol+-+Version+1.1) that would solve this problem.
>>>>
>>>> Instead of hashing on the hot rod endpoint address (host:port), we could hash on JGroups's Address UTF-8 toString representation, which is the logical address. The advantage here is any cache can calculate the hash on this from the start, no need to wait for Hot Rod server to start. The downside is that Hot Rod clients need to be aware of this UTF-8 string in order to hash to the same thing, so it'd mean shipping this back to the clients alongside the hot rod endpoint info.
>>>
>>>
>>> I have a few concerns, maybe not an issue with the way HotRod uses this,
>>> but I wanted to bring this up anyway:
>>
>> These are very valid concerns.
>>
>>> - What if a logical name is null ? A user can do JChannel.setName(null);
>>
>> Hmmm, what's the JGroups address toString() representation in that case?
>
>
> The underlying UUID would be printed (not a nice sight !)...
>

Hmm, I sometimes see UUIDs in the test suite logs. This makes me
think, can we rely on the discovery protocol always giving us the
logical names of the other cluster members during/after the
viewAccepted() callback?

>
>
>>> - What if the logical name changes at runtime ? Certainly not
>>> recomended, but the APi allows this, or I should rather say, doesn't
>>> prevent it... :-(
>>
>> If the logical name changes, is a view change triggered?
>
>
> No. Maybe I should disallow changing a logical name after connect() has
> been called...
>
>
>> How are other nodes gonna locate this node if it's logical address has changed?
>
>
> Through the discovery process. Again, maybe I should just disallow this
> use case, as it IMO doesn't make any sense.
>
>
>> Looks dangerous from a JGroups perspective, why is it allowed?
>
>
> I like to give users a lot of rope to hang themselves ... :-) ha ha
>
> OK, you convinced me: https://issues.jboss.org/browse/JGRP-1395, for
> 3.0.2 and 3.1.0
>

+1

>
>>> - What if we have multiple members with the same logical name ?
>>
>> Can that happen?
>
>
> Yes
>

It's not very likely with standalone Infinispan, as we also add a
random number to the configured node name. We also have the option of
enforcing a unique node name by just failing to start when we detect
duplicates.

Don't know how AS sets the node names, but I'm pretty sure they also
ensure a unique node name.

>
>> How can JGroups work correctly if multiple members have the same logical name? How can unicasts be directed correctly in this case?
>
>
> Because the logical name is just a nice way to hide a UUID. JGroups
> never uses the logical address for anything but pretty-printing; instead
> the underlying UUID is used for addressing purposes.
>
>
>>> - At startup, we may not yet have the logical names of all members, as
>>> this is done as part of the discovery protocol…
>>
>> Hmmm, don't think that's important. What matters is that when node A starts, after the channel is open, it's logical address is availble locally.
>
>
> OK , that'll be the case
>
>
>> That's the same right now. When Hot Rod starts, it maps the running node's JGroups address to its Hot Rod enpoint address.
>
>
> OK
>
>
>>> - Can't you hash on the address ? It'll be available as soon as
>>> connect() returns. Is that too late ?
>>
>> That would mean hashing a Java object which is not portable, hence why we need a hash on something that's portable, i.e. a UTF-8 String.
>
>
> Can't you grab the UUID which is 2 longs and simply send 2 longs across
> the wire ? The UUID is guaranteed to be unique.
>

If we commit to sending a UUID to the HotRod clients it will be harder
to send something else in the future. One of the reasons we chose to
send a string was so that we could change the way we generate that
string without breaking the clients.

For example I'm very interested in starting nodes with a fixed hash
seed, so that after a restart the local cache store is still relevant
for this node. It would be also nice if the order of nodes on the hash
wheel wouldn't change between test runs.

As a compromise we could send a 32-bit hash to the clients and specify
the algorithm to generate virtual node hashes from it in the HotRod
protocol. That way we won't "require" hashes to change on every
startup, but we can still use UUID by default to make it more
reliable.

>
>>> Note that you can also generate your own addresses (using
>>> AddressGenerator) if that helps…
>>
>> Hmmmm, actually, this could be useful. What are the chances of generating two identical addresses? :)
>
> That depends on the AddressGenerator you rpovide. The idea behind
> AddressGenerator is to use a UUID, but add some additional data to every
> address, such as machineId, rackId and siteId, as used in RELAY.
>

Our hash seeds never change, so it would be wasteful to send them with
every message.
On the other hand, I could say the same thing about TopologyAwareAddress :)

> --
> Bela Ban
> Lead JGroups (http://www.jgroups.org)
> JBoss / Red Hat
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev