7:10 a.m.
Hi,
We've been having a discussion this morning with regards to the Hot Rod changes
introduced in 5.1 with regards to hashing.
When Hot Rod server is deployed in AS, in order to start correctly, it requires the Hot
Rod server to start before any other (clustered) caches in AS. This is because any hashing
can only be done once the hash has been calculated on the hot rod endpoint.
Although this can be fixed in a hacky way (have all caches configured to start lazily and
let the Hot Rod server start the topology cache and then all defined caches, ugh),
we're considering a slight change to Hot Rod protocol v 1.1
(https://docs.jboss.org/author/display/ISPN/Hot+Rod+Protocol+-+Version+1.1) that would
solve this problem.
Instead of hashing on the hot rod endpoint address (host:port), we could hash on
JGroups's Address UTF-8 toString representation, which is the logical address. The
advantage here is any cache can calculate the hash on this from the start, no need to wait
for Hot Rod server to start. The downside is that Hot Rod clients need to be aware of this
UTF-8 string in order to hash to the same thing, so it'd mean shipping this back to
the clients alongside the hot rod endpoint info.
In spite of the added bytes, there's a benefit to this, and that is the fact that
we're not tying a specific format to the String. That is, we're just telling the
clients to take this UTF-8 string and hash on that. So, it's internal representation
could evolve over time with no impact to the client. In the current v 1.1 protocol,
clients and servers assume a String of format host:port to do the hashing.
A third way to fix this would have been to have Hot Rod servers run with a different cache
manager to the rest (but still same JGroups channel), but according to Dan, AS7 is not
capable of doing this.
So, what does people think of this protocol change in v1.1? This version was introduced in
5.1 and since we still have not released yet, this is the right time to consider protocol
changes like this. Personally, I'm in favour.
Cheers,
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
7:23 a.m.
New subject: Shipping logical address String in Hot Rod protocol v1.1?
On 11/29/11 2:10 PM, Galder Zamarreño wrote:
Hi,
We've been having a discussion this morning with regards to the Hot Rod changes
introduced in 5.1 with regards to hashing.
When Hot Rod server is deployed in AS, in order to start correctly, it requires the Hot
Rod server to start before any other (clustered) caches in AS. This is because any hashing
can only be done once the hash has been calculated on the hot rod endpoint.
Although this can be fixed in a hacky way (have all caches configured to start lazily and
let the Hot Rod server start the topology cache and then all defined caches, ugh),
we're considering a slight change to Hot Rod protocol v 1.1
(https://docs.jboss.org/author/display/ISPN/Hot+Rod+Protocol+-+Version+1.1) that would
solve this problem.
Instead of hashing on the hot rod endpoint address (host:port), we could hash on
JGroups's Address UTF-8 toString representation, which is the logical address. The
advantage here is any cache can calculate the hash on this from the start, no need to wait
for Hot Rod server to start. The downside is that Hot Rod clients need to be aware of this
UTF-8 string in order to hash to the same thing, so it'd mean shipping this back to
the clients alongside the hot rod endpoint info.
I have a few concerns, maybe not an issue with the way HotRod uses this,
but I wanted to bring this up anyway:
- What if a logical name is null ? A user can do JChannel.setName(null);
- What if the logical name changes at runtime ? Certainly not
recomended, but the APi allows this, or I should rather say, doesn't
prevent it... :-(
- What if we have multiple members with the same logical name ?
- At startup, we may not yet have the logical names of all members, as
this is done as part of the discovery protocol...
- Can't you hash on the address ? It'll be available as soon as
connect() returns. Is that too late ?
Note that you can also generate your own addresses (using
AddressGenerator) if that helps...
--
Bela Ban
Lead JGroups (http://www.jgroups.org)
JBoss / Red Hat
8:50 a.m.
New subject: Shipping logical address String in Hot Rod protocol v1.1?
On Nov 29, 2011, at 2:23 PM, Bela Ban wrote:
On 11/29/11 2:10 PM, Galder Zamarreño wrote:
> Hi,
>
> We've been having a discussion this morning with regards to the Hot Rod changes
introduced in 5.1 with regards to hashing.
>
> When Hot Rod server is deployed in AS, in order to start correctly, it requires the
Hot Rod server to start before any other (clustered) caches in AS. This is because any
hashing can only be done once the hash has been calculated on the hot rod endpoint.
>
> Although this can be fixed in a hacky way (have all caches configured to start lazily
and let the Hot Rod server start the topology cache and then all defined caches, ugh),
we're considering a slight change to Hot Rod protocol v 1.1
(https://docs.jboss.org/author/display/ISPN/Hot+Rod+Protocol+-+Version+1.1) that would
solve this problem.
>
> Instead of hashing on the hot rod endpoint address (host:port), we could hash on
JGroups's Address UTF-8 toString representation, which is the logical address. The
advantage here is any cache can calculate the hash on this from the start, no need to wait
for Hot Rod server to start. The downside is that Hot Rod clients need to be aware of this
UTF-8 string in order to hash to the same thing, so it'd mean shipping this back to
the clients alongside the hot rod endpoint info.
I have a few concerns, maybe not an issue with the way HotRod uses this,
but I wanted to bring this up anyway:
These are very valid concerns.
- What if a logical name is null ? A user can do
JChannel.setName(null);
Hmmm, what's the JGroups address toString() representation in that case?
- What if the logical name changes at runtime ? Certainly not
recomended, but the APi allows this, or I should rather say, doesn't
prevent it... :-(
If the logical name changes, is a view change triggered?
How are other nodes gonna locate this node if it's logical address has changed?
Looks dangerous from a JGroups perspective, why is it allowed?
- What if we have multiple members with the same logical name ?
Can that happen? How can JGroups work correctly if multiple members have the same logical
name? How can unicasts be directed correctly in this case?
- At startup, we may not yet have the logical names of all members,
as
this is done as part of the discovery protocol…
Hmmm, don't think that's important. What matters is that when node A starts, after
the channel is open, it's logical address is availble locally.
That's the same right now. When Hot Rod starts, it maps the running node's JGroups
address to its Hot Rod enpoint address.
- Can't you hash on the address ? It'll be available as soon
as
connect() returns. Is that too late ?
That would mean hashing a Java object which is not portable, hence why we need a hash on
something that's portable, i.e. a UTF-8 String.
Note that you can also generate your own addresses (using
AddressGenerator) if that helps…
Hmmmm, actually, this could be useful. What are the chances of generating two identical
addresses? :)
--
Bela Ban
Lead JGroups (http://www.jgroups.org)
JBoss / Red Hat
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
9:09 a.m.
New subject: Shipping logical address String in Hot Rod protocol v1.1?
On 11/29/11 3:50 PM, Galder Zamarreño wrote:
On Nov 29, 2011, at 2:23 PM, Bela Ban wrote:
>
>
> On 11/29/11 2:10 PM, Galder Zamarreño wrote:
>> Hi,
>>
>> We've been having a discussion this morning with regards to the Hot Rod
changes introduced in 5.1 with regards to hashing.
>>
>> When Hot Rod server is deployed in AS, in order to start correctly, it requires
the Hot Rod server to start before any other (clustered) caches in AS. This is because any
hashing can only be done once the hash has been calculated on the hot rod endpoint.
>>
>> Although this can be fixed in a hacky way (have all caches configured to start
lazily and let the Hot Rod server start the topology cache and then all defined caches,
ugh), we're considering a slight change to Hot Rod protocol v 1.1
(https://docs.jboss.org/author/display/ISPN/Hot+Rod+Protocol+-+Version+1.1) that would
solve this problem.
>>
>> Instead of hashing on the hot rod endpoint address (host:port), we could hash on
JGroups's Address UTF-8 toString representation, which is the logical address. The
advantage here is any cache can calculate the hash on this from the start, no need to wait
for Hot Rod server to start. The downside is that Hot Rod clients need to be aware of this
UTF-8 string in order to hash to the same thing, so it'd mean shipping this back to
the clients alongside the hot rod endpoint info.
>
>
> I have a few concerns, maybe not an issue with the way HotRod uses this,
> but I wanted to bring this up anyway:
These are very valid concerns.
> - What if a logical name is null ? A user can do JChannel.setName(null);
Hmmm, what's the JGroups address toString() representation in that case?
The underlying UUID would be printed (not a nice sight !)...
> - What if the logical name changes at runtime ? Certainly not
> recomended, but the APi allows this, or I should rather say, doesn't
> prevent it... :-(
If the logical name changes, is a view change triggered?
No. Maybe I should disallow changing a logical name after connect() has
been called...
How are other nodes gonna locate this node if it's logical
address has changed?
Through the discovery process. Again, maybe I should just disallow this
use case, as it IMO doesn't make any sense.
Looks dangerous from a JGroups perspective, why is it allowed?
I like to give users a lot of rope to hang themselves ... :-) ha ha
OK, you convinced me: https://issues.jboss.org/browse/JGRP-1395, for
3.0.2 and 3.1.0
> - What if we have multiple members with the same logical name ?
Can that happen?
Yes
How can JGroups work correctly if multiple members have the same
logical name? How can unicasts be directed correctly in this case?
Because the logical name is just a nice way to hide a UUID. JGroups
never uses the logical address for anything but pretty-printing; instead
the underlying UUID is used for addressing purposes.
> - At startup, we may not yet have the logical names of all
members, as
> this is done as part of the discovery protocol…
Hmmm, don't think that's important. What matters is that when node A starts,
after the channel is open, it's logical address is availble locally.
OK , that'll be the case
That's the same right now. When Hot Rod starts, it maps the
running node's JGroups address to its Hot Rod enpoint address.
OK
> - Can't you hash on the address ? It'll be available as
soon as
> connect() returns. Is that too late ?
That would mean hashing a Java object which is not portable, hence why we need a hash on
something that's portable, i.e. a UTF-8 String.
Can't you grab the UUID which is 2 longs and simply send 2 longs across
the wire ? The UUID is guaranteed to be unique.
> Note that you can also generate your own addresses (using
> AddressGenerator) if that helps…
Hmmmm, actually, this could be useful. What are the chances of generating two identical
addresses? :)
That depends on the AddressGenerator you rpovide. The idea behind
AddressGenerator is to use a UUID, but add some additional data to every
address, such as machineId, rackId and siteId, as used in RELAY.
--
Bela Ban
Lead JGroups (http://www.jgroups.org)
JBoss / Red Hat
11:16 a.m.
On Tue, Nov 29, 2011 at 5:09 PM, Bela Ban <bban(a)redhat.com> wrote:
On 11/29/11 3:50 PM, Galder Zamarreño wrote:
>
> On Nov 29, 2011, at 2:23 PM, Bela Ban wrote:
>
>>
>>
>> On 11/29/11 2:10 PM, Galder Zamarreño wrote:
>>> Hi,
>>>
>>> We've been having a discussion this morning with regards to the Hot Rod
changes introduced in 5.1 with regards to hashing.
>>>
>>> When Hot Rod server is deployed in AS, in order to start correctly, it
requires the Hot Rod server to start before any other (clustered) caches in AS. This is
because any hashing can only be done once the hash has been calculated on the hot rod
endpoint.
>>>
>>> Although this can be fixed in a hacky way (have all caches configured to
start lazily and let the Hot Rod server start the topology cache and then all defined
caches, ugh), we're considering a slight change to Hot Rod protocol v 1.1
(https://docs.jboss.org/author/display/ISPN/Hot+Rod+Protocol+-+Version+1.1) that would
solve this problem.
>>>
>>> Instead of hashing on the hot rod endpoint address (host:port), we could hash
on JGroups's Address UTF-8 toString representation, which is the logical address. The
advantage here is any cache can calculate the hash on this from the start, no need to wait
for Hot Rod server to start. The downside is that Hot Rod clients need to be aware of this
UTF-8 string in order to hash to the same thing, so it'd mean shipping this back to
the clients alongside the hot rod endpoint info.
>>
>>
>> I have a few concerns, maybe not an issue with the way HotRod uses this,
>> but I wanted to bring this up anyway:
>
> These are very valid concerns.
>
>> - What if a logical name is null ? A user can do JChannel.setName(null);
>
> Hmmm, what's the JGroups address toString() representation in that case?
The underlying UUID would be printed (not a nice sight !)...
Hmm, I sometimes see UUIDs in the test suite logs. This makes me
think, can we rely on the discovery protocol always giving us the
logical names of the other cluster members during/after the
viewAccepted() callback?
>> - What if the logical name changes at runtime ? Certainly not
>> recomended, but the APi allows this, or I should rather say, doesn't
>> prevent it... :-(
>
> If the logical name changes, is a view change triggered?
No. Maybe I should disallow changing a logical name after connect() has
been called...
> How are other nodes gonna locate this node if it's logical address has changed?
Through the discovery process. Again, maybe I should just disallow this
use case, as it IMO doesn't make any sense.
> Looks dangerous from a JGroups perspective, why is it allowed?
I like to give users a lot of rope to hang themselves ... :-) ha ha
OK, you convinced me: https://issues.jboss.org/browse/JGRP-1395, for
3.0.2 and 3.1.0
+1
>> - What if we have multiple members with the same logical name ?
>
> Can that happen?
Yes
It's not very likely with standalone Infinispan, as we also add a
random number to the configured node name. We also have the option of
enforcing a unique node name by just failing to start when we detect
duplicates.
Don't know how AS sets the node names, but I'm pretty sure they also
ensure a unique node name.
> How can JGroups work correctly if multiple members have the same logical name? How
can unicasts be directed correctly in this case?
Because the logical name is just a nice way to hide a UUID. JGroups
never uses the logical address for anything but pretty-printing; instead
the underlying UUID is used for addressing purposes.
>> - At startup, we may not yet have the logical names of all members, as
>> this is done as part of the discovery protocol…
>
> Hmmm, don't think that's important. What matters is that when node A starts,
after the channel is open, it's logical address is availble locally.
OK , that'll be the case
> That's the same right now. When Hot Rod starts, it maps the running node's
JGroups address to its Hot Rod enpoint address.
OK
>> - Can't you hash on the address ? It'll be available as soon as
>> connect() returns. Is that too late ?
>
> That would mean hashing a Java object which is not portable, hence why we need a hash
on something that's portable, i.e. a UTF-8 String.
Can't you grab the UUID which is 2 longs and simply send 2 longs across
the wire ? The UUID is guaranteed to be unique.
If we commit to sending a UUID to the HotRod clients it will be harder
to send something else in the future. One of the reasons we chose to
send a string was so that we could change the way we generate that
string without breaking the clients.
For example I'm very interested in starting nodes with a fixed hash
seed, so that after a restart the local cache store is still relevant
for this node. It would be also nice if the order of nodes on the hash
wheel wouldn't change between test runs.
As a compromise we could send a 32-bit hash to the clients and specify
the algorithm to generate virtual node hashes from it in the HotRod
protocol. That way we won't "require" hashes to change on every
startup, but we can still use UUID by default to make it more
reliable.
>> Note that you can also generate your own addresses (using
>> AddressGenerator) if that helps…
>
> Hmmmm, actually, this could be useful. What are the chances of generating two
identical addresses? :)
That depends on the AddressGenerator you rpovide. The idea behind
AddressGenerator is to use a UUID, but add some additional data to every
address, such as machineId, rackId and siteId, as used in RELAY.
Our hash seeds never change, so it would be wasteful to send them with
every message.
On the other hand, I could say the same thing about TopologyAwareAddress :)
--
Bela Ban
Lead JGroups (http://www.jgroups.org)
JBoss / Red Hat
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
1:46 a.m.
On 11/29/11 6:16 PM, Dan Berindei wrote:
> The underlying UUID would be printed (not a nice sight !)...
>
Hmm, I sometimes see UUIDs in the test suite logs. This makes me
think, can we rely on the discovery protocol always giving us the
logical names of the other cluster members during/after the
viewAccepted() callback?
No, you can't. For example, if someone did not list *all* IP addresses
in TCPPING.initial_hosts, we would not get the logical names of some
members.
> Can't you grab the UUID which is 2 longs and simply send 2
longs across
> the wire ? The UUID is guaranteed to be unique.
>
If we commit to sending a UUID to the HotRod clients it will be harder
to send something else in the future. One of the reasons we chose to
send a string was so that we could change the way we generate that
string without breaking the clients.
Why don't you send the UUID as a (16 byte) string then ?
--
Bela Ban
Lead JGroups (http://www.jgroups.org)
JBoss / Red Hat
1:59 a.m.
On Wed, Nov 30, 2011 at 9:46 AM, Bela Ban <bban(a)redhat.com> wrote:
On 11/29/11 6:16 PM, Dan Berindei wrote:
>> The underlying UUID would be printed (not a nice sight !)...
>>
>
> Hmm, I sometimes see UUIDs in the test suite logs. This makes me
> think, can we rely on the discovery protocol always giving us the
> logical names of the other cluster members during/after the
> viewAccepted() callback?
No, you can't. For example, if someone did not list *all* IP addresses
in TCPPING.initial_hosts, we would not get the logical names of some
members.
Ok, so using the logical names is definitely out.
>> Can't you grab the UUID which is 2 longs and simply send 2 longs across
>> the wire ? The UUID is guaranteed to be unique.
>>
>
> If we commit to sending a UUID to the HotRod clients it will be harder
> to send something else in the future. One of the reasons we chose to
> send a string was so that we could change the way we generate that
> string without breaking the clients.
Why don't you send the UUID as a (16 byte) string then ?
Yeah, that would work. However, a UUID is not always a valid UTF-8
string, so we should probably define it in the protocol as an array of
bytes (without any meaning).
--
Bela Ban
Lead JGroups (http://www.jgroups.org)
JBoss / Red Hat
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
2:04 a.m.
On 11/30/11 8:59 AM, Dan Berindei wrote:
> Why don't you send the UUID as a (16 byte) string then ?
>
Yeah, that would work. However, a UUID is not always a valid UTF-8
string, so we should probably define it in the protocol as an array of
bytes (without any meaning).
Yes. We did something similar in JGroups, take a look at
ENCRYPT.byteArrayToHexString().
--
Bela Ban
Lead JGroups (http://www.jgroups.org)
JBoss / Red Hat
3:28 a.m.
On Wed, Nov 30, 2011 at 10:04 AM, Bela Ban <bban(a)redhat.com> wrote:
On 11/30/11 8:59 AM, Dan Berindei wrote:
>> Why don't you send the UUID as a (16 byte) string then ?
>>
>
> Yeah, that would work. However, a UUID is not always a valid UTF-8
> string, so we should probably define it in the protocol as an array of
> bytes (without any meaning).
Yes. We did something similar in JGroups, take a look at
ENCRYPT.byteArrayToHexString().
Since HotRod is a binary protocol I meant to send the UUID as it is
(no encoding whatsoever).
I think sending the UUID as a hex-encoded string, a raw byte array, or
a 32-bit hash (since that's all we need on the client) are all valid
choices.
--
Bela Ban
Lead JGroups (http://www.jgroups.org)
JBoss / Red Hat
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
5:32 a.m.
On Nov 30, 2011, at 10:28 AM, Dan Berindei wrote:
On Wed, Nov 30, 2011 at 10:04 AM, Bela Ban <bban(a)redhat.com>
wrote:
>
>
> On 11/30/11 8:59 AM, Dan Berindei wrote:
>
>>> Why don't you send the UUID as a (16 byte) string then ?
>>>
>>
>> Yeah, that would work. However, a UUID is not always a valid UTF-8
>> string, so we should probably define it in the protocol as an array of
>> bytes (without any meaning).
>
>
> Yes. We did something similar in JGroups, take a look at
> ENCRYPT.byteArrayToHexString().
>
Since HotRod is a binary protocol I meant to send the UUID as it is
(no encoding whatsoever).
I think sending the UUID as a hex-encoded string, a raw byte array, or
a 32-bit hash (since that's all we need on the client) are all valid
choices.
Right, I think relying on the UUID underneath the JGroups address is a better choice than
the logical address itself, so let's use that as base.
My preference is for a 32-bit hash which is nice and compact.
https://issues.jboss.org/browse/ISPN-1578
> --
> Bela Ban
> Lead JGroups (http://www.jgroups.org)
> JBoss / Red Hat
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
10:45 a.m.
New subject: Shipping logical address String in Hot Rod protocol v1.1?
On 29 Nov 2011, at 15:09, Bela Ban wrote:
>
> Looks dangerous from a JGroups perspective, why is it allowed?
I like to give users a lot of rope to hang themselves ... :-) ha ha
How very C++ of you. :)
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Lead, Infinispan
http://www.infinispan.org
5050
days inactive
5058
days old
infinispan-dev@lists.jboss.org
10 comments
4 participants
participants (4)
-
Bela Ban -
Dan Berindei -
Galder Zamarreño -
Manik Surtani