On Tue, Apr 13, 2010 at 12:44 PM, Manik Surtani
<manik(a)jboss.org> wrote:
> On 12 Apr 2010, at 02:53, Vladimir Ralev wrote:
>
>> (resend to group)
>>
>> A couple of notes and questions:
>>
>> 1. Indeed from our POV the jvmRoute-based load balancing combined with DIST is a
bit problematic because it concentrates the LB effort in the HTTP container logic on the
serer side. There are many issues with non-HTTP protocols, but we can put this aside for
now and just focus on the web-based chatroom case. Other protocols use other load
balancers anyway.
>
> Depends if the protocol supports some concept of a session id. This is the critical
bit, AIUI, pls correct me if I am wrong...
Yes, right now we care only about protocols that have session id. But
most protocols don't have a reliable cookie/url param concept to
implement a jvmRoute-like method. In most cases, what we can do is let
some non-HTTP load balancer pick a node and then "drag" the associated
HTTP sessions to that node with jvmRoute modifications. But we also
have a CH-based load balancer for those who need CH for all protocols.
Right now it works with JB Cache, but when we switch to DIST I am a
bit worried because there would be no buddy data gravitation on
demand(?) [$1].
Well, there is, in the form of remote lookups. Which can get expensive over time. Hmm,
then maybe we would need to expose the CH algo. Have you looked at the HotRod protocol
(on the wiki)?
>
>> 2. The current plan for jvmRoute-based load balancing
https://jira.jboss.org/jira/browse/JBAS-7853 seems to simply check if the data is
available locally in the node and if not try to determine where is it. But DIST will
probably maintain at least two nodes where the data is local at all times. Thus chatroom
users will probably end up dispersed between those two or more nodes, because they hit
mod_cluster independently and eventually they will be routed randomly. Unless infinispan
locality-check AND location-determination APIs somehow have a concept of a "primary
node" and all requests with same affinity key end up there?
>
> Yes, this is the plan, after the first request (by each independent user) is routed
to a random node by the LB and is 'corrected' so that subsequent interactions by
that user is directed to one of the nodes where the state is local.
>
>> 3. At this point, we could implement some jvmRoute hack in the application to
redirect one group of users to the other node, but it would be better if either we have a
primary node concept in DIST or have a mod_cluster "lesser", but deterministic
CH algorithm that takes URL params into account?
>
> The "primary node" concept is implicit. So on the server side, when you
check for an ownership list of a key, you get back a list of servers. This list is
deterministic anywhere in the cluster - including the *order* of the list, so the first
entry could be considered "primary".
This may work. I wonder if joining and leaving nodes would change the
primary node over time even if the first(primary) node in the list
itself is not dead?
If new nodes (with new addresses) join, yes they may become "primary" in this
sense. So then such sessions would continue on the new node (which is fine, right? The
data will still be local).
>> 4. Even if mod_cluster was aware of the CH in Infinispan it
would have to deal with this issue. And as mentioned earlier our load balancer (MLB) will
probably work this way a least for non-HTTP protocols.
>
> Then really no point in trying to solve this on the mod_cluster level. :)
If mod_cluster is able to pick primary node there are tons of
advantages in solving it in mod_cluster. And our MLB is much slower
than mod_cluster + it will have to be modified anyway due to [$1].
>
>> 5. Is there any plan in either AS or Infinispan for fault-tolerant timers? This
is another example where you want to have the timers replicated on N nodes, but need only
one primary node to actually execute the logic.
>
> Nope (Infinispan).
>
>>
>> Another off-topic question: How does mod_cluster detect dead nodes?
>>
>>
>> ----- Original Message -----
>> From: "Brian Stansberry" <brian.stansberry(a)redhat.com>
>> To: "Manik Surtani" <msurtani(a)redhat.com>
>> Cc: "Vladimir Ralev" <vralev(a)redhat.com>, "Brian
Stansberry" <bstansbe(a)redhat.com>, "Jean Deruelle"
<jderuell(a)redhat.com>, "Paul Ferraro" <paul.ferraro(a)redhat.com>,
"cluster-dev" <jboss-cluster-dev(a)lists.jboss.org>
>> Sent: Friday, April 9, 2010 6:00:07 PM GMT +02:00 Athens, Bucharest, Istanbul
>> Subject: Re: Consistent Hashing in mod_cluster
>>
>> I'm putting this on the cluster-dev list as there's no reason for it to
>> continue in private.
>>
>> Manik, I agree with your comment about avoiding tight coupling between
>> the ISPN hash algorithm and something inside mod_cluster. The approach
>> we decided to take was to change the jvmRoute -- see
>>
https://jira.jboss.org/jira/browse/JBAS-7853. The Mobicents folks are
>> already familiar with that approach since the AS has long done that on
>> failover. They've added a browser redirect technique that would even
>> avoid the "1 'inefficient' dispatch" issue you noted.
>>
>> What's different about what Vladimir's raising here is the need to try
>> to group sessions together; i.e. use something other than the cache key
>> (i.e. jsessionid) as the factor in the hash algorithm. He mentioned a
>> use case for multiple users in a chat where the goal would be to cache
>> sessions locally with chat state. AIUI there's also a use case where a
>> user has a SIP session and an HTTP session, so the goal would be to have
>> both session cached together.
>>
>> Doing that will clearly require quite a bit of AS work -- some pluggable
>> policy for determining what the "affinity key" is based on the call
>> context (null == use the cache key for hashing). Then use the
>> affinityKey can calling into ISPN (i.e. ISPN-359) and when determining
>> where to fail over to (JBAS-7853).
>>
>> This will also be useful for the HTTPSession + SFSB in same request use
>> case, where colocating the SFSB and the web session is optimal.
>>
>> I hadn't opened a JIRA for that yet; just tried to but JIRA seems to not
>> be working?
>>
>> Hmm, an issue is if the caches involved in the overall picture (e.g.
>> httpsession cache and "chat state" cache) end up using different
>> CacheManagers/JGroups Channels. This would quite likely happen (see
>>
http://community.jboss.org/thread/149868 for discussion why). If that's
>> the case, the views between the channels could end up slightly different
>> -- e.g. servers A and B are booting at the same time, so due to slight
>> differences in the boot, Channel for HTTPSession cache ends up with view
>> {A, B} and Channel for "chat state" cache ends up with {B, A}. In that
>> case I wouldn't expect there to be a reliable, performant way to ensure
>> affinity, since the view is a major factor in the hashing algorithm.
>>
>> On 04/09/2010 04:53 AM, Manik Surtani wrote:
>>> I made a few comments on the JIRA to respond to your comment; Brian, correct
me if I have missed something, but this is pretty much what we spoke about in Brno? Do
you have a JIRA in JBAS for this as well?
>>>
>>>
>>> On 8 Apr 2010, at 19:39, Vladimir Ralev wrote:
>>>
>>>> Thanks, this will work great for us. However, this issue seems to be
scoped purely in Infinispan
https://jira.jboss.org/jira/browse/ISPN-359
>>>>
>>>> A reciprocal feature is needed in mod_cluster. Would it be possible
mod_cluster to use a URI parameter, a cookie as affinity key. AFAIK from mod_jk, it only
supports using the jsessionid cookie and uri param (hardcoded). Ultimately we'd like
to have it use a customer GET param or cookie for affinity. I couldnt find any doc related
to that. Would this be possible?
>>>>
>>>> ----- Original Message -----
>>>> From: "Manik Surtani"<msurtani(a)redhat.com>
>>>> To: "Brian Stansberry"<brian.stansberry(a)redhat.com>
>>>> Cc: "Vladimir Ralev"<vralev(a)redhat.com>, "Brian
Stansberry"<bstansbe(a)redhat.com>, "Jean
Deruelle"<jderuell(a)redhat.com>, "Paul
Ferraro"<paul.ferraro(a)redhat.com>
>>>> Sent: Thursday, April 8, 2010 1:43:00 PM GMT +02:00 Athens, Bucharest,
Istanbul
>>>> Subject: Re: Consistent Hashing in mod_cluster
>>>>
>>>>
>>>> On 6 Apr 2010, at 20:02, Brian Stansberry wrote:
>>>>
>>>>> Hi Vladimir,
>>>>>
>>>>> I've been on vacation. I replied to your question on the wiki
page, but what you're talking about here is a bigger picture thing.
>>>>>
>>>>> Sounds like your situation is a good use case for
https://jira.jboss.org/jira/browse/ISPN-359 which is an Infinispan JIRA. (At least I
believe that's the JIRA that gets into allowing the caller to provide hints such that
separate keys hash to the same storage nodes -- Manik, please correct me if I'm
wrong.)
>>>>
>>>> Yup, this is correct.
>>>>
>>>>>
>>>>> On 03/31/2010 01:58 PM, Vladimir Ralev wrote:
>>>>>> Hello Brian,
>>>>>>
>>>>>> Not sure if you are following
http://community.jboss.org/wiki/Consistent-hashingbasedJBCdatapartitionin... so
decided to email you.
>>>>>>
>>>>>> We are very interested in the consistent hashing feature in the
load balancer working cooperatively with the cache and would like to follow the
development and contribute if needed. Particularly, we in Moicents already have a number
of use-cases that concern bothtelecom and pure web applications. For example - a web based
chat room will have a number of participants with different HTTP sessions and they share
the same state somewhere in another cache at the same node. It would be desireable to
failover such groups of users together to avoid having them to lookup the chat state in
remote caches over the network. That is why, if possible, we are interested in using not
just the HTTP session ID for lookups, but have a configurable affinity key/mask that is
taken from the messages, so that you could route all requests with
http://redhat.com/?chatroom=1 always to the same node, and "chatroom" as an URL
param is the affinity key with value "1".
>>>>>>
>>>>>> Sailfin/Glassfish have something similar with their CLB.
>>>>>>
>>>>>> In Mobicents we would be using it to coordinate not only
mod_cluster and cache, but the SIP load balancer and other protocol load balancers as
well. In fact, our load balancer already supports HTTP consistent hashing, however it
doesn't have the high performance of mod_cluster (no AJP either) and is only
recommended when SIP+HTTP coordination is a requirement for our telco users.
>>>>>>
>>>>>>
http://docs.google.com/present/view?id=dc5jp5vx_89cxdvtxcm (Slide
17 has a helpful diagram)
>>>>>>
>>>>>> The other technique we are using to coordinate multiprotocol load
balancers with local cache is this
>>>>>>
>>>>>>
http://docs.google.com/present/edit?id=0AdgrvwaZgnJ1ZGM1anA1dnhfOTZnZjYyc...
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Brian Stansberry
>>>>> Lead, AS Clustering
>>>>> JBoss by Red Hat
>>>>
>>>
>>
>>
>> --
>> Brian Stansberry
>> Lead, AS Clustering
>> JBoss by Red Hat
>> _______________________________________________
>> jboss-cluster-dev mailing list
>> jboss-cluster-dev(a)lists.jboss.org
>>
https://lists.jboss.org/mailman/listinfo/jboss-cluster-dev
>
>
> --
> Manik Surtani
> manik(a)jboss.org
> Lead, Infinispan
> Lead, JBoss Cache
>
http://www.infinispan.org
>
http://www.jbosscache.org
>
>
>
>
>
> _______________________________________________
> jboss-cluster-dev mailing list
> jboss-cluster-dev(a)lists.jboss.org
>
https://lists.jboss.org/mailman/listinfo/jboss-cluster-dev
>