Re: [jboss-cluster-dev] Consistent Hashing in mod_cluster

(resend to group) A couple of notes and questions: 1. Indeed from our POV the jvmRoute-based load balancing combined with DIST is a bit problematic because it concentrates the LB effort in the HTTP container logic on the serer side. There are many issues with non-HTTP protocols, but we can put this aside for now and just focus on the web-based chatroom case. Other protocols use other load balancers anyway. 2. The current plan for jvmRoute-based load balancing https://jira.jboss.org/jira/browse/JBAS-7853 seems to simply check if the data is available locally in the node and if not try to determine where is it. But DIST will probably maintain at least two nodes where the data is local at all times. Thus chatroom users will probably end up dispersed between those two or more nodes, because they hit mod_cluster independently and eventually they will be routed randomly. Unless infinispan locality-check AND location-determination APIs somehow have a concept of a "primary node" and all requests with same affinity key end up there? 3. At this point, we could implement some jvmRoute hack in the application to redirect one group of users to the other node, but it would be better if either we have a primary node concept in DIST or have a mod_cluster "lesser", but deterministic CH algorithm that takes URL params into account? 4. Even if mod_cluster was aware of the CH in Infinispan it would have to deal with this issue. And as mentioned earlier our load balancer (MLB) will probably work this way a least for non-HTTP protocols. 5. Is there any plan in either AS or Infinispan for fault-tolerant timers? This is another example where you want to have the timers replicated on N nodes, but need only one primary node to actually execute the logic. Another off-topic question: How does mod_cluster detect dead nodes? ----- Original Message ----- From: "Brian Stansberry" <brian.stansberry(a)redhat.com&gt; To: "Manik Surtani" <msurtani(a)redhat.com&gt; Cc: "Vladimir Ralev" <vralev(a)redhat.com&gt;, "Brian Stansberry" <bstansbe(a)redhat.com&gt;, "Jean Deruelle" <jderuell(a)redhat.com&gt;, "Paul Ferraro" <paul.ferraro(a)redhat.com&gt;, "cluster-dev" <jboss-cluster-dev(a)lists.jboss.org&gt; Sent: Friday, April 9, 2010 6:00:07 PM GMT +02:00 Athens, Bucharest, Istanbul Subject: Re: Consistent Hashing in mod_cluster I'm putting this on the cluster-dev list as there's no reason for it to continue in private. Manik, I agree with your comment about avoiding tight coupling between the ISPN hash algorithm and something inside mod_cluster. The approach we decided to take was to change the jvmRoute -- see https://jira.jboss.org/jira/browse/JBAS-7853. The Mobicents folks are already familiar with that approach since the AS has long done that on failover. They've added a browser redirect technique that would even avoid the "1 'inefficient' dispatch" issue you noted. What's different about what Vladimir's raising here is the need to try to group sessions together; i.e. use something other than the cache key (i.e. jsessionid) as the factor in the hash algorithm. He mentioned a use case for multiple users in a chat where the goal would be to cache sessions locally with chat state. AIUI there's also a use case where a user has a SIP session and an HTTP session, so the goal would be to have both session cached together. Doing that will clearly require quite a bit of AS work -- some pluggable policy for determining what the "affinity key" is based on the call context (null == use the cache key for hashing). Then use the affinityKey can calling into ISPN (i.e. ISPN-359) and when determining where to fail over to (JBAS-7853). This will also be useful for the HTTPSession + SFSB in same request use case, where colocating the SFSB and the web session is optimal. I hadn't opened a JIRA for that yet; just tried to but JIRA seems to not be working? Hmm, an issue is if the caches involved in the overall picture (e.g. httpsession cache and "chat state" cache) end up using different CacheManagers/JGroups Channels. This would quite likely happen (see http://community.jboss.org/thread/149868 for discussion why). If that's the case, the views between the channels could end up slightly different -- e.g. servers A and B are booting at the same time, so due to slight differences in the boot, Channel for HTTPSession cache ends up with view {A, B} and Channel for "chat state" cache ends up with {B, A}. In that case I wouldn't expect there to be a reliable, performant way to ensure affinity, since the view is a major factor in the hashing algorithm. On 04/09/2010 04:53 AM, Manik Surtani wrote: -- Brian Stansberry Lead, AS Clustering JBoss by Red Hat