[infinispan-dev] Introducing JGroups scopes

[Manik Surtani]

Just to clarify, there are 2 phases to this, if you will.

Phase 1: Infinispan makes use of scopes internally, for transactional and non-transactional calls to gain greater concurrency in the way JGroups broadcasts messages.

Phase 2: Allow user defined scopes.

Lets start with 1 and make sure we have a workable structure first.  So to revisit the problem (Vladimir/Bela - pls correct me if I am wrong), JGroups serializes the delivery of messages off the wire per-sender.  This means that Node A may send many messages, but a given recipient, say Node B, would queue up these messages and process them in order.  This means that even though we may have many threads on A doing non-related work, on disjoint data sets, and although this happens in parallel on A, by the time they get to B they get processed in order.  

To work around this, we use the OOB flag for synchronous messages (since we hold locks for these until the RPC returns and we know that data from such transactions cannot get reordered leading to inconsistent data).

The issue is that if we use _async_ mode, we cannot use OOB since things can get reordered and hence inconsistent, and we _rely_ on JGroups processing messages in order to maintain consistency.

Scopes helps us parallelize this.  On a very basic level, if the scope is a cache name, then at least transactions on different caches in the same cache manager can be handled in parallel.  But it can get better: I propose using some mechanism to represent the lock or locks touched by the transaction as your scope.  So any other transaction touching one or more of the same locks will be serialized, everything else could happen in parallel.  Perhaps a Bloom filter?

Cheers
Manik


On 31 Jan 2011, at 13:19, Galder Zamarreño wrote:

> Hmmmm, not sure about adding a method to IC, cos that would lead to code like this which is very verbose:
> 
> cache.getAdvancedCache().getInvocationContextContainer().getInvocationContext().setScope("X")
> 
> As a user, I'd prefer something like:
> 
> cache.getAdvancedCache().withScope("X")
> 
> However, this would only be used to attach a particular scope. I envision a global, cache manager level configuration such as that marks all caches in that cache manager to be scoped by the cache name. That way, HTTP session repl code would need no changing at all and with a single flag, they could make each individual cache to act in its own scope. The code above would allow two caches to share the same scope for those situations where we want sequential calls for the two caches, i.e.:
> 
> cache1.getAdvancedCache().withScope("X")
> cache1.put()
> cache2.getAdvancedCache().withScope("X")
> cache2.put()
> 
> By the way, I don't understand your comment of "refine this scope by adding GlobalTransaction and so on".  If we take the cache1/cache2 example and imagine that these two would be called within a transaction, I'd imagine that the transaction itself would act as a scope and so would not need for a particular scope definition? i.e.
> 
> tx.begin()
> cache1.put()
> cache2.put()
> tx.commit()
> 
> At first glance, 5.0 would be the perfect target for this, but we'd need to take in account for any requirements coming from Paul since HTTP session repl could greatly benefit from it.
> 
> As far as 2LC is concerned, within a deployment multiple txs will be operating on same caches so scoping could not be used. However different deployments could benefit from deployment level scoping so that messages for different caches can be sent in parallel. In AS environment different deployments could share the same 2LC cache manager and so the configuration option above would not work, but I'd need a deployment level scope that I'd use with cache.withScope().
> 
> Cheers,
> 
> On Jan 28, 2011, at 8:54 PM, Vladimir Blagojevic wrote:
> 
>> Forgot to mention that scopes require addition of SCOPE protocol to the 
>> stack. So we can probably do this on 5.0 branch only.
>> On 11-01-28 4:52 PM, Vladimir Blagojevic wrote:
>>> Hey,
>>> 
>>> JGroups scopes [1] have been implemented for a while now and we should
>>> think about using them in Infinispan. I looked around a bit to see how
>>> we can use scope and satisfy requirements Manik outlined [2]. I think a
>>> good idea would be to introduce scope method in InvocationContext? We
>>> can start by implementing scope to return hash of cache name where
>>> invocation originated and subsequently refine this scope by adding
>>> GlobalTransaction and so on. Users, if they really need to scope their
>>> calls could do so by attaching additional markers to InvocationContext
>>> or smth similar.
>>> 
>>> WDYT?
>>> 
>>> 
>>> [1]https://issues.jboss.org/browse/JGRP-822
>>> [2]https://issues.jboss.org/browse/ISPN-3
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> 
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> --
> Galder Zamarreño
> Sr. Software Engineer
> Infinispan, JBoss Cache
> 
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Manik Surtani
manik at jboss.org
twitter.com/maniksurtani

Lead, Infinispan
http://www.infinispan.org