8:52 p.m.
Hey,
JGroups scopes [1] have been implemented for a while now and we should
think about using them in Infinispan. I looked around a bit to see how
we can use scope and satisfy requirements Manik outlined [2]. I think a
good idea would be to introduce scope method in InvocationContext? We
can start by implementing scope to return hash of cache name where
invocation originated and subsequently refine this scope by adding
GlobalTransaction and so on. Users, if they really need to scope their
calls could do so by attaching additional markers to InvocationContext
or smth similar.
WDYT?
[1]https://issues.jboss.org/browse/JGRP-822
[2]https://issues.jboss.org/browse/ISPN-3
8:54 p.m.
Forgot to mention that scopes require addition of SCOPE protocol to the
stack. So we can probably do this on 5.0 branch only.
On 11-01-28 4:52 PM, Vladimir Blagojevic wrote:
Hey,
JGroups scopes [1] have been implemented for a while now and we should
think about using them in Infinispan. I looked around a bit to see how
we can use scope and satisfy requirements Manik outlined [2]. I think a
good idea would be to introduce scope method in InvocationContext? We
can start by implementing scope to return hash of cache name where
invocation originated and subsequently refine this scope by adding
GlobalTransaction and so on. Users, if they really need to scope their
calls could do so by attaching additional markers to InvocationContext
or smth similar.
WDYT?
[1]https://issues.jboss.org/browse/JGRP-822
[2]https://issues.jboss.org/browse/ISPN-3
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
2:19 p.m.
Hmmmm, not sure about adding a method to IC, cos that would lead to code like this which
is very verbose:
cache.getAdvancedCache().getInvocationContextContainer().getInvocationContext().setScope("X")
As a user, I'd prefer something like:
cache.getAdvancedCache().withScope("X")
However, this would only be used to attach a particular scope. I envision a global, cache
manager level configuration such as that marks all caches in that cache manager to be
scoped by the cache name. That way, HTTP session repl code would need no changing at all
and with a single flag, they could make each individual cache to act in its own scope. The
code above would allow two caches to share the same scope for those situations where we
want sequential calls for the two caches, i.e.:
cache1.getAdvancedCache().withScope("X")
cache1.put()
cache2.getAdvancedCache().withScope("X")
cache2.put()
By the way, I don't understand your comment of "refine this scope by adding
GlobalTransaction and so on". If we take the cache1/cache2 example and imagine that
these two would be called within a transaction, I'd imagine that the transaction
itself would act as a scope and so would not need for a particular scope definition? i.e.
tx.begin()
cache1.put()
cache2.put()
tx.commit()
At first glance, 5.0 would be the perfect target for this, but we'd need to take in
account for any requirements coming from Paul since HTTP session repl could greatly
benefit from it.
As far as 2LC is concerned, within a deployment multiple txs will be operating on same
caches so scoping could not be used. However different deployments could benefit from
deployment level scoping so that messages for different caches can be sent in parallel. In
AS environment different deployments could share the same 2LC cache manager and so the
configuration option above would not work, but I'd need a deployment level scope that
I'd use with cache.withScope().
Cheers,
On Jan 28, 2011, at 8:54 PM, Vladimir Blagojevic wrote:
Forgot to mention that scopes require addition of SCOPE protocol to
the
stack. So we can probably do this on 5.0 branch only.
On 11-01-28 4:52 PM, Vladimir Blagojevic wrote:
> Hey,
>
> JGroups scopes [1] have been implemented for a while now and we should
> think about using them in Infinispan. I looked around a bit to see how
> we can use scope and satisfy requirements Manik outlined [2]. I think a
> good idea would be to introduce scope method in InvocationContext? We
> can start by implementing scope to return hash of cache name where
> invocation originated and subsequently refine this scope by adding
> GlobalTransaction and so on. Users, if they really need to scope their
> calls could do so by attaching additional markers to InvocationContext
> or smth similar.
>
> WDYT?
>
>
> [1]https://issues.jboss.org/browse/JGRP-822
> [2]https://issues.jboss.org/browse/ISPN-3
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
2:33 p.m.
Just to clarify, there are 2 phases to this, if you will.
Phase 1: Infinispan makes use of scopes internally, for transactional and
non-transactional calls to gain greater concurrency in the way JGroups broadcasts
messages.
Phase 2: Allow user defined scopes.
Lets start with 1 and make sure we have a workable structure first. So to revisit the
problem (Vladimir/Bela - pls correct me if I am wrong), JGroups serializes the delivery of
messages off the wire per-sender. This means that Node A may send many messages, but a
given recipient, say Node B, would queue up these messages and process them in order.
This means that even though we may have many threads on A doing non-related work, on
disjoint data sets, and although this happens in parallel on A, by the time they get to B
they get processed in order.
To work around this, we use the OOB flag for synchronous messages (since we hold locks for
these until the RPC returns and we know that data from such transactions cannot get
reordered leading to inconsistent data).
The issue is that if we use _async_ mode, we cannot use OOB since things can get reordered
and hence inconsistent, and we _rely_ on JGroups processing messages in order to maintain
consistency.
Scopes helps us parallelize this. On a very basic level, if the scope is a cache name,
then at least transactions on different caches in the same cache manager can be handled in
parallel. But it can get better: I propose using some mechanism to represent the lock or
locks touched by the transaction as your scope. So any other transaction touching one or
more of the same locks will be serialized, everything else could happen in parallel.
Perhaps a Bloom filter?
Cheers
Manik
On 31 Jan 2011, at 13:19, Galder Zamarreño wrote:
Hmmmm, not sure about adding a method to IC, cos that would lead to
code like this which is very verbose:
cache.getAdvancedCache().getInvocationContextContainer().getInvocationContext().setScope("X")
As a user, I'd prefer something like:
cache.getAdvancedCache().withScope("X")
However, this would only be used to attach a particular scope. I envision a global, cache
manager level configuration such as that marks all caches in that cache manager to be
scoped by the cache name. That way, HTTP session repl code would need no changing at all
and with a single flag, they could make each individual cache to act in its own scope. The
code above would allow two caches to share the same scope for those situations where we
want sequential calls for the two caches, i.e.:
cache1.getAdvancedCache().withScope("X")
cache1.put()
cache2.getAdvancedCache().withScope("X")
cache2.put()
By the way, I don't understand your comment of "refine this scope by adding
GlobalTransaction and so on". If we take the cache1/cache2 example and imagine that
these two would be called within a transaction, I'd imagine that the transaction
itself would act as a scope and so would not need for a particular scope definition? i.e.
tx.begin()
cache1.put()
cache2.put()
tx.commit()
At first glance, 5.0 would be the perfect target for this, but we'd need to take in
account for any requirements coming from Paul since HTTP session repl could greatly
benefit from it.
As far as 2LC is concerned, within a deployment multiple txs will be operating on same
caches so scoping could not be used. However different deployments could benefit from
deployment level scoping so that messages for different caches can be sent in parallel. In
AS environment different deployments could share the same 2LC cache manager and so the
configuration option above would not work, but I'd need a deployment level scope that
I'd use with cache.withScope().
Cheers,
On Jan 28, 2011, at 8:54 PM, Vladimir Blagojevic wrote:
> Forgot to mention that scopes require addition of SCOPE protocol to the
> stack. So we can probably do this on 5.0 branch only.
> On 11-01-28 4:52 PM, Vladimir Blagojevic wrote:
>> Hey,
>>
>> JGroups scopes [1] have been implemented for a while now and we should
>> think about using them in Infinispan. I looked around a bit to see how
>> we can use scope and satisfy requirements Manik outlined [2]. I think a
>> good idea would be to introduce scope method in InvocationContext? We
>> can start by implementing scope to return hash of cache name where
>> invocation originated and subsequently refine this scope by adding
>> GlobalTransaction and so on. Users, if they really need to scope their
>> calls could do so by attaching additional markers to InvocationContext
>> or smth similar.
>>
>> WDYT?
>>
>>
>> [1]https://issues.jboss.org/browse/JGRP-822
>> [2]https://issues.jboss.org/browse/ISPN-3
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Lead, Infinispan
http://www.infinispan.org
2:39 p.m.
Let's take a step back and ask ourselves what it is that we want to
scope ? Off of the top of my head the following things come to mind:
- HTTP sessions. I guess they're independent, so we can have a scope per
session. Do you guys currently use 1 Cache per session ? Then Galder's
suggestion below makes sense
- SFSBs: do we have 1 scope per bean ? What if bean X has a ref to bean
Y ? Would they have to have the same scope ?
- Hibernate 2LC: scope per entity ? I don't know how entities are broken
into tuples in the 2LC.
On 1/31/11 2:19 PM, Galder Zamarreño wrote:
Hmmmm, not sure about adding a method to IC, cos that would lead to
code like this which is very verbose:
cache.getAdvancedCache().getInvocationContextContainer().getInvocationContext().setScope("X")
As a user, I'd prefer something like:
cache.getAdvancedCache().withScope("X")
However, this would only be used to attach a particular scope. I envision a global, cache
manager level configuration such as that marks all caches in that cache manager to be
scoped by the cache name. That way, HTTP session repl code would need no changing at all
and with a single flag, they could make each individual cache to act in its own scope. The
code above would allow two caches to share the same scope for those situations where we
want sequential calls for the two caches, i.e.:
cache1.getAdvancedCache().withScope("X")
cache1.put()
cache2.getAdvancedCache().withScope("X")
cache2.put()
By the way, I don't understand your comment of "refine this scope by adding
GlobalTransaction and so on". If we take the cache1/cache2 example and imagine that
these two would be called within a transaction, I'd imagine that the transaction
itself would act as a scope and so would not need for a particular scope definition? i.e.
tx.begin()
cache1.put()
cache2.put()
tx.commit()
At first glance, 5.0 would be the perfect target for this, but we'd need to take in
account for any requirements coming from Paul since HTTP session repl could greatly
benefit from it.
As far as 2LC is concerned, within a deployment multiple txs will be operating on same
caches so scoping could not be used. However different deployments could benefit from
deployment level scoping so that messages for different caches can be sent in parallel. In
AS environment different deployments could share the same 2LC cache manager and so the
configuration option above would not work, but I'd need a deployment level scope that
I'd use with cache.withScope().
Cheers,
On Jan 28, 2011, at 8:54 PM, Vladimir Blagojevic wrote:
> Forgot to mention that scopes require addition of SCOPE protocol to the
> stack. So we can probably do this on 5.0 branch only.
> On 11-01-28 4:52 PM, Vladimir Blagojevic wrote:
>> Hey,
>>
>> JGroups scopes [1] have been implemented for a while now and we should
>> think about using them in Infinispan. I looked around a bit to see how
>> we can use scope and satisfy requirements Manik outlined [2]. I think a
>> good idea would be to introduce scope method in InvocationContext? We
>> can start by implementing scope to return hash of cache name where
>> invocation originated and subsequently refine this scope by adding
>> GlobalTransaction and so on. Users, if they really need to scope their
>> calls could do so by attaching additional markers to InvocationContext
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
2:42 p.m.
Even more fine grained than that, any 2 txs should be allowed to complete in parallel (if
they competed for locks, one would block on the other in the LockManager and wouldn't
progress in parallel anyway). This is the logic that allows us to use OOB for sync calls.
The hard part is, how do we encode this information into a scope ID. AIUI, scope is a
String? And you use simple string matching? In which case Bloom filters won't work
...
On 31 Jan 2011, at 13:39, Bela Ban wrote:
Let's take a step back and ask ourselves what it is that we want to
scope ? Off of the top of my head the following things come to mind:
- HTTP sessions. I guess they're independent, so we can have a scope per
session. Do you guys currently use 1 Cache per session ? Then Galder's
suggestion below makes sense
- SFSBs: do we have 1 scope per bean ? What if bean X has a ref to bean
Y ? Would they have to have the same scope ?
- Hibernate 2LC: scope per entity ? I don't know how entities are broken
into tuples in the 2LC.
On 1/31/11 2:19 PM, Galder Zamarreño wrote:
> Hmmmm, not sure about adding a method to IC, cos that would lead to code like this
which is very verbose:
>
>
cache.getAdvancedCache().getInvocationContextContainer().getInvocationContext().setScope("X")
>
> As a user, I'd prefer something like:
>
> cache.getAdvancedCache().withScope("X")
>
> However, this would only be used to attach a particular scope. I envision a global,
cache manager level configuration such as that marks all caches in that cache manager to
be scoped by the cache name. That way, HTTP session repl code would need no changing at
all and with a single flag, they could make each individual cache to act in its own scope.
The code above would allow two caches to share the same scope for those situations where
we want sequential calls for the two caches, i.e.:
>
> cache1.getAdvancedCache().withScope("X")
> cache1.put()
> cache2.getAdvancedCache().withScope("X")
> cache2.put()
>
> By the way, I don't understand your comment of "refine this scope by adding
GlobalTransaction and so on". If we take the cache1/cache2 example and imagine that
these two would be called within a transaction, I'd imagine that the transaction
itself would act as a scope and so would not need for a particular scope definition? i.e.
>
> tx.begin()
> cache1.put()
> cache2.put()
> tx.commit()
>
> At first glance, 5.0 would be the perfect target for this, but we'd need to take
in account for any requirements coming from Paul since HTTP session repl could greatly
benefit from it.
>
> As far as 2LC is concerned, within a deployment multiple txs will be operating on
same caches so scoping could not be used. However different deployments could benefit from
deployment level scoping so that messages for different caches can be sent in parallel. In
AS environment different deployments could share the same 2LC cache manager and so the
configuration option above would not work, but I'd need a deployment level scope that
I'd use with cache.withScope().
>
> Cheers,
>
> On Jan 28, 2011, at 8:54 PM, Vladimir Blagojevic wrote:
>
>> Forgot to mention that scopes require addition of SCOPE protocol to the
>> stack. So we can probably do this on 5.0 branch only.
>> On 11-01-28 4:52 PM, Vladimir Blagojevic wrote:
>>> Hey,
>>>
>>> JGroups scopes [1] have been implemented for a while now and we should
>>> think about using them in Infinispan. I looked around a bit to see how
>>> we can use scope and satisfy requirements Manik outlined [2]. I think a
>>> good idea would be to introduce scope method in InvocationContext? We
>>> can start by implementing scope to return hash of cache name where
>>> invocation originated and subsequently refine this scope by adding
>>> GlobalTransaction and so on. Users, if they really need to scope their
>>> calls could do so by attaching additional markers to InvocationContext
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Lead, Infinispan
http://www.infinispan.org
2:55 p.m.
On 1/31/11 2:33 PM, Manik Surtani wrote:
Just to clarify, there are 2 phases to this, if you will.
Phase 1: Infinispan makes use of scopes internally, for transactional and
non-transactional calls to gain greater concurrency in the way JGroups broadcasts
messages.
Phase 2: Allow user defined scopes.
Lets start with 1 and make sure we have a workable structure first. So to revisit the
problem (Vladimir/Bela - pls correct me if I am wrong), JGroups serializes the delivery of
messages off the wire per-sender.
Correct
This means that Node A may send many messages, but a given
recipient, say Node B, would queue up these messages and process them in order. This
means that even though we may have many threads on A doing non-related work, on disjoint
data sets, and although this happens in parallel on A, by the time they get to B they get
processed in order
Yes
To work around this, we use the OOB flag for synchronous messages
(since we hold locks for these until the RPC returns and we know that data from such
transactions cannot get reordered leading to inconsistent data).
The issue is that if we use _async_ mode, we cannot use OOB since things can get
reordered and hence inconsistent, and we _rely_ on JGroups processing messages in order to
maintain consistency.
Yes
Scopes helps us parallelize this. On a very basic level, if the
scope is a cache name, then at least transactions on different caches in the same cache
manager can be handled in parallel.
Correct
But it can get better: I propose using some mechanism to represent
the lock or locks touched by the transaction as your scope. So any other transaction
touching one or more of the same locks will be serialized, everything else could happen in
parallel. Perhaps a Bloom filter?
Do you always use locks, even if there's no TX, or the locking level is
NONE ?
If you have a TX, you could use the TX-ID as scope.
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
3:02 p.m.
A scope is a short.
When discussing scopes, this was seen as sufficient, but if you want
bloom filters this won't suffice...
The question is can you do with roughly 32'000 scopes per member ? If
this is the case, you could maintain a hashmap of scope-ids and bloom
filters (for example)...
If not, we need to revisit this. If you need a different type, let's
discuss this, but it should be a type that can be used as a key into a
hashmap...
On 1/31/11 2:42 PM, Manik Surtani wrote:
Even more fine grained than that, any 2 txs should be allowed to
complete in parallel (if they competed for locks, one would block on the other in the
LockManager and wouldn't progress in parallel anyway). This is the logic that allows
us to use OOB for sync calls. The hard part is, how do we encode this information into a
scope ID. AIUI, scope is a String? And you use simple string matching? In which case
Bloom filters won't work ...
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
3:09 p.m.
On 31 Jan 2011, at 13:55, Bela Ban wrote:
> But it can get better: I propose using some mechanism to represent the lock or locks
touched by the transaction as your scope. So any other transaction touching one or more
of the same locks will be serialized, everything else could happen in parallel. Perhaps a
Bloom filter?
Do you always use locks, even if there's no TX, or the locking level is
NONE ?
If you skip locking then things could get reordered/inconsistent even on one node. So it
is irrelevant in a distributed sense.
If you have a TX, you could use the TX-ID as scope.
I know this is what we originally thought, but I think this might be insufficient.
Different scopes are delivered in parallel. So tx1 and tx2 *could* be delivered in
parallel, even though they touch the same keys. E.g., they happen in sequence on node A
(due to locking), but since we're talking async RPC, the 1PC prepare/commit messages
could get reordered and applied on B in a different order. And this is wrong.
Some form of understanding which keys are touched, and grouping transactions that share
locks into the same scope, is what is needed.
If there are external hints, e.g., http session IDs as you suggested, this makes it much
easier.
Cheers
Manik
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Lead, Infinispan
http://www.infinispan.org
1:54 p.m.
Yes. Essentially the same logic as to the locks acquired.
On 3 Feb 2011, at 12:27, Mircea Markus wrote:
>
> Some form of understanding which keys are touched, and grouping transactions that
share locks into the same scope, is what is needed.
What if we run outside tx. Each key would be in its own scope?
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Lead, Infinispan
http://www.infinispan.org
4852
days inactive
4858
days old
infinispan-dev@lists.jboss.org
10 comments
5 participants
participants (5)
-
Bela Ban -
Galder Zamarreño -
Manik Surtani -
Mircea Markus -
Vladimir Blagojevic