I wrote the original test, to simulate a transactional write occuring (still in prepare phase) while a node joins. So to this end, I think we can safely run the test without L1 caching as the behaviour we are trying to test is still correct. However I'm puzzled that the L1 invalidation kills this - the L1 invalidation should be a fail-fast mechanism.

On 22 Nov 2010, at 20:24, Vladimir Blagojevic wrote: > Well we have L1 data in the same container as all other data and as such > is treated no different that non-L1 data - including locking. So the issue is that a reader has locked the entry in memory because it is in L1, and this is why the transactional write from elsewhere cannot be applied? Readers don't block writers... > > On 10-11-22 1:55 PM, Manik Surtani wrote: >> I wrote the original test, to simulate a transactional write occuring (still in prepare phase) while a node joins. So to this end, I think we can safely run the test without L1 caching as the behaviour we are trying to test is still correct. >> >> However I'm puzzled that the L1 invalidation kills this - the L1 invalidation should be a fail-fast mechanism. >> > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 10-11-23 8:11 AM, Manik Surtani wrote: > Is this related? > > https://jira.jboss.org/browse/ISPN-595 > > Lets have a chat when you're online later today... > No, this is different I believe. I do not see from his stack trace that InvalidateCommand is involved.

Here is a JIRA https://jira.jboss.org/browse/ISPN-799

I like option 1 as well. I am interested in details of how this can be done. How do we set lock acquisition timeout for InvalidateL1Command, issue command to abort specific tx etc etc?

Cheers. On 10-11-24 12:36 PM, Manik Surtani wrote: > Right, I've spotted it. The test failure itself is intermittent due to the way addresses are organised in the hash wheel, so you are correct that it is a timing issue. Anyway, it still is a very real problem. Just to re-iterate and to make sure we are talking about the same thing: > > 1. View is {A, B, C} > 2. K is mapped to {A, B} > 3. A tx starts to update K, and is prepared. Locks now held for K on {A, B} > 4. D joins. D is placed on the hash wheel between A and B. So the new view is {A, D, B, C} > 5. As per the test (artificial, I know, but could still happen), the tx waits for a long time before committing. In the case of the test, artificially waits until D has finished joining before committing, by use of a latch. > 6. D never joins as even though it receives the prepare for the tx and could potentially commit itself (as a new owner), it fails as it is unable to invalidate K on B. > > There are a few solutions here: > > 1) This is pretty easy to detect. Attempt to acquire the lock with a smaller lock acquisition timeout and if the transaction is still stuck, abort the transaction and proceed with the join. > 2) If the blocking node is *not* the transaction originator (as in this case: the tx was started on A), then just force lock removal and tx rollback on B *only*. Let the tx complete on A, since the new joiner will receive the transactional event and will be able to apply it as a new owner. > > My vote is to go for solution 1 - a bit more crude, but 2 would be very complex to implement. And even then, would only solve for the invalidation being blocked on a node that did not originate the transaction. E.g., the tx originated on A but the lock issue was on B. If, however, the tx originated on B, *and* B no longer owns the entry in question, then 2 is no longer a solution and the only solution would be 1. > > Thoughts? > > Cheers > Manik > > PS: Do we have a JIRA for this?

Hey, On 10-11-26 12:15 PM, Manik Surtani wrote: > >> I like option 1 as well. I am interested in details of how this can be done. How do we set lock acquisition timeout for InvalidateL1Command, issue command to abort specific tx etc etc? > Propose a design, we can review it here. > Ok, how about this: 1) JoinTask issues invalidateL1Command as it is now in JoinTask 2) In case of TimeoutException where cause is inability to acquire lock L on key K then issue rollback for tx T

3) Repeat 1 4) Loop 1-3 until success or timeout Questions: 1) For step2, can we simply do commandsFactory.buildRollbackCommand(globalTx) and broadcast this message? 2) In order to achieve the above we have to add globalTx as parameter to TimeoutException where globalTx is a the tx associated with T. Is there another way to rollback a user transaction from JoinTask, another mechanism perhaps?

3) I would also move JOIN_REHASH_END end message sent to signal successful join *after* invalidation is done. Agree? Otherwise, we can have overlapping joins.

> > Perhaps InvalidateL1Command should carry a 'force' flag and if this is true, then the recipient attempts to 'force' the lock by causing the existing lock owner to roll back. If we can do this that would be great! Why not force L1Invalidate immediately? Users can redo their rolled back transactions. Surely, invalidate L1 , a system level command has to have precedence over any user tx, right? Is there a mechanism in place to do this already?

On 10-11-27 12:29 AM, Manik Surtani wrote: > Nope, no such mechanism. This needs to be written. What if we add a new lockAndRecord method in LockManager with signature: LockingResult lockAndRecord(Object key, InvocationContext ctx, boolean force) throws InterruptedException; and LockingResult is enum or a simple class with fields: boolean acquired; Object owner;

In case of force, I believe we can atomically change ownership of a lock, assign owner to previous owner of a lock and thus provide callers of lockAndRecord with a mechanism to issue a rollback on a correct global tx in case global tx was previous owner. In normal cases, just as it is now, we assign value of acquired and in case !acquired owner is current lock owner. Questions: 1) If an owner was a thread, not a global tx, what are the consequences for that thread - is it subsequently going to be prevented from writing to value key by the locked key?

What I'm saying It might be very wrong, but trying :) Isn't it possible to make the invalidation of K on B part of the transaction commit? I.e. the invalidation on B sees that K is locked by an tx that is not committed and it skips the invalidation. When tx commits (second phase) it multicasts the commit message to all the lock owners_including_ B (all the owners == {A, B, D}). When CommitCOmmand is received, B checks weather or not it still is an data owner for each key: if it is the it applies it, otherwise it removes it.

On 10-11-30 10:49 AM, Vladimir Blagojevic wrote: > I like your solution. It seems to be less disruptive to ongoing > transactions then the other two solutions. > > How would you safely detect that K is locked by another tx and thus skip > locking? I do *not* think I can do the following in LockingInterceptor: public Object visitInvalidateCommand(InvocationContext ctx, InvalidateCommand command) throws Throwable { try { if (command.getKeys() != null) { for (Object key : command.getKeys()) { if(!lockManager.isLocked(key)) entryFactory.wrapEntryForWriting(ctx, key, false, true, false, false, false); } } return invokeNextInterceptor(ctx, command);

} catch (Throwable te) { return cleanLocksAndRethrow(ctx, te); } finally { doAfterCall(ctx); } }

On 10-11-30 1:35 PM, Mircea Markus wrote: > On 30 Nov 2010, at 14:30, Vladimir Blagojevic wrote: > >> On 10-11-30 10:49 AM, Vladimir Blagojevic wrote: >>> I like your solution. It seems to be less disruptive to ongoing >>> transactions then the other two solutions. >>> >>> How would you safely detect that K is locked by another tx and thus skip >>> locking? >> I do *not* think I can do the following in LockingInterceptor: >> >> public Object visitInvalidateCommand(InvocationContext ctx, InvalidateCommand command) throws Throwable { >> try { >> if (command.getKeys() != null) { >> for (Object key : command.getKeys()) { >> if(!lockManager.isLocked(key)) >> entryFactory.wrapEntryForWriting(ctx, key, false, true, false, false, false); >> } >> } >> return invokeNextInterceptor(ctx, command); > Perhaps you only want to run invokeNext for the keys for which you acquired locks? I would love to but I do not see a method wrapEntryforWritingIfYouCan :-) What do you do in these kinds of situations? Set timeout to 10 msec and catch TimeoutException?

Vladimir

On 30 Nov 2010, at 16:51, Mircea Markus wrote: > > On 30 Nov 2010, at 16:42, Vladimir Blagojevic wrote: > >> On 10-11-30 1:35 PM, Mircea Markus wrote: >>> On 30 Nov 2010, at 14:30, Vladimir Blagojevic wrote: >>> >>>> On 10-11-30 10:49 AM, Vladimir Blagojevic wrote: >>>>> I like your solution. It seems to be less disruptive to ongoing >>>>> transactions then the other two solutions. >>>>> >>>>> How would you safely detect that K is locked by another tx and thus skip >>>>> locking? >>>> I do *not* think I can do the following in LockingInterceptor: >>>> >>>> public Object visitInvalidateCommand(InvocationContext ctx, InvalidateCommand command) throws Throwable { >>>> try { >>>> if (command.getKeys() != null) { >>>> for (Object key : command.getKeys()) { >>>> if(!lockManager.isLocked(key)) >>>> entryFactory.wrapEntryForWriting(ctx, key, false, true, false, false, false); >>>> } >>>> } >>>> return invokeNextInterceptor(ctx, command); >>> Perhaps you only want to run invokeNext for the keys for which you acquired locks? >> >> I would love to but I do not see a method wrapEntryforWritingIfYouCan :-) What do you do in these kinds of situations? Set timeout to 10 msec and catch TimeoutException? > My point is you don't want to invalidate(invoke next) keys for which you don't have the locks. These would be invalidated(i.e. removed) at commit time. Right, provided the tx knows to invalidate these keys at commit (or rollback). Right?

-- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 30 Nov 2010, at 17:08, Mircea Markus wrote: > > On 30 Nov 2010, at 17:03, Manik Surtani wrote: > >> >> On 30 Nov 2010, at 16:51, Mircea Markus wrote: >> >>> >>> On 30 Nov 2010, at 16:42, Vladimir Blagojevic wrote: >>> >>>> On 10-11-30 1:35 PM, Mircea Markus wrote: >>>>> On 30 Nov 2010, at 14:30, Vladimir Blagojevic wrote: >>>>> >>>>>> On 10-11-30 10:49 AM, Vladimir Blagojevic wrote: >>>>>>> I like your solution. It seems to be less disruptive to ongoing >>>>>>> transactions then the other two solutions. >>>>>>> >>>>>>> How would you safely detect that K is locked by another tx and thus skip >>>>>>> locking? >>>>>> I do *not* think I can do the following in LockingInterceptor: >>>>>> >>>>>> public Object visitInvalidateCommand(InvocationContext ctx, InvalidateCommand command) throws Throwable { >>>>>> try { >>>>>> if (command.getKeys() != null) { >>>>>> for (Object key : command.getKeys()) { >>>>>> if(!lockManager.isLocked(key)) >>>>>> entryFactory.wrapEntryForWriting(ctx, key, false, true, false, false, false); >>>>>> } >>>>>> } >>>>>> return invokeNextInterceptor(ctx, command); >>>>> Perhaps you only want to run invokeNext for the keys for which you acquired locks? >>>> >>>> I would love to but I do not see a method wrapEntryforWritingIfYouCan :-) What do you do in these kinds of situations? Set timeout to 10 msec and catch TimeoutException? >>> My point is you don't want to invalidate(invoke next) keys for which you don't have the locks. These would be invalidated(i.e. removed) at commit time. >> >> Right, provided the tx knows to invalidate these keys at commit (or rollback). Right? > At commit/rollback, the tx can iterate over they keys involved (ones brought here by prepare): if the key is no longer mapped to this node then simply remove it; otherwise apply the change. Not unless L1 is enabled and you *want* the entry in L1 there.

-- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 1 Dec 2010, at 01:17, Mircea Markus wrote: > > On 30 Nov 2010, at 17:25, Manik Surtani wrote: > >> >> On 30 Nov 2010, at 17:08, Mircea Markus wrote: >> >>> >>> On 30 Nov 2010, at 17:03, Manik Surtani wrote: >>> >>>> >>>> On 30 Nov 2010, at 16:51, Mircea Markus wrote: >>>> >>>>> >>>>> On 30 Nov 2010, at 16:42, Vladimir Blagojevic wrote: >>>>> >>>>>> On 10-11-30 1:35 PM, Mircea Markus wrote: >>>>>>> On 30 Nov 2010, at 14:30, Vladimir Blagojevic wrote: >>>>>>> >>>>>>>> On 10-11-30 10:49 AM, Vladimir Blagojevic wrote: >>>>>>>>> I like your solution. It seems to be less disruptive to ongoing >>>>>>>>> transactions then the other two solutions. >>>>>>>>> >>>>>>>>> How would you safely detect that K is locked by another tx and thus skip >>>>>>>>> locking? >>>>>>>> I do *not* think I can do the following in LockingInterceptor: >>>>>>>> >>>>>>>> public Object visitInvalidateCommand(InvocationContext ctx, InvalidateCommand command) throws Throwable { >>>>>>>> try { >>>>>>>> if (command.getKeys() != null) { >>>>>>>> for (Object key : command.getKeys()) { >>>>>>>> if(!lockManager.isLocked(key)) >>>>>>>> entryFactory.wrapEntryForWriting(ctx, key, false, true, false, false, false); >>>>>>>> } >>>>>>>> } >>>>>>>> return invokeNextInterceptor(ctx, command); >>>>>>> Perhaps you only want to run invokeNext for the keys for which you acquired locks? >>>>>> >>>>>> I would love to but I do not see a method wrapEntryforWritingIfYouCan :-) What do you do in these kinds of situations? Set timeout to 10 msec and catch TimeoutException? >>>>> My point is you don't want to invalidate(invoke next) keys for which you don't have the locks. These would be invalidated(i.e. removed) at commit time. >>>> >>>> Right, provided the tx knows to invalidate these keys at commit (or rollback). Right? >>> At commit/rollback, the tx can iterate over they keys involved (ones brought here by prepare): if the key is no longer mapped to this node then simply remove it; otherwise apply the change. >> >> Not unless L1 is enabled and you *want* the entry in L1 there. > even if the entry is removed from L1 this is not incorrect (at most sub-optimal). Of course, but you can't expect *every* transaction to *always* remove entries it touches from L1. This completely defeats the purpose of L1.

The transaction needs to be made aware that some of the keys it touched has been exposed to rehashing and hence should be removed from L1 when the tx completes. I don't think this is hard to do: what we'd need is: * A subclass of the TxInvocationContext (DistributedTxInvocationContext?) which maintains a set of keys potentially rehashed * An L1InvalidationCommand would try and invalidate a key. If it is unable to lock this key _and_ the lock is held by a transaction, add the key to the given transaction's context's potentially rehashed key set * When the transaction completes, it invalidates any keys before releasing locks. WDYT? -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Of course, but you can't expect *every* transaction to *always* remove entries it touches from L1. This completely defeats the purpose of L1. The transaction needs to be made aware that some of the keys it touched has been exposed to rehashing and hence should be removed from L1 when the tx completes. I don't think this is hard to do: what we'd need is: * A subclass of the TxInvocationContext (DistributedTxInvocationContext?) which maintains a set of keys potentially rehashed * An L1InvalidationCommand would try and invalidate a key. If it is unable to lock this key _and_ the lock is held by a transaction, add the key to the given transaction's context's potentially rehashed key set * When the transaction completes, it invalidates any keys before releasing locks. WDYT?

On 1 Dec 2010, at 14:56, Vladimir Blagojevic wrote: > I don't know guys but although this sounds good it is a bit complicated > you have to admit! > > I have a stupid question. Why do we allow transactions to lock keys that > are in L1? I mean these keys are transient on non-owner nodes so why > lock them to begin with? Simpler design/impl as it is in-line with replication as well. Happy to consider other prototype imls. Cheers Manik

> I would love to but I do not see a method wrapEntryforWritingIfYouCan :-) What do you do in these kinds of situations? Set timeout to 10 msec and catch TimeoutException? My point is you don't want to invalidate(invoke next) keys for which you don't have the locks. These would be invalidated(i.e. removed) at commit time.

> > 1) This is pretty easy to detect. Attempt to acquire the lock with a smaller lock acquisition timeout and if the transaction is still stuck, abort the transaction and proceed with the join. > 2) If the blocking node is *not* the transaction originator (as in this case: the tx was started on A), then just force lock removal and tx rollback on B *only*. Let the tx complete on A, since the new joiner will receive the transactional event and will be able to apply it as a new owner. What I'm saying It might be very wrong, but trying :) Isn't it possible to make the invalidation of K on B part of the transaction commit? I.e. the invalidation on B sees that K is locked by an tx that is not committed and it skips the invalidation. When tx commits (second phase) it multicasts the commit message to all the lock owners _including_ B (all the owners == {A, B, D}). When CommitCOmmand is received, B checks weather or not it still is an data owner for each key: if it is the it applies it, otherwise it removes it.