Hi guys, I've just had a chat with the CloudTM team(CC) around the integration of TOB/TOA[1] into Infinispan. Here are some points: - the TOB and TOA code has been reviewed in detail by us. The only part missing is the state transfer integration - there's not a lot of sense in integrating TOB/TOM over the existing state transfer as we would not back port that to 5.1 and it would be dropped in 5.2 - CloudTM would rebase the TOB/TOA work on top of the alpha NBST[2](ATM planned at the end of next week/3 Aug) and we'll integrate that - first releases of the TOB/TOA would be marked as experimental in 5.2 How does that sound? Cheers, Mircea [1] TOA used to be referred to as TOM (M from multicast). In JGroups terminology that's an Anycast, so we decided to be consistent with that and use TOAnycast. [2] https://community.jboss.org/wiki/Non-blockingStateTransfer _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Does total order make a difference for NBST, in another words is it easier to implement NBST having total order guarantees?

Sounds good to me, we should have a little more breathing room after the NBST alpha to look at the state transfer integration. Cheers Dan On Tue, Jul 24, 2012 at 6:08 PM, Mircea Markus <mircea.markus(a)jboss.com <mailto:mircea.markus@jboss.com>> wrote: Hi guys, I've just had a chat with the CloudTM team(CC) around the integration of TOB/TOA[1] into Infinispan. Here are some points: - the TOB and TOA code has been reviewed in detail by us. The only part missing is the state transfer integration - there's not a lot of sense in integrating TOB/TOM over the existing state transfer as we would not back port that to 5.1 and it would be dropped in 5.2 - CloudTM would rebase the TOB/TOA work on top of the alpha NBST[2](ATM planned at the end of next week/3 Aug) and we'll integrate that - first releases of the TOB/TOA would be marked as experimental in 5.2 How does that sound? Cheers, Mircea [1] TOA used to be referred to as TOM (M from multicast). In JGroups terminology that's an Anycast, so we decided to be consistent with that and use TOAnycast. [2] https://community.jboss.org/wiki/Non-blockingStateTransfer _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org <mailto:infinispan-dev@lists.jboss.org> https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Hi Paolo, the pending transactions were indeed a problem but this was solved by issues ISPN-2306 [1] and ISPN-2312 [2]. Regarding point 1.5, this was implemented in StateConsumerImpl.applyTransactions(..) and StatePorviderImpl.getTransactionsForSegments(..). There was indeed a missing piece that was corrected by ISPN-2306 (the lookedupEntries field was not populated for transferred transactions, this is solved by re-executing prepare on the new node). ISPN-2312 also solves an issue with acquiring locks by these transferred transactions, now we should be ok. Regarding point 1.7, if the data is not available yet on a new owner the write skew check will not detect any issue on this node indeed but the check will fail on the other owners and cause a rollback on all nodes anyway, so we should be safe here. Is the NBST alternative you mentioned applicable to TO only? We are very interested to hear about you plans even if they are in an early stage. Thanks, Adrian [1] ISPN-2306 Remove the code that resends PrepareCommands [https://issues.jboss.org/browse/ISPN-2306] [2] ISPN-2312 TransactionTable does not compute minViewId correctly after NBST was introduced [https://issues.jboss.org/browse/ISPN-2312] On 09/14/20 12 07:51 PM, Paolo Romano wrote: > Hi all, > > with Pedro we have been reasoning on the integration of TO-based > replication protocols, and a few questions popped out. > > We may be missing something here, but it seems that the current NBST > implementation is still not providing support for pending > transactions, namely transactions that are prepared but not yet > committed at the time in which the node receives a state transfer > request. We have tried to figure out how you plan to do this by > checking the design document: > > https://community.jboss.org/wiki/Non-BlockingStateTransferV2 > > but we still have some doubts. The relevant extract seems to be the > following: > > 1.3 For new segments, it asks one of the current owners (the > donor) for the transactions and locks. > 1.4 The donor only replies with the transactions and locks after > it has installed the new toplogy. We need this to make sure a proper > pending CH is also installed on the donor and request forwarding to > the new pending owners is in place (forwarding is explained in > request handling section). > 1.5 The transactions and locks are applied on acceptor. > 1.6 The acceptor requests new data segments from donor > asynchronously. > 1.7 Unblock all incoming commands. We are now prepared to > process commands although some data segments are still flowing in. > > In point 1.7, you say that you can already process commands for data > segments that a node has not received yet. However, if the command > is, say, a prepare for a key in a missing data segment and the > transaction is requesting a validation (write-skew check), you would > still need to block as you need the most updated data version to > validate it. Are we getting it right? > > Also, it seems that point 1.5 has not been coded yet. We are asking > this, as these functionalities are likely to be useful also for the > NBST version used by TO-based replication protocols. Thus, it'd > probably be better to wait for these parts to be stable and re-use > them, instead than implementing them from scratch and end-up possibly > with conflicting/incompatible implementations. When do you plan to > have this functionality implemented? > > Finally, I wanted to point out that Roberto and Sebastiano have been > thinking about an alternative version of the NBST, which should > further reduce the blocking time of transactions. They're currently > still at the design stage, and are working to prepare a document that > we would like to share with you to get feedback/comments etc. We plan > to have a draft of the algorithm by Sept. 20. > > Cheers, > > Paolo > > PS: I'll be travelling starting tomorrow and during next week, so I > may not be responding to emails very quickly. > > > On 7/25/12 12:16 PM, Dan Berindei wrote: >> Sounds good to me, we should have a little more breathing room after >> the NBST alpha to look at the state transfer integration. >> >> Cheers >> Dan >> >> >> On Tue, Jul 24, 2012 at 6:08 PM, Mircea Markus >> <mircea.markus(a)jboss.com <mailto:mircea.markus@jboss.com>> wrote: >> >> Hi guys, >> >> I've just had a chat with the CloudTM team(CC) around the >> integration of TOB/TOA[1] into Infinispan. Here are some points: >> - the TOB and TOA code has been reviewed in detail by us. The >> only part missing is the state transfer integration >> - there's not a lot of sense in integrating TOB/TOM over the >> existing state transfer as we would not back port that to 5.1 >> and it would be dropped in 5.2 >> - CloudTM would rebase the TOB/TOA work on top of the alpha >> NBST[2](ATM planned at the end of next week/3 Aug) and we'll >> integrate that >> - first releases of the TOB/TOA would be marked as experimental >> in 5.2 >> >> How does that sound? >> >> Cheers, >> Mircea >> >> [1] TOA used to be referred to as TOM (M from multicast). In >> JGroups terminology that's an Anycast, so we decided to be >> consistent with that and use TOAnycast. >> [2] https://community.jboss.org/wiki/Non-blockingStateTransfer >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> <mailto:infinispan-dev@lists.jboss.org> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Hi Adrian, Thank you for the work =). Please see inline some comments. Cheers, Pedro On 24-09-2012 09:35, Adrian Nistor wrote: Hi Paolo, the pending transactions were indeed a problem but this was solved by issues ISPN-2306 [1] and ISPN-2312 [2]. Regarding point 1.5, this was implemented in StateConsumerImpl.applyTransactions(..) and StatePorviderImpl.getTransactionsForSegments(..). There was indeed a missing piece that was corrected by ISPN-2306 (the lookedupEntries field was not populated for transferred transactions, this is solved by re-executing prepare on the new node). ISPN-2312 also solves an issue with acquiring locks by these transferred transactions, now we should be ok. Regarding point 1.7, if the data is not available yet on a new owner the write skew check will not detect any issue on this node indeed but the check will fail on the other owners and cause a rollback on all nodes anyway, so we should be safe here. I'm not sure if this works fine when the number of owners is equals to 1. Isn't it possible a node send the prepapre right after the Rebalance_Start to the joiner (the new owner)? If the joiner does not have the data, it cannot perform a true write skew check. Am I missing something?

Is the NBST alternative you mentioned applicable to TO only? We are very interested to hear about you plans even if they are in an early stage. About the NBST for the TO, I have to block transactions that are writing in keys of incoming segments. I didn't find any work around for this issue. Thanks, Adrian [1] ISPN-2306 Remove the code that resends PrepareCommands [ https://issues.jboss.org/browse/ISPN-2306] [2] ISPN-2312 TransactionTable does not compute minViewId correctly after NBST was introduced [https://issues.jboss.org/browse/ISPN-2312] On 09/14/20 12 07:51 PM, Paolo Romano wrote: Hi all, with Pedro we have been reasoning on the integration of TO-based replication protocols, and a few questions popped out. We may be missing something here, but it seems that the current NBST implementation is still not providing support for pending transactions, namely transactions that are prepared but not yet committed at the time in which the node receives a state transfer request. We have tried to figure out how you plan to do this by checking the design document: https://community.jboss.org/wiki/Non-BlockingStateTransferV2 but we still have some doubts. The relevant extract seems to be the following: 1.3 For new segments, it asks one of the current owners (the donor) for the transactions and locks. 1.4 The donor only replies with the transactions and locks after it has installed the new toplogy. We need this to make sure a proper pending CH is also installed on the donor and request forwarding to the new pending owners is in place (forwarding is explained in request handling section). 1.5 The transactions and locks are applied on acceptor. 1.6 The acceptor requests new data segments from donor asynchronously. 1.7 Unblock all incoming commands. We are now prepared to process commands although some data segments are still flowing in. In point 1.7, you say that you can already process commands for data segments that a node has not received yet. However, if the command is, say, a prepare for a key in a missing data segment and the transaction is requesting a validation (write-skew check), you would still need to block as you need the most updated data version to validate it. Are we getting it right? Also, it seems that point 1.5 has not been coded yet. We are asking this, as these functionalities are likely to be useful also for the NBST version used by TO-based replication protocols. Thus, it'd probably be better to wait for these parts to be stable and re-use them, instead than implementing them from scratch and end-up possibly with conflicting/incompatible implementations. When do you plan to have this functionality implemented? Finally, I wanted to point out that Roberto and Sebastiano have been thinking about an alternative version of the NBST, which should further reduce the blocking time of transactions. They're currently still at the design stage, and are working to prepare a document that we would like to share with you to get feedback/comments etc. We plan to have a draft of the algorithm by Sept. 20. Cheers, Paolo PS: I'll be travelling starting tomorrow and during next week, so I may not be responding to emails very quickly. On 7/25/12 12:16 PM, Dan Berindei wrote: Sounds good to me, we should have a little more breathing room after the NBST alpha to look at the state transfer integration. Cheers Dan On Tue, Jul 24, 2012 at 6:08 PM, Mircea Markus <mircea.markus(a)jboss.com&gt;wrote: > Hi guys, > > I've just had a chat with the CloudTM team(CC) around the integration > of TOB/TOA[1] into Infinispan. Here are some points: > - the TOB and TOA code has been reviewed in detail by us. The only part > missing is the state transfer integration > - there's not a lot of sense in integrating TOB/TOM over the existing > state transfer as we would not back port that to 5.1 and it would be > dropped in 5.2 > - CloudTM would rebase the TOB/TOA work on top of the alpha NBST[2](ATM > planned at the end of next week/3 Aug) and we'll integrate that > - first releases of the TOB/TOA would be marked as experimental in 5.2 > > How does that sound? > > Cheers, > Mircea > > [1] TOA used to be referred to as TOM (M from multicast). In JGroups > terminology that's an Anycast, so we decided to be consistent with that and > use TOAnycast. > [2] https://community.jboss.org/wiki/Non-blockingStateTransfer > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing listinfinispan-dev@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing listinfinispan-dev@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing listinfinispan-dev@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 9/26/12 4:30 PM, Dan Berindei wrote: > > When we start the rebalance, we always keep the previous owner. So the > joiner becomes a backup owner, but the previous owner is still the > primary owner until the rebalance is done (which would guarantee the > joiner has the data). > > We can't do the write skew check if the previous owner left, but then > it was the only owner so we don't have any other copy of the data in > the cluster in order to perform the write skew check. so, if I understand right, it happens in this way (simplified)? node1: primary owner ==> key1, key2 node2: joins and the new consistent hash says that key 2 should be in node2 ... event: rebalance_start node1: primary owner ==> key1, key2 node2: backup owner ==> key2 ... event: ch_update node1: primary owner ==> key1 node2: primary owner ==> key2 _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev