> > Well, probably not, because we only want to send keys to nodes that > actually need to store them... > Sending the whole tx as a multicast would certainly be more efficient than what we do now with lots of targets. With unicasts we could send only the minimum required data to each target, but that computation would be complex and error-prone.

On 3/13/12 6:35 AM, Manik Surtani wrote: > > On 12 Mar 2012, at 08:03, Dan Berindei wrote: > >>> >>> Well, probably not, because we only want to send keys to nodes that >>> actually need to store them... >>> >> >> Sending the whole tx as a multicast would certainly be more efficient >> than what we do now with lots of targets. >> With unicasts we could send only the minimum required data to each >> target, but that computation would be complex and error-prone. > > Well, this is what ANYCAST was all about initially, where JGroups would decide, based on the recipient list versus the total cluster size, on whether to send multiple unicasts or a multicast. But we didn't end up doing this in the end, perhaps we need to revisit. > > Bela, I'm guessing this thread was prompted by the poor performance in DIST that was reported, right? I'd like to profile the test provided to understand where we should be looking in the first place. E.g., is it the fact that we have too many RPCs? Or maybe a locking/concurrency issue elsewhere, etc. If you have done any of this analysis already, we should talk about that. Yes. My current findings are that we're doing unneeded sync RPCs even if <async.../> is defined. I'll run this with the latest Infinispan and see if it's still the case (Galder mentioned this was gone in 5.2 master).

Also, there should be performance improvements by locking only the primary owners (changes by Mircea and/or Dan), so I'll need to re-run with the latest and greatest of Infinispan (and JGroups). -- Bela Ban, JGroups lead (http://www.jgroups.org) _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 13 Mar 2012, at 03:28, Bela Ban wrote: > > > On 3/13/12 6:35 AM, Manik Surtani wrote: >> >> On 12 Mar 2012, at 08:03, Dan Berindei wrote: >> >>>> >>>> Well, probably not, because we only want to send keys to nodes that >>>> actually need to store them... >>>> >>> >>> Sending the whole tx as a multicast would certainly be more efficient >>> than what we do now with lots of targets. >>> With unicasts we could send only the minimum required data to each >>> target, but that computation would be complex and error-prone. >> >> Well, this is what ANYCAST was all about initially, where JGroups would decide, based on the recipient list versus the total cluster size, on whether to send multiple unicasts or a multicast. But we didn't end up doing this in the end, perhaps we need to revisit. >> >> Bela, I'm guessing this thread was prompted by the poor performance in DIST that was reported, right? I'd like to profile the test provided to understand where we should be looking in the first place. E.g., is it the fact that we have too many RPCs? Or maybe a locking/concurrency issue elsewhere, etc. If you have done any of this analysis already, we should talk about that. > > > Yes. My current findings are that we're doing unneeded sync RPCs even if > <async.../> is defined. I'll run this with the latest Infinispan and see > if it's still the case (Galder mentioned this was gone in 5.2 master). > Also, there should be performance improvements by locking only the > primary owners (changes by Mircea and/or Dan), so I'll need to re-run > with the latest and greatest of Infinispan (and JGroups). Any such changes should also be in 5.1.x. If you do see any sync RPCs (except perhaps remote GETs) when running in async mode, then that is almost certainly a bug.

On Sat, Mar 10, 2012 at 7:07 PM, Bela Ban <bban(a)redhat.com&gt; wrote: > Can you confirm that my understanding of how DIST works is correct ? > > > #1 Non transactional modifications > - E.g. a PUT > - The PUT is sent to the primary owner (e.g. B) and all backup owners > (e.g. C) and is applied immediately, with only local lock acquisition > (lock-put-unlock) This is correct, the key lock is acquired in parallel on all the owners, so two concurrent PUTs could result in different owners having different values.

On Sat, Mar 10, 2012 at 7:07 PM, Bela Ban&lt;bban(a)redhat.com&gt; wrote: > Can you confirm that my understanding of how DIST works is correct ? > > > #1 Non transactional modifications > - E.g. a PUT > - The PUT is sent to the primary owner (e.g. B) and all backup owners > (e.g. C) and is applied immediately, with only local lock acquisition > (lock-put-unlock) This is correct, the key lock is acquired in parallel on all the owners, so two concurrent PUTs could result in different owners having different values.

> (or is the PUT only sent to B, which in turn then updates C?) > > > #2 Transactional modifications > - The modifications involve a bunch of keys > - When the TX commits: > - A PREPARE message with the relevant keys is sent to all primary > owners P (to the backup owners as well?) The PREPARE is sent to the backup owners as well. The complete list of modifications is sent to all the owners, primary or backup.

> If so, then I can assume that a transactional modification touching a > number of keys will almost always touch *all* nodes ? Example: > - We have 10 nodes > - numOwners = 2 > - If we have a good consistent hash, I can assume that I have to modifiy > 5 different keys (10 / 2) on average in a TX to touch *all* nodes in the > cluster with the PREPARE/COMMIT phase, correct ? > You have to modify *at least* 5 different keys to touch all the nodes. Some keys will share the primary owner, but also one key's primary owner can be another's backup owner - so the chances to touch all nodes with only 5 keys are very low. It turns out the probability of a tx touching all the nodes depends on whether we have virtual nodes enabled or not - with lots of virtual nodes all the {(i, j) where 0<= i != j< numNodes} combinations are valid, whereas without virtual nodes only numNodes combinations are allowed - {(i, (i+1)%clusterSize) where 0<= i< numNodes}. I've run some simulations with IPython assuming virtual nodes, and these are the number of keys per tx required to touch all the nodes 50% of the time for a few representative cluster sizes: 4: 4, 8: 10, 16: 25, 32: 61

These are the numbers of keys required to touch numNodes/2 nodes 50% of the time: 4: 1, 8: 2, 16: 5, 32: 11

And these are the numbers of keys required to touch numNodes/2 nodes 90% of the time: 4: 1, 8: 3, 16: 7, 32: 13 > If my last statement is correct, is it safe to assume that with DIST and > transactional modifications, I will have a lot of TX contention / > collisions ? > Not sure what you mean by lot of TX contention - lock contention should only depend on the dataset size, unless we use lock striping, in which case it depends on the configured concurrency level.

Network traffic would probably increase quadratically with the number of keys/tx because we do send all the modifications to all the owners, but we could fix that by either transmitting only the relevant bits to each owner

or by switching to multicast after the number of targets hits a threshold.

> If this is correct, this would IMO lay even more importance onto the > work done by the Cloud-TM team, replacing 2PC with total order. Also, if > we touch almost all nodes, would it make sense to use SEQUENCER for > *all* updates ? Would this obviliate the need for TOM (total order for > partial replication) ? > I don't enough about Cloud-TM to comment here, but it appears SEQUENCER requires the coordinator to receive forward requests from all the nodes and send nothing back on the unicast channel.

Sending the whole tx as a multicast would certainly be more efficient than what we do now with lots of targets.

On Thu, Mar 15, 2012 at 10:31 AM, Bela Ban&lt;bban(a)redhat.com&gt; wrote: > > On 3/12/12 3:03 PM, Dan Berindei wrote: >> On Sat, Mar 10, 2012 at 7:07 PM, Bela Ban&lt;bban(a)redhat.com&gt; wrote: >>> Can you confirm that my understanding of how DIST works is correct ? >>> >>> >>> #1 Non transactional modifications >>> - E.g. a PUT >>> - The PUT is sent to the primary owner (e.g. B) and all backup owners >>> (e.g. C) and is applied immediately, with only local lock acquisition >>> (lock-put-unlock) >> This is correct, the key lock is acquired in parallel on all the >> owners, so two concurrent PUTs could result in different owners having >> different values. > > So the section that issues the (say) 2 unicast PUTs isn't protected by a > lock ? I'm not suggesting to include the waiting for the results from > the 2 futures should be synchronized, but only the *sending* of the 2 > unicasts. Then if thread 1 acquired that lock first and thread 2 second, > thread's 1's modifications would be delivered *before* thread 2's > modifications. > Of course, this would order only messages from the same sender. > We do acquire the key lock on the originator as well, I just missed it when I read the code. >>> (or is the PUT only sent to B, which in turn then updates C?) >>> >>> >>> #2 Transactional modifications >>> - The modifications involve a bunch of keys >>> - When the TX commits: >>> - A PREPARE message with the relevant keys is sent to all primary >>> owners P (to the backup owners as well?) >> The PREPARE is sent to the backup owners as well. >> The complete list of modifications is sent to all the owners, primary or backup. > > OK, we discussed this in IRC, thanks for the clarification. > > If we touch a lot of keys, then sending *all* of the keys to all owners > may be sub-optimal; as an optimization, we may want to send only the > keys to the nodes which need to store them. This would make the PREPARES > potentially much smaller. > Agree, but it's a non-trivial optimization. For instance if there is a view change between the prepare and the commit, the recipient of the commit may not have all the modifications in the list. With recovery enabled the originator would also have to get the list of modifications from all the targets and combine it into a single PREPARE command, much more complex than what happens now. > >>> If so, then I can assume that a transactional modification touching a >>> number of keys will almost always touch *all* nodes ? Example: >>> - We have 10 nodes >>> - numOwners = 2 >>> - If we have a good consistent hash, I can assume that I have to modifiy >>> 5 different keys (10 / 2) on average in a TX to touch *all* nodes in the >>> cluster with the PREPARE/COMMIT phase, correct ? >>> >> You have to modify *at least* 5 different keys to touch all the nodes. >> Some keys will share the primary owner, but also one key's primary >> owner can be another's backup owner - so the chances to touch all >> nodes with only 5 keys are very low. >> >> It turns out the probability of a tx touching all the nodes depends on >> whether we have virtual nodes enabled or not - with lots of virtual >> nodes all the {(i, j) where 0<= i != j< numNodes} combinations are >> valid, whereas without virtual nodes only numNodes combinations are >> allowed - {(i, (i+1)%clusterSize) where 0<= i< numNodes}. >> >> I've run some simulations with IPython assuming virtual nodes, and >> these are the number of keys per tx required to touch all the nodes >> 50% of the time for a few representative cluster sizes: >> 4: 4, 8: 10, 16: 25, 32: 61 > > So we have to touch roughly clusterSize * 2 keys to send modification > requests to all nodes. > Actually the number of required keys is growing a little faster. The chances of 2 * clusterSize keys touching all the nodes are 2: 1.0, 4: 0.985, 8: 0.91, 16: 0.796, 32: 0.585, 64: 0.327, 128: 0.092 >> These are the numbers of keys required to touch numNodes/2 nodes 50% >> of the time: >> 4: 1, 8: 2, 16: 5, 32: 11 > > OK > > >> And these are the numbers of keys required to touch numNodes/2 nodes >> 90% of the time: >> 4: 1, 8: 3, 16: 7, 32: 13 >> >> >>> If my last statement is correct, is it safe to assume that with DIST and >>> transactional modifications, I will have a lot of TX contention / >>> collisions ? >>> >> Not sure what you mean by lot of TX contention - lock contention >> should only depend on the dataset size, unless we use lock striping, >> in which case it depends on the configured concurrency level. > > I meant TX rollbacks due to overlapping locks at different nodes, the > stuff Pedro wrote about in his paper on total order. > Hmm, I thought because we sort the keys before locking it shouldn't be possible to have deadlocks between prepare commands. I was assuming that the Tx aborts in Pedro's tests were due to write skew check failures, but I just read his message again and he mentions write skew check is disabled. I must be missing something...

>> Network traffic would probably increase quadratically with the number >> of keys/tx because we do send all the modifications to all the owners, >> but we could fix that by either transmitting only the relevant bits to >> each owner > > IMO that's an optimization that should be done regardless, see my > comment above. > It does complicate other parts of the code, so I don't think it's such a clear win. >> or by switching to multicast after the number of targets hits a threshold. > > The problem of switching between unicasts and multicasts is that there > is no ordering between unicast and multicast messages. So if you send an > anycast M1 (2 unicasts) to B,C, followed by a multicast M2, M1 or M2 > could be delivered first. So A could deliver M1 first, folllowed by M2, > while B would deliver M2 followed by M1. > > If the above scenario is not a problem because you acquire locks anyway, > we could do it though... might be a nice optimization ! > > >>> If this is correct, this would IMO lay even more importance onto the >>> work done by the Cloud-TM team, replacing 2PC with total order. Also, if >>> we touch almost all nodes, would it make sense to use SEQUENCER for >>> *all* updates ? Would this obviliate the need for TOM (total order for >>> partial replication) ? >>> >> I don't enough about Cloud-TM to comment here, but it appears >> SEQUENCER requires the coordinator to receive forward requests from >> all the nodes and send nothing back on the unicast channel. > > Yes, this is basically the same problem: with SEQUENCER, there is no > ordering between unicasts and multicasts, either. > > > >> Sending the whole tx as a multicast would certainly be more efficient >> than what we do now with lots of targets. > > Just as confirmation of my understanding: > - We have {A,B,C,D,E} > - A TX modifies keys K1 (maps to A:B), K2 (maps to B:C), K5 (maps to E:A) > > Do we send 3 anycasts A:B, B:C and E:A or do we send 1 anycast A:B:C:E ? > I think it's the latter, but please confirm... > Nope, we haven't switched to anycasts yet. We do serialize the command beforehand, so we don't have triple serialization overhead, but we do need an extra FutureCollator object. My assumption is that the multicast itself is more efficient, because from the originator to the switch it's just one packet. I wasn't thinking of the extra stuff Infinispan and JGroups have to keep for the multiple unicast requests. Cheers Dan _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Hi Pedro On Thu, Mar 15, 2012 at 4:42 PM, Pedro Ruivo <pruivo(a)gsd.inesc-id.pt&gt; wrote: > On 3/15/12 1:36 PM, Dan Berindei wrote: >> On Thu, Mar 15, 2012 at 10:31 AM, Bela Ban&lt;bban(a)redhat.com&gt; wrote: >>> On 3/12/12 3:03 PM, Dan Berindei wrote: >>>> On Sat, Mar 10, 2012 at 7:07 PM, Bela Ban&lt;bban(a)redhat.com&gt; wrote: >>>>> If my last statement is correct, is it safe to assume that with DIST and >>>>> transactional modifications, I will have a lot of TX contention / >>>>> collisions ? >>>>> >>>> Not sure what you mean by lot of TX contention - lock contention >>>> should only depend on the dataset size, unless we use lock striping, >>>> in which case it depends on the configured concurrency level. >>> >>> I meant TX rollbacks due to overlapping locks at different nodes, the >>> stuff Pedro wrote about in his paper on total order. >>> >> Hmm, I thought because we sort the keys before locking it shouldn't be >> possible to have deadlocks between prepare commands. I was assuming >> that the Tx aborts in Pedro's tests were due to write skew check >> failures, but I just read his message again and he mentions write skew >> check is disabled. >> I must be missing something... > I think that a transaction aborts if a scenario like this occurs: > > 4 nodes, N1 to N4. > N2 is the primary owner of KeyA. > N3 is the primary owner of KeyB. > N1 is executing the transaction Tx1 which writes in A and B > N4 is executing the transaction Tx2 which writes in A and B. > Both transactions try to prepare at the same time. This scenario can > occurs (I think): > > N2 -> deliver(Tx1), lock(KeyA), deliver(Tx2), tryLock(KeyA) //Tx2 is > blocked until the lock of KeyA is released > N3 -> deliver(Tx2), lock(KeyB), deliver(Tx1), tryLock(KeyB) //Tx1 is > blocked until the lock of KeyB is released > > Eventually Tx1 or Tx2 (or both) will be aborted by a timeout. Is this > behavior correct? Am I missing something? > Thanks for the example Pedro, a deadlock can indeed appear in this scenario. I kept thinking that because we sort the keys, the Tx2 prepare won't start locking KeyB until it has successfully acquired KeyA. But that's not true, because the Tx2 prepare on N3 doesn't even try to acquire KeyA (as N3 is not the primary owner). So it can go ahead and lock KeyB instead, leading to the deadlock you described.

4 nodes, N1 to N4. N2 is the primary owner of KeyA. N3 is the primary owner of KeyB. N1 is executing the transaction Tx1 which writes in A and B N4 is executing the transaction Tx2 which writes in A and B. Both transactions try to prepare at the same time. This scenario can occurs (I think): N2 -> deliver(Tx1), lock(KeyA), deliver(Tx2), tryLock(KeyA) //Tx2 is blocked until the lock of KeyA is released N3 -> deliver(Tx2), lock(KeyB), deliver(Tx1), tryLock(KeyB) //Tx1 is blocked until the lock of KeyB is released Eventually Tx1 or Tx2 (or both) will be aborted by a timeout. Is this behavior correct? Am I missing something?

On 3/15/12 2:36 PM, Dan Berindei wrote: >> If we touch a lot of keys, then sending *all* of the keys to all owners >> may be sub-optimal; as an optimization, we may want to send only the >> keys to the nodes which need to store them. This would make the PREPARES >> potentially much smaller. >> > > Agree, but it's a non-trivial optimization. For instance if there is a > view change between the prepare and the commit, the recipient of the > commit may not have all the modifications in the list. Can't we treat this as 2 state transfers ?

The first sends the key set to all affected nodes. The key set may not be accurate, in that some keys are missing, and others are not needed. But it is the *recipient* of the state transfer which discards unneeded keys (which it shouldn't own).

The missing keys will then be sent in the second state transfer.

This schem would batch state transfers in a queue: if we have more than 1 ST, we combine them. E.g. if we have view V5,V6,V7 in the queue,  then we initiate a state transfer that transfers keys based on the diff between V5 and V7. If later a V8 is inserted into the queue, we batch the diffs between V7 and V8.

> With recovery enabled the originator would also have to get the list > of modifications from all the targets and combine it into a single > PREPARE command, much more complex than what happens now. Is this really necessary ? Wouldn't the above schem work as well, and still be correct ?

>> I meant TX rollbacks due to overlapping locks at different nodes,  the >> stuff Pedro wrote about in his paper on total order. >> > > Hmm, I thought because we sort the keys before locking it shouldn't be > possible to have deadlocks between prepare commands. I was assuming > that the Tx aborts in Pedro's tests were due to write skew check > failures, but I just read his message again and he mentions write skew > check is disabled. > I must be missing something... Even if we sort the keys, we can still have deadlocks, caused by PREPAREs arriving at different times at different nodes, e.g. the classic deadlock of TX1 acquiring locks on A, TX2 acquiring locks on B, then TX1 acquiring locks on B and TX2 acquiring locks on A.

Hmm. Given the complexity of these issues, I'm starting to think whether it really makes sense to try to implement state transfer they way we are planning this. The track we're on leads to loads of complexity and tons of edge cases. We're trying to provide the C, the A and the P (of CAP), with perhaps a slighly degraded A. This may not be feasible... especially on large clusters where nodes come and go with a higher frequency than in a small cluster.

*Maybe* to a certain degree we can simplify this with total order, but haven't spent much thought on this. I think Manik's eventual consistency work will probably be the best solution...

Comments inline. On 3/16/12 9:43 AM, Dan Berindei wrote: > On Fri, Mar 16, 2012 at 9:27 AM, Bela Ban&lt;bban(a)redhat.com&gt;  wrote: >> >> >> On 3/15/12 2:36 PM, Dan Berindei wrote: >> >>>> If we touch a lot of keys, then sending *all* of the keys to all owners >>>> may be sub-optimal; as an optimization, we may want to send only the >>>> keys to the nodes which need to store them. This would make the PREPARES >>>> potentially much smaller. >>>> >>> >>> Agree, but it's a non-trivial optimization. For instance if there is a >>> view change between the prepare and the commit, the recipient of the >>> commit may not have all the modifications in the list. >> >> >> Can't we treat this as 2 state transfers ? >> > > I was talking about the originator of the transaction sending the > prepare command before a view change and the commit command after the > view change (to the new owners). > > I'm not sure how 2 state transfers would solve this problem. State > transfer does not copy the actual values (modifications list) of a > pending transaction to the new owners, only the list of possibly > locked keys. So the new owner either has the entire mods list (it > received the prepare command), or it doesn't have anything (because it > didn't). With partial mods lists sent to all the owner, the originator > will basically have to resend the (partial) mods lists to everyone > involved inside the commit command. How about we take a note of keys that are blocked (due to other TXs) and exclude them from ST, but don't block on the pending TXs ? When the keys are available, we resend them. This is different from having to wait for pending TXs to complete inside of ST, but ST of locked keys becomes asynchronous... Not sure this makes sense... :-)

>> The first sends the key set to all affected nodes. The key set may not >> be accurate, in that some keys are missing, and others are not needed. >> But it is the *recipient* of the state transfer which discards unneeded >> keys (which it shouldn't own). >> > > What is this key set? Where do we get it from? As the computation of the diff between 2 subsequent views.

>> This schem would batch state transfers in a queue: if we have more than >> 1 ST, we combine them. E.g. if we have view V5,V6,V7 in the queue,  then >> we initiate a state transfer that transfers keys based on the diff >> between V5 and V7. If later a V8 is inserted into the queue, we batch >> the diffs between V7 and V8. >> > > We already do queue state transfers if all the view changes are joins. > But for leaves and merges we interrupt the current state transfer and > restart it "from the top". > > Say we have V5[A,B], V6[A, B, C], V7[A, B, C, D] queued. We do a > single state transfer, V5->V7. But when we receive V8[B, C, D], the > blocking design interrupts the state transfer, rolls back to V5, and > then starts a new state transfer V5->V8. > > The non-blocking design I proposed does basically the same thing, > except it removes the rollback to V5. So we start a state transfer > V5->V7, we receive the V8 with a leaver, and we replace the state > transfer in progress with another state transfer V5->V8. OK

Changed the subject... I got unsure about what Infinispan's DIST mode provides with respect to CAP (see below). I would argue availability and partitioning, but not consistency, as a network partition and subsequent healing might lead to incorrect values... Thoughts ? On 3/22/12 3:33 PM, Dan Berindei wrote: >> We're trying to provide the C, the A and the P (of CAP), with perhaps a >> slighly degraded A. This may not be feasible... especially on large >> clusters where nodes come and go with a higher frequency than in a small >> cluster. >> > > Slight correction here: we are only trying to provide C and A. We > don't really have anything to handle P yet...

If we touch a lot of keys, then sending *all* of the keys to all owners may be sub-optimal; as an optimization, we may want to send only the keys to the nodes which need to store them. This would make the PREPARES potentially much smaller.

On 3/15/12 4:25 PM, Manik Surtani wrote: > > On 15 Mar 2012, at 04:31, Bela Ban wrote: > >> If we touch a lot of keys, then sending *all* of the keys to all owners >> may be sub-optimal; as an optimization, we may want to send only the >> keys to the nodes which need to store them. This would make the PREPARES >> potentially much smaller. > > > Not really - if we are sending different prepares to different nodes (containing different writes) then the serialisation overhead goes up since each recipient gets a different byte buffer.  In the current case we reuse the byte buffer. In my experience, serialization is never the bottleneck, it is *de-serialization*, so this shouldn't be a problem. Also, I don't see why serialization of keys [1..9] shouldn't take roughly the same time as serialization of keys [1..3], [4..6], [7..9].

If we have a TX which touches a lot of keys, unless we break the key set down into their associated targets, we would have to potentially transfer a *big* state !

Let's take a look at an example: - Nodes are N1-N5, keys are 1-10 With the current state transfer, we'd serialize the 10 keys into a buffer of (say) 10. That buffer is then sent to (say) all nodes, so we have 5 messages of 10 = 50. Now say the keys are only sent to the nodes which should store them: - N1: 1, 5, 9 - N2: 1,2, 5,6, 9,10 - N3: 2,3 6,7, 10 - N4: 3,4 7,8 - N5: 4,8 This is a total of 5 messages of sizes 3, 6, 5, 4 and 2 = 20. The biggest message is 6, and the total size of half of the current approach. Wrt serialization: we could maintain a simple hashmap of the current keys serialized, e.g. - N1: we serialize keys 1, 5 and 9 and add them to the hashmap - N2: we serialize keys 2, 6 and 10 and add them to the hashmap. Keys 1, 5 and 9 have already been serialized and the byte buffer of their values can just be reused from the hashmap. If we do this intelligently, we could sort this temp hashmap (we do know the hashwheel after all, don't we), and remove keys/values that were already sent, as they're not going to be used again. This would reduce the max memory spike for the temp hashmap. > So when the cluster is small and all the owners would receive most of > the keys anyway, the savings in deserialization are offset by the > increased costs in serialization. You don't serialize the same key twice, see above. You just copy the buffer into the resulting buffer, this is very efficient. With scatter/gather (NIO2, JGroups 3.3), we would not even have to copy the buffers, but pass all of them into the JGroups message.

>> If we have a TX which touches a lot of keys, unless we break the key set >> down into their associated targets, we would have to potentially >> transfer a *big* state ! >> > > If we serialize the keys separately for each target, the originator > will have to hold in memory numOwners * numKeys serialized entries > until the messages are garbage collected. That's 20, correct. But with the current approach, it is 50 (key set of 10 sent to all 5 nodes).

>  With the current approach, > we only have to hold in memory numKeys serialized entries on the > originator - at the expense of extra load on the recipients. You're sending the 10 serialized keys to 5 nodes, so you'll have 5 messages with 10 each in memory, that's worse then my suggested approach.

On 3/10/12 5:07 PM, Bela Ban wrote: > If so, then I can assume that a transactional modification touching a > number of keys will almost always touch *all* nodes ? Example: > - We have 10 nodes > - numOwners = 2 > - If we have a good consistent hash, I can assume that I have to modifiy > 5 different keys (10 / 2) on average in a TX to touch *all* nodes in the > cluster with the PREPARE/COMMIT phase, correct ? > > If my last statement is correct, is it safe to assume that with DIST and > transactional modifications, I will have a lot of TX contention / > collisions ? We have run experiments with ISPN 5.2 and TPC-C (1 warehouse, which gives a high probability of contention among transactions), and compared it with ISPN 5.0 (where locks were acquired on all replicas of a key, not only on the primary). The results running w/o write skew check and 10 nodes on our cluster (number of owners=2) follow: Tx/sec Abort Rate 5.2 12 15 5.0 3 30 5.0-TOM 60 0

> Also, if we touch almost all nodes, would it make sense to use SEQUENCER for > *all* updates ? Would this obviliate the need for TOM (total order for > partial replication) ? This could be done, you are right, it's what sometimes is called "non-genuine" partial replication. Our take on this is that this will work good on small scale clusters, not on large ones.

Hi, Comment inline. Cheers, Pedro On 3/15/12 8:34 AM, Bela Ban wrote: > > On 3/12/12 7:13 PM, Pedro Ruivo wrote: >> On 3/10/12 5:07 PM, Bela Ban wrote: >>> If so, then I can assume that a transactional modification touching a >>> number of keys will almost always touch *all* nodes ? Example: >>> - We have 10 nodes >>> - numOwners = 2 >>> - If we have a good consistent hash, I can assume that I have to modifiy >>> 5 different keys (10 / 2) on average in a TX to touch *all* nodes in the >>> cluster with the PREPARE/COMMIT phase, correct ? >>> >>> If my last statement is correct, is it safe to assume that with DIST and >>> transactional modifications, I will have a lot of TX contention / >>> collisions ? >> We have run experiments with ISPN 5.2 and TPC-C (1 warehouse, which >> gives a high probability of contention among transactions), and compared >> it with ISPN 5.0 (where locks were acquired on all replicas of a key, >> not only on the primary). >> >> The results running w/o write skew check and 10 nodes on our cluster >> (number of owners=2) follow: >> >> Tx/sec Abort Rate >> 5.2 12 15 >> 5.0 3 30 >> 5.0-TOM 60 0 > > Excellent ! It shows that 2PC has really improved between 5.0 and 5.2... > > Have you run TOM on Infinispan-5.2 / JGroups 3.1 yet ? It should > theoretically still be 60 TXs/sec. But even compared to 12, this is > still much better ! > > I've not run it yet. I want to make the code stable (TOM with and without the write skew check) before starting the benchmark of the protocol. I'm still having some problems with the write skew check :P >>> Also, if we touch almost all nodes, would it make sense to use SEQUENCER for >>> *all* updates ? Would this obviliate the need for TOM (total order for >>> partial replication) ? >> This could be done, you are right, it's what sometimes is called >> "non-genuine" partial replication. Our take on this is that this will >> work good on small scale clusters, not on large ones. > > I agree > > > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev