[ https://issues.jboss.org/browse/ISPN-2156?page=com.atlassian.jira.plugin.... ] Mircea Markus updated ISPN-2156: -------------------------------- Description: To summarise: When using distributed caches, when we need to batch-load a set of data into the cluster inserting bathes of keys that map to the same node should significantly increase the performance. Why? during the prepare phase each node receives the complete list of modifications in that transaction and not only the modification pertaining to it. E.g. say we have the following key->node mapping: {code} k1 -> A k2 -> B k3 -> C {code} Where k1, k2 and k3 are keys; A, B and C are nodes. If Tx1 writes (k1,k2,k3) then during the prepare A,B and C will receive the the same package containing all the modification - namely (k1, k2,k3). There are several reasons for doing this (apparently) unoptimized approach: serialize the prepare only once, better handling of recovery information. Now if you group transactions/batches base on key distribution the amount of redundant traffic is significantly reduced - and that translates in better performance especially when the datasets you're inserting is quite high. This JIRA is basically about benchmarking and blogging about this approach. A entry in the FAQ would be helpful as well. was:The entire discussion around it can be found here: http://lists.jboss.org/pipermail/infinispan-dev/2012-July/010968.html Forum Reference: http://lists.jboss.org/pipermail/infinispan-dev/2012-July/010968.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-2156?page=com.atlassian.jira.plugin.... ] Manik Surtani updated ISPN-2156: -------------------------------- Summary: Benchmark and blog about a fast method of loading data into Infinispan (was: Benchmark and blog about a fast method of loading that into Infinispan ) Description: To summarise: When using distributed caches, when we need to batch-load a set of data into the cluster inserting batches of keys that map to the same node should significantly increase the performance. Why? during the prepare phase each node receives the complete list of modifications in that transaction and not only the modification pertaining to it. E.g. say we have the following key->node mapping: {code} k1 -> A k2 -> B k3 -> C {code} Where k1, k2 and k3 are keys; A, B and C are nodes. If Tx1 writes (k1,k2,k3) then during the prepare A,B and C will receive the the same package containing all the modification - namely (k1, k2,k3). There are several reasons for doing this (apparently) unoptimized approach: serialize the prepare only once, better handling of recovery information. Now if you group transactions/batches base on key distribution the amount of redundant traffic is significantly reduced - and that translates in better performance especially when the datasets you're inserting is quite high. This JIRA is basically about benchmarking and blogging about this approach. A entry in the FAQ would be helpful as well. was: To summarise: When using distributed caches, when we need to batch-load a set of data into the cluster inserting bathes of keys that map to the same node should significantly increase the performance. Why? during the prepare phase each node receives the complete list of modifications in that transaction and not only the modification pertaining to it. E.g. say we have the following key->node mapping: {code} k1 -> A k2 -> B k3 -> C {code} Where k1, k2 and k3 are keys; A, B and C are nodes. If Tx1 writes (k1,k2,k3) then during the prepare A,B and C will receive the the same package containing all the modification - namely (k1, k2,k3). There are several reasons for doing this (apparently) unoptimized approach: serialize the prepare only once, better handling of recovery information. Now if you group transactions/batches base on key distribution the amount of redundant traffic is significantly reduced - and that translates in better performance especially when the datasets you're inserting is quite high. This JIRA is basically about benchmarking and blogging about this approach. A entry in the FAQ would be helpful as well. Forum Reference: http://lists.jboss.org/pipermail/infinispan-dev/2012-July/010968.html (was: http://lists.jboss.org/pipermail/infinispan-dev/2012-July/010968.html) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira