Manik Surtani wrote: I don't know the DIST code very well yet but here's an idea: Maybe instead of doing the state transfer for the key, rehash() could queue these state transfer requests and do the following: The next time a put/get for a key that needs to be rehashed comes along, do the reshahing then. Obviously, this has issues for gets arriving in nodes not containing the key looked after. In theory, you'd need to use the hash that was originally used to store that key to be able to find it but not necessarily. You could try with the current hash result and if that doesn't return anything, do a clustered get on all nodes in cluster. As a response to this clustered get, the node could initiate an state transfer of this key to the new host node if still necessary. It could well happen that after N views, the owners of the keys should still be the same as it was N views ago. If a get arrives on a node A already containing the key but the key is also present in node B that needs to pass the key node C, not much we can do until a request comes in to node C or a node not containing that key. For puts, updates on nodes that should transfer the key would trigger the transfering together with the update. The major downside I see to this is from a user perspective. The user does not have an up to date view of where the keys should reside until a put/get is done in all keys that should have been rehashed. -- Galder Zamarreño Sr. Software Maintenance Engineer JBoss, a division of Red Hat