[infinispan-dev] Dynamic data placement

[Pedro Ruivo]

Hi all,

In the Cloud-TM project we are working on a technique to change 
dynamically the distribution of data across the nodes of an ISPN data 
grid. The idea is basically to determine at run-time which are the keys 
most accessed by each member/node and move these keys to that nodes. The 
ultimate goal is to maximize data locality and reduce the number of 
remote get operation.

One of the ingredients of the solution is to define a mechanism that 
allows the entire data grid, which is running in a configuration A, 
having a given data distribution strategy (determined by the current 
hash function implementation), to switch to a configuration B, in which 
all nodes have agreed on a new data distribution strategy, and have 
completed to relocate accordingly the keys.

Our idea therefore is to add a functionality in ISPN, which allows to 
support the dynamic change of the data distribution criterion (namely 
the consistent hash function).

To us this mechanism resembles closely the state transfer problem, 
except that in the latter case the keys are relocated when a node joins 
or leaves the system.

We were thinking to implement this by introducing a command (say, 
RedistributedDataCommand) that would take as input a new hash function, 
and that would internally trigger a state transfer based on the hash 
function passed as input parameter. Another alternative would be to 
change the CacheViewControlCommand, allowing it to distributed across 
all the nodes the new hash function, as well as to trigger the state 
transfer.

The point is that we would like to do this in a clean way, in order to 
simplify a possible future integration. We will therefore be grateful 
for any feedback that you will be able to provide.

Cheers,
Pedro Ruivo