[infinispan-dev] Verifying Consistent Hash based routing using stats

Hey Vittorio, Expanding this to dev list in case anyone has interest. Latest commit is in [1]. The relevant code that verifies Consistent Hash (CH) based routing is in [2]. The way it works is this: Assuming I have 3 nodes, A, B and C, with 2 as number of copies for distribution, I do the following: 0. Get stats from nodes A, B, C. 1. Generate a key K1 whose owners are A,B. 2. Generate a key K2 whose owners are B,C. 3. Generate a key K3 whose owners are C,A. 4. The primary node is always the first node of the owners, so if CH routing works fine, K1 would targeted for node A, K2 for node B and K3 for node C. 5. Call puts on keys K1, K2 and K3. 6. Get stats from nodes A, B, C. 7. For all nodes, "stores" value in latest stats should be +1 compared to initial stats. This would pass even if with a round-robin load balance policy. 8. For all nodes, "currentNumberOfEntries" value in latest stats should be (Number of copies) * ("stores"). This is the key assumption that guarantees CH routing works as expected. ^ To be more precise, this can only be guaranteed under certain circumstances: First, the ownership needs to be spread around, so a node does not take owner ship of 3 keys and another of only 1. We can guarantee that based on how we generate the keys. The second part is that if a put happens in a non-owner, this non-owner node will increase its "stores" values but not its "currentNumberOfEntries" since the key does not belong in this node. So, if any put request happened in a non-owner, there would be an inbalance in the ratio of currentNumberOfEntries to "stores". Hope this helps. Cheers, [1] https://github.com/galderz/js-client/commit/c57c285561b73e67b7ebce04de67d... [2] https://github.com/galderz/js-client/blob/t_steps/spec/infinispan_cluster... -- Galder Zamarreño Infinispan, Red Hat