[infinispan-dev] L1OnRehash Discussion

[William Burns]

Hello everyone,

I wanted to discuss what I would say as dubious benefit of L1OnRehash
especially compared to the benefits it provide.

L1OnRehash is used to retain a value by moving a previously owned
value into the L1 when a rehash occurs and this node no longer owns
that value  Also any current L1 values are removed when a rehash
occurs.  Therefore it can only save a single remote get for only a few
keys when a rehash occurs.

This by itself is fine however L1OnRehash has many edge cases to
guarantee consistency as can be seen from
https://issues.jboss.org/browse/ISPN-3838.  This can get quite
complicated for a feature that gives marginal performance increases
(especially given that this value may never have been read recently -
at least normal L1 usage guarantees this).

My first suggestion is instead to deprecate the L1OnRehash
configuration option and to remove this logic.

My second suggestion is a new implementation of L1OnRehash that is
always enabled when L1 threshold is configured to 0.  For those not
familiar L1 threshold controls whether invalidations are broadcasted
instead of individual messages.  A value of 0 means to always
broadcast.  This would allow for some benefits that we can't currently
do:

1. L1 values would never have to be invalidated on a rehash event
(guarantee locality reads under rehash)
2. L1 requestors would not have to be tracked any longer

However every write would be required to send an invalidation which
could slow write performance in additional cases (since we currently
only send invalidations when requestors are found).  The difference
would be lessened with udp, which is the transport I would assume
someone would use when configuring L1 threshold to 0.

What do you guys think?  I am thinking that no one minds the removal
of L1OnRehash that we have currently (if so let me know).  I am quite
curious what others think about the changes for L1 threshold value of
0, maybe this configuration value is never used?

Thanks,

  - Will