[infinispan-dev] Late end invalidation delivery stops put from load - ISPN-8114

[Galder Zamarreño]

Hey Radim,

Re: https://issues.jboss.org/browse/ISPN-8114

I've been looking at the trace logs of this failure. I've extracted the most interesting parts of this failure into [1].

What happens is that after loading the entries into the cache, the end invalidation message to allow put from loads to succeed does not get executed in time before the put from load is attempted. As a result of this, the put from load does not happen and hence the entry is not loaded into the cache.

The end invalidation message eventually gets through. There's a gap between receiving the JGroups message and the actual execution, but that's due to the delivery mode of the message.

I'm not sure how we should fix this. Options:

1) A thread sleep before loading entries "might work" but for a CI test this could always backfire with the right timing sets.

2) Find a way to hook into the PFLValidator class and only load after we know end invalidation has been received by all nodes.

3) Make end invalidation message sync? This would expensive. Even with async, changing delivery mode might have worked here... but under the right circumstances you could still get the same issue with async.

I'm keen on trying to find a potential solution using 2), but wondered if you have other ideas.

Cheers,

[1] https://gist.github.com/galderz/0bce6dce16de018375e43e25c0cf3913
--
Galder Zamarreño
Infinispan, Red Hat