Hey Radim, Re: https://issues.jboss.org/browse/ISPN-8114 I've been looking at the trace logs of this failure. I've extracted the most interesting parts of this failure into [1]. What happens is that after loading the entries into the cache, the end invalidation message to allow put from loads to succeed does not get executed in time before the put from load is attempted. As a result of this, the put from load does not happen and hence the entry is not loaded into the cache. The end invalidation message eventually gets through. There's a gap between receiving the JGroups message and the actual execution, but that's due to the delivery mode of the message. I'm not sure how we should fix this. Options: 1) A thread sleep before loading entries "might work" but for a CI test this could always backfire with the right timing sets. 2) Find a way to hook into the PFLValidator class and only load after we know end invalidation has been received by all nodes. 3) Make end invalidation message sync? This would expensive. Even with async, changing delivery mode might have worked here... but under the right circumstances you could still get the same issue with async.

I'm keen on trying to find a potential solution using 2), but wondered if you have other ideas. Cheers, [1] https://gist.github.com/galderz/0bce6dce16de018375e43e25c0cf3913 -- Galder Zamarreño Infinispan, Red Hat

On 3 Aug 2017, at 11:34, Galder Zamarreño <galder(a)redhat.com&gt; wrote: Thx Radim, I'll look into wrapping PerCacheInboundInvocationHandler. Cheers, -- Galder Zamarreño Infinispan, Red Hat > On 2 Aug 2017, at 18:41, Radim Vansa <rvansa(a)redhat.com&gt; wrote: > > On 08/02/2017 01:02 PM, Galder Zamarreño wrote: >> Hey Radim, >> >> Re: https://issues.jboss.org/browse/ISPN-8114 >> >> I've been looking at the trace logs of this failure. I've extracted the most interesting parts of this failure into [1]. >> >> What happens is that after loading the entries into the cache, the end invalidation message to allow put from loads to succeed does not get executed in time before the put from load is attempted. As a result of this, the put from load does not happen and hence the entry is not loaded into the cache. >> >> The end invalidation message eventually gets through. There's a gap between receiving the JGroups message and the actual execution, but that's due to the delivery mode of the message. >> >> I'm not sure how we should fix this. Options: >> >> 1) A thread sleep before loading entries "might work" but for a CI test this could always backfire with the right timing sets. >> >> 2) Find a way to hook into the PFLValidator class and only load after we know end invalidation has been received by all nodes. >> >> 3) Make end invalidation message sync? This would expensive. Even with async, changing delivery mode might have worked here... but under the right circumstances you could still get the same issue with async. > > 1) is unreliable and wrong from a testsuite perspective, and 3) is completely wrong (making sth sync just because it's easier to test it that way). > > Spying on PFVL is an option, but I would rather wrap PerCacheInboundInvocationHandler (I hope that's the correct way). Note that there's even a TestingUtil.wrapInboundInvocationHandler() helper method. > > If I am missing the complexity, please elaborate. > > Radim > >> I'm keen on trying to find a potential solution using 2), but wondered if you have other ideas. >> >> Cheers, >> >> [1] https://gist.github.com/galderz/0bce6dce16de018375e43e25c0cf3913 >> -- >> Galder Zamarreño >> Infinispan, Red Hat >> > > > -- > Radim Vansa <rvansa(a)redhat.com&gt; > JBoss Performance Team _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev