Radim, I would contend that the first and foremost guarantee that put() makes is to leave the cache in a consistent state. So we can't just throw an exception and give up, leaving k=v on one owner and k=null on another.
Secondly, put(k, v) being atomic means that it either succeeds, it writes k=v in the cache, and it returns the previous value, or it doesn't succeed, and it doesn't write k=v in the cache. Returning the wrong previous value is bad, but leaving k=v in the cache is just as bad, even if the all the owners have the same value.
And last, we can't have one node seeing k=null, then k=v, then k=null again, when the only write we did on the cache was a put(k, v). So trying to undo the write would not help.
In the end, we have to make a compromise, and I think returning the wrong value in some of the cases is a reasonable compromise. Of course, we should document that :)
I also believe ISPN-2956 could be fixed so that HotRod behaves just like embedded mode after the ISPN-3422 fix, by adding a RETRY flag to the HotRod protocol and to the cache itself.
Incidentally, transactional caches have a similar problem when the originator leaves the cluster: ISPN-3421 [1]
And we can't handle transactional caches any better than non-transactional caches until we expose transactions to the HotRod client.
Cheers
Dan