10:59 a.m.
Hi all,
simple question: What are the consistency guaranties that is supposed to
be ensured?
I have the following scenario (happened in a test case):
NonOwner: remote get key
BackupOwner: receives the remote get and replies (with the correct value)
BackupOwner: put in L1 the value
PrimaryOwner: [at the same time] is committing a transaction that will
update the key.
PrimaryOwer: receives the remote get after sending the commit. The
invalidation for L1 is not sent to NonOwner.
The test finishes and I perform a check for the key value in all the
caches. The NonOwner returns the L1 cached value (==test fail).
IMO, this is bug (or not) depending what guaranties we provide.
wdyt?
Pedro
11:38 a.m.
On Tue, Jul 2, 2013 at 6:36 PM, Pedro Ruivo <pedro(a)infinispan.org> wrote:
Manik had a patch for staggering remote GET calls, but it was slowing down
reads by 25%: http://markmail.org/message/vsx46qbfzzxkkl4w
1:41 p.m.
On 2 July 2013 17:24, Dan Berindei <dan.berindei(a)gmail.com> wrote:
It's not wrong, sending the invalidation only from the primary
owner is
wrong :)
Agreed, sending a GET operation to multiple nodes might not be wrong
per-se but is the root cause of such race conditions, and other subtle
complexities we might not even be aware of yet.
I don't know why it was slower, but since the result doesn't make
sense we should look at it a second time rather than throwing the code
away.
Sending invalidations from a non-primary owner is an interesting
approach, but then we're having each owner to maintain an independent
list of nodes who have read the value.
For each write, the primary node would send an invalidation to each
registered node, plus the copy to the secondary nodes, which in turn
sends more L1 invalidation nodes to each of their registered nodes..
what's the likelihood of duplication of invalidation messages here?
Sounds like a big network traffic amplifier, lots of network traffic
triggered for each write.
It also implies that we don't have reliability on the list of
registered nodes, as each owner will be maintaining a different set.
In this case we should also have each node invalidate its L1 stored
entries when the node from which they got these entries has left the
cluster.
Having it all dealt by the primary owner makes for a much simpler
design and also makes it more likely that a single L1 invalidate
message is sent via multicast, or at least with less duplication.
Cheers,
Sanne
5:26 a.m.
On Tue, Jul 2, 2013 at 8:41 PM, Sanne Grinovero <sanne(a)infinispan.org>wrote:
It does make sense: statistically, the backup owner will sometimes reply
faster than the primary owner.
http://markmail.org/message/qmpn7yueym4tbnve
http://www.bailis.org/blog/doing-redundant-work-to-speed-up-distributed-q...
The likelihood of duplication is very near to 100%, indeed, and in non-tx
caches it would add another RPC to the critical path.
As always, it's a compromise: if we do something to speed up writes, it
will slow down reads. Perhaps we could send the request to the primary
owners only when L1 is enabled, as the number of remote gets should be
smaller, and send the request to all the owners when L1 is disabled, and
the number of remote gets is higher.
Pedro's suggestion to send the request to all the owners, but only write
the value to L1 if the first reply was from the primary owner, sounds like
it should work just as well. It would make L1 slightly less efficient, but
it wouldn't have latency spikes caused by a delay on the primary owner.
Right now we invalidate from L1 all the keys for which the list of owners
changed, whether they're still alive or not, because we don't keep track of
the node we got each entry from.
If we only sent remote get commands to the primary owner, we'd have to
invalidate from L1 all the keys for which the primary owner changed.
One thing that we don't do at the moment, but we should do whether we send
the invalidations from the primary owner or from all the owners, is to
clean up the requestor lists for the keys that a node no longer owns.
The simplest design would be to never keep track of requestors and always
send a multicast from the originator. In fact, the default configuration is
to always send multicasts (but we still keep track of requestors and we
send the invalidation from the primary owner).
Intuitively, unicasts would be preferable for keys that have a low
read:write ratio, as in a write-intensive scenario, but I wonder if
disabling L1 wouldn't be even better for that scenario.
Cheers
Dan
5:19 p.m.
On 3 July 2013 10:26, Dan Berindei <dan.berindei(a)gmail.com> wrote:
Of course, I remember the discussion but you can put may questions
marks on this decision. First off, this is doubling the load on
network, which is supposedly our most precious resource, so I highly
question how we're measuring the "benefit" of the second request.
If you read the articles you linked to you'll see google applies such
a strategy to improve the tail latency, but only sends the second
request when the first one is not getting a fast answer, and in their
tests this seems to pay off to a mere 5% of increased network usage..
I would say that's a significantly different level of trade off.
Also, opposed to Google's BigTable and Apache Cassandra which use
these techniques, Infinispan is not supporting an eventually
consistent model which makes it far more dangerous to read the
slightly out of date value from the non-owner... sure we can resolve
those things, but it gets hairy.
In this case specifically, reading from the primary owner only seems a
much "cleaner" solution, as IMHO it's a move towards simplification
rather than adding yet another special case in the codebase.
That's far from an ideal solution; we don't have a clue on how to
measure what "slightly less efficient" means: that might reveal to be
"unbearably worse" for some usage pattern.
While we have no clue how worse it can be, it will definitely always
provide a worse cache hit/miss ratio, so it's easily proven that it's
going to be sub optimal in all cases.
If you really go for something like that, at least take the value from
the primary owners when it arrives (second, third, .. whatever but at
some point you might get it) and then store it in L1: will cost you a
second unmarshalling operation but that's far better than causing
(several?) cache misses.
Well put
Cheers,
Sanne
12:35 p.m.
On Mon, Jul 8, 2013 at 12:19 AM, Sanne Grinovero <sanne(a)infinispan.org>wrote:
In this case specifically, reading from the primary owner only seems a
It's cleaner, but it slows down reads, as we've seen in our own tests.
Also, the consistency problems with staggered remote gets are the same as
the consistency problems with simultaneous remote gets. So it would be much
harder to go back and experiment with staggered gets once we simplify the
code on the assumption that we should only ever read from the primary owner.
True, in a worst case scenario, the primary owner could be replying
consistently slower than the others, and the entry may never be stored in
L1. And it wouldn't make sense to try this with numOwners = 10, the chances
of the primary owner replying first would be slim.
I think my "slightly less efficient" description would be more accurate if
we had staggered gets, if we ever get that patch in...
If numOwners = 1 there is only one way to read the value, so it's clearly
optimal in one case :)
That sounds nice, I wonder how easily Will could make it work with the L1
synchronization stuff he has implemented.
10:11 a.m.
On Tue, Jul 9, 2013 at 9:35 AM, Dan Berindei <dan.berindei(a)gmail.com> wrote:
Yeah it sounds like that would be a perfect fit here. Since you would
want any write/invalidation that occurs for that key to cancel out the
L1 being updated from the primary owner get that hasn't completed yet.
11:29 a.m.
On Tue, Jul 2, 2013 at 5:59 PM, Pedro Ruivo <pedro(a)infinispan.org> wrote:
I assume you meant NonOwner here?
At some point, BackupOwner has to execute the commit as well, and it should
send an InvalidateL1Command(key) to NonOwner. However, one of the bugs that
Will is working on could prevent that invalidation from working (maybe
https://issues.jboss.org/browse/ISPN-2965).
It's a bug!
IMO, at least in DIST_SYNC mode with sync commit, we should guarantee that
stale entries are removed from non-owners before the TM.commit() call
returns on the originator.
With other configurations we probably can't guarantee that, instead we
should guarantee that stale entries are removed from non-owners "soon"
after the TM.commit() call returns on the originator.
I don't think it's ok to say that a stale entry can stay in L1 indefinitely
in any configuration - otherwise why have L1 invalidation at all?
11:55 a.m.
On Tue, Jul 2, 2013 at 6:35 PM, Pedro Ruivo <pedro(a)infinispan.org> wrote:
Oops, you're right! And I'm pretty sure I made the same assumptions in my
replies to Will's L1 thread...
I guess we could make it work either sending the invalidations from all the
owners (slowing down writes, because most of the time we'd send the same
commands twice), or by sending remote get commands ONLY to the primary
owner (which will slow down remote reads).
Staggering remote GET commands won't work, because with staggering you
still have the possibility of the first request reaching the primary owner
only after the second request returned and the entry was written to L1.
Sending the invalidations from the tx originator after the commit might
work as well, but only if L1.invalidationTreshold == 0 and all the L1
invalidations are sent as multicasts.
11:09 a.m.
On Wed, Jul 3, 2013 at 5:28 AM, Dan Berindei <dan.berindei(a)gmail.com> wrote:
I would agree, having both would be the best of both worlds. This way
the L1 value could still be most likely cached (what 95% of the time).
Also I wonder if the performance issue was because the stagger timeout
was 100ms - seems a little high.
The value from the return is not a bug (test needs fixing), but the
value being stored in the L1 would definitely be a bug.
4278
days inactive
4286
days old
infinispan-dev@lists.jboss.org
16 comments
4 participants
participants (4)
-
Dan Berindei -
Pedro Ruivo -
Sanne Grinovero -
William Burns