Re: [infinispan-dev] DIST-SYNC, put(), a problem and a solution
On 29/07/14 16:39, Dan Berindei wrote:
Investigation:
------------
This mitigated the problem somewhat, but when I increased the requester
threads to 100, I had the same problem again. Apparently, the Infinispan
internal thread pool uses a rejection policy of "run" and thus uses the
JGroups (OOB) thread when exhausted.
We can't use another rejection policy in the remote executor because the
message won't be re-delivered by JGroups, and we can't use a queue either.
Yes I'm aware of that and "run" is our only option for the Infinispan
internal thread pool
Suggested solution
----------------
The modification RPC needs to be invoked *outside of the lock scope*:
- lock K
- modify K
- unlock K
- send modification to backup owner(s) // outside the lock scope
The primary owner puts the modification of K into a queue from where a
separate thread/task removes it. The thread then invokes the PUT(K) on
the backup owner(s).
Does the replication thread execute the PUT(k) synchronously, or
asynchronously? I assume asynchronously, otherwise the replication
thread wouldn't be able to keep up with the writers.
Async would be preferred, but the order of the updates needs to be
guaranteed. This could be done with the sequence numbers suggested by
Sanne, or using total order/TOA.
The queue has the modified keys in FIFO order, so the
modifications
arrive at the backup owner(s) in the right order.
Sending the RPC to the backup owners asynchronously, while holding the
key lock, would do the same thing.
Yes - but if there's a chance that the send() system call blocks, e.g.
on TCP when the send window is full, the async repl should be outside
the lock scope.
If those update messages to the backup owner(s) are regular (not OOB)
messages, FIFO would ensure that they're processed in the order in which
they were sent. If they're OOB messages, we'd have to somehow guarantee
ordering, e.g. using seq numbers.
This requires that the way GET is implemented changes slightly:
instead
of invoking a GET on all owners of K, we only invoke it on the primary
owner, then the next-in-line etc.
I have a WIP branch for this and it seemed to work fine. Test suite
speed seemed about the same, but I didn't get to do a real performance test.
Hmm, it should reduce overall traffic, which indirectly should lead to
better performance. I hope you wrap your work with a JIRA and commit it !
--
Bela Ban, JGroups lead (http://www.jgroups.org)