<div dir="ltr"><div class="gmail_extra">Yeah, I wouldn't call this a "simple" solution...<br><br></div><div class="gmail_extra">The distribution/replication interceptors are quite high in the interceptor stack, so we'd have to save the state of the interceptor stack (basically the thread's stack) somehow and resume processing it on the thread receiving the responses. In a language that supports continuations that would be a piece of cake, but since we're in Java we'd have to completely change the way the interceptor stack works.<br>
<br></div><div class="gmail_extra">Actually we do hold the lock on modified keys while the command is replicated to the other owners. But think locking wouldn't be a problem: we already allow locks to be owned by transactions instead of threads, so it would just be a matter of creating a "lite transaction" for non-transactional caches. Obviously the TransactionSynchronizerInterceptor would have to go, but I see that as a positive thing ;)<br>
<br></div><div class="gmail_extra">So yeah, it could work, but it would take a huge amount of effort and it's going to obfuscate the code. Plus, I'm not at all convinced that it's going to improve performance that much compared to a new thread pool.<br>
</div><div class="gmail_extra"><br></div><div class="gmail_extra">Cheers<br></div><div class="gmail_extra">Dan<br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 1, 2013 at 10:59 AM, Radim Vansa <span dir="ltr"><<a href="mailto:rvansa@redhat.com" target="_blank">rvansa@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Yeah, that would work if it is possible to break execution path into the FutureListener from the middle of interceptor stack - I am really not sure about that but as in current design no locks should be held when a RPC is called, it may be possible.<br>
<br>
Let's see what someone more informed (Dan?) would think about that.<br>
<br>
Thanks, Bela<br>
<span class=""><font color="#888888"><br>
Radim<br>
</font></span><div class=""><div class="h5"><br>
----- Original Message -----<br>
| From: "Bela Ban" <<a href="mailto:bban@redhat.com">bban@redhat.com</a>><br>
| To: <a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
| Sent: Friday, February 1, 2013 9:39:43 AM<br>
| Subject: Re: [infinispan-dev] Threadpools in a large cluster<br>
|<br>
| It looks like the core problem is an incoming RPC-1 which triggers<br>
| another blocking RPC-2: the thread delivering RPC-1 is blocked<br>
| waiting<br>
| for the response from RPC-2, and can therefore not be used to serve<br>
| other requests for the duration of RPC-2. If RPC-2 takes a while,<br>
| e.g.<br>
| waiting to acquire a lock in the remote node, then it is clear that<br>
| the<br>
| thread pool will quickly exceed its max size.<br>
|<br>
| A simple solution would be to prevent invoking blocking RPCs *from<br>
| within* a received RPC. Let's take a look at an example:<br>
| - A invokes a blocking PUT-1 on B<br>
| - B forwards the request as blocking PUT-2 to C and D<br>
| - When PUT-2 returns and B gets the responses from C and D (or the<br>
| first<br>
| one to respond, don't know exactly how this is implemented), it sends<br>
| the response back to A (PUT-1 terminates now at A)<br>
|<br>
| We could change this to the following:<br>
| - A invokes a blocking PUT-1 on B<br>
| - B receives PUT-1. Instead of invoking a blocking PUT-2 on C and D,<br>
| it<br>
| does the following:<br>
| - B invokes PUT-2 and gets a future<br>
| - B adds itself as a FutureListener, and it also stores the<br>
| address of the original sender (A)<br>
| - When the FutureListener is invoked, B sends back the result<br>
| as a<br>
| response to A<br>
| - Whenever a member leaves the cluster, the corresponding futures are<br>
| cancelled and removed from the hashmaps<br>
|<br>
| This could probably be done differently (e.g. by sending asynchronous<br>
| messages and implementing a finite state machine), but the core of<br>
| the<br>
| solution is the same; namely to avoid having an incoming thread block<br>
| on<br>
| a sync RPC.<br>
|<br>
| Thoughts ?<br>
|<br>
|<br>
|<br>
|<br>
| On 2/1/13 9:04 AM, Radim Vansa wrote:<br>
| > Hi guys,<br>
| ><br>
| > after dealing with the large cluster for a while I find the way how<br>
| > we use OOB threads in synchronous configuration non-robust.<br>
| > Imagine a situation where node which is not an owner of the key<br>
| > calls PUT. Then the a RPC is called to the primary owner of that<br>
| > key, which reroutes the request to all other owners and after<br>
| > these reply, it replies back.<br>
| > There are two problems:<br>
| > 1) If we do simultanously X requests from non-owners to the primary<br>
| > owner where X is OOB TP size, all the OOB threads are waiting for<br>
| > the responses and there is no thread to process the OOB response<br>
| > and release the thread.<br>
| > 2) Node A is primary owner of keyA, non-primary owner of keyB and B<br>
| > is primary of keyB and non-primary of keyA. We got many requests<br>
| > for both keyA and keyB from other nodes, therefore, all OOB<br>
| > threads from both nodes call RPC to the non-primary owner but<br>
| > there's noone who could process the request.<br>
| ><br>
| > While we wait for the requests to timeout, the nodes with depleted<br>
| > OOB threadpools start suspecting all other nodes because they<br>
| > can't receive heartbeats etc...<br>
| ><br>
| > You can say "increase your OOB tp size", but that's not always an<br>
| > option, I have currently set it to 1000 threads and it's not<br>
| > enough. In the end, I will be always limited by RAM and something<br>
| > tells me that even nodes with few gigs of RAM should be able to<br>
| > form a huge cluster. We use 160 hotrod worker threads in JDG, that<br>
| > means that 160 * clusterSize = 10240 (64 nodes in my cluster)<br>
| > parallel requests can be executed, and if 10% targets the same<br>
| > node with 1000 OOB threads, it stucks. It's about scaling and<br>
| > robustness.<br>
| ><br>
| > Not that I'd have any good solution, but I'd really like to start a<br>
| > discussion.<br>
| > Thinking about it a bit, the problem is that blocking call (calling<br>
| > RPC on primary owner from message handler) can block non-blocking<br>
| > calls (such as RPC response or command that never sends any more<br>
| > messages). Therefore, having a flag on message "this won't send<br>
| > another message" could let the message be executed in different<br>
| > threadpool, which will be never deadlocked. In fact, the pools<br>
| > could share the threads but the non-blocking would have always a<br>
| > few threads spare.<br>
| > It's a bad solution as maintaining which message could block in the<br>
| > other node is really, really hard (we can be sure only in case of<br>
| > RPC responses), especially when some locks come. I will welcome<br>
| > anything better.<br>
|<br>
| --<br>
| Bela Ban, JGroups lead (<a href="http://www.jgroups.org" target="_blank">http://www.jgroups.org</a>)<br>
|<br>
| _______________________________________________<br>
| infinispan-dev mailing list<br>
| <a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
| <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
|<br>
_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
</div></div></blockquote></div><br></div></div>