<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Feb 26, 2013 at 12:57 PM, Pedro Ruivo <span dir="ltr"><<a href="mailto:pedro@infinispan.org" target="_blank">pedro@infinispan.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">hi,<br>
<br>
I found the blocking problem with the state transfer this morning. It happens because of the reordering of a regular and OOB message.<br>
<br>
Below, is a simplification of what is happening for two nodes<br>
<br>
A: total order broadcasts rebalance_start<br>
<br>
B: (incoming thread) delivers rebalance_start<br>
B: has no segments to request so the rebalance is done<br>
B: sends async request with rebalance_confirm (unicast #x)<br>
B: sends the rebalance_start response (unicast #x+1) (the response is a regular message)<br>
<br>
A: receives rebalance_start response (unicast #x+1)<br>
A: in UNICAST2, it detects the message is out-of-order and blocks the response in the sender window (i.e. the message #x is missing)<br>
A: receives the rebalance_confirm (unicast #x)<br>
A: delivers rebalance_confirm. Infinispan blocks this command until all the rebalance_start responses are received ==> this originates a deadlock! (because the response is blocked in unicast layer)<br>
<br>
Question: can the request's response message be sent always as OOB? (I think the answer should be no...)<br>
<br></blockquote><div><br></div><div>We could, if Bela adds the send(Message) method to the Response interface... and personally I think it would be better to make all responses OOB (as in JGroups 3.2.x). I don't have any data to back this up, though...<br>
<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
My suggestion: when I deliver a rebalance_confirm command (that it is send async), can I move it to a thread in async_thread_pool_executor?<br>
<br></blockquote><div><br><div>I have WIP fix for
<a href="https://issues.jboss.org/browse/ISPN-2825">https://issues.jboss.org/browse/ISPN-2825</a>, which should stop blocking the
REBALANCE_CONFIRM commands on the coordinator:
<a href="https://github.com/danberindei/infinispan/tree/t_2825_m">https://github.com/danberindei/infinispan/tree/t_2825_m</a><br><br></div>I
haven't issued a PR yet because I'm still getting a failure in
ClusterTopologyManagerTest, I think because of a JGroups issue (RSVP not receiving an ACK from itself). I'll let you know when I find out...<br><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Weird thing: last night I tried more than 5x time in a row with UNICAST3 and it never blocks. can this meaning a problem with UNICAST3 or I had just lucky?<br>
<br></blockquote><div><br>Even though the REBALANCE_CONFIRM command is sent async, the message is still OOB. I think UNICAST/2/3 should not block any regular message waiting for the processing of an OOB message, as long as that message was received, so maybe the problem is in UNICAST2?<br>
<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Any other suggestion?<br>
<br>
Cheers,<br>
Pedro<br>
<br>
<br>
</blockquote></div><br></div></div>