<div dir="ltr">I was able to reproduce the deadlock reliably with a stress test: <a href="https://issues.jboss.org/browse/JGRP-1675?focusedCommentId=12823031&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12823031">https://issues.jboss.org/browse/JGRP-1675?focusedCommentId=12823031&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12823031</a><br>
<br><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Oct 16, 2013 at 6:13 PM, Radim Vansa <span dir="ltr"><<a href="mailto:rvansa@redhat.com" target="_blank">rvansa@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="im">On 10/16/2013 04:08 PM, Bela Ban wrote:<br>
> (1) Taking flow control out of a config is a *bad thing* unless<br>
> * your message is small or<br>
> * you make a blocking RPC. In this case you could use NO_FC,<br>
> bypassing flow control. Not sure if Infinispan does this<br>
</div>Infinispan mostly uses dispatchers with blocking RPC - there are some<br>
messages with ResponseMode.GET_NONE, but these are usually used only<br>
after some blocking communication (I think that TxNotificationCommand is<br>
sent this way). So, for Infinispan the FC may not be really needed.<br></blockquote><div><br></div><div>Infinispan RPCs are indeed synchonous most of the time, but they aren't necessarily small. Especially for state transfer, but also for Map/Reduce (see <a href="https://issues.jboss.org/browse/ISPN-3375">https://issues.jboss.org/browse/ISPN-3375</a>) so I'm pretty sure we still need flow control.<br>
<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class="im"><br>
><br>
> (2) Use MFC/UFC rather than FC, which is deprecated<br>
</div>I think that FC is misleading acronym for FlowControl - as MFC and UFC<br>
inherit from FlowControl, shortening it to FC is touting.<br></blockquote><div><br></div><div>I'm also guilty of abbreviating UFC/MFC to FC many times. Bela, you can rest assured that none of our configs use the actual FC protocol.<br>
<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span class=""><font color="#888888"><br>
Radim<br>
</font></span><div class=""><div class="h5"><br>
><br>
> On 10/11/13 3:22 PM, Erik Salter wrote:<br>
>> Hi Radim,<br>
>><br>
>> If you're using VMs, this may be of use. I actually experienced a JGroups<br>
>> lockup in production with FC in decrementCredits. What happened in my case<br>
>> is that virtualization tools had actually screwed with the system clock --<br>
>> it was reported that JBoss took ~1 year to start.<br>
>><br>
>> <a href="http://www.vmware.com/pdf/vmware_timekeeping.pdf" target="_blank">http://www.vmware.com/pdf/vmware_timekeeping.pdf</a><br>
>><br>
>> The upshot is that I had to take FC out of my stack going forward.<br>
>><br>
>> Erik<br>
>><br>
>> -----Original Message-----<br>
>> From: <a href="mailto:infinispan-dev-bounces@lists.jboss.org">infinispan-dev-bounces@lists.jboss.org</a><br>
>> [mailto:<a href="mailto:infinispan-dev-bounces@lists.jboss.org">infinispan-dev-bounces@lists.jboss.org</a>] On Behalf Of Radim Vansa<br>
>> Sent: Friday, October 11, 2013 8:41 AM<br>
>> To: <a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
>> Subject: [infinispan-dev] Is anyone else experiencing JGRP-1675<br>
>><br>
>> Hi,<br>
>><br>
>> since Infinispan moved to JGroups 3.4, we're experiencing occassional<br>
>> deadlocks in some tests - most of threads that send anything over JGroups<br>
>> are waiting in JGroups' FlowControl.decrementCredits. The problem sometimes<br>
>> goes away after several seconds, but it produces some ugly spikes in our<br>
>> througput/response time charts. Originally this affected just some RadarGun<br>
>> tests but this is appearing in some client-server tests as well (we've<br>
>> recently investigated an issue where this appeared in a regular soak test).<br>
>> I was looking into that [1] for some time but haven't really figured out the<br>
>> cause. The workaround is to set up MFC and UFC credits high enough (I use<br>
>> 10M) and stuff works then. I was trying to reproduce that on pure JGroups,<br>
>> but unsuccessfully.<br>
>> I am not asking anyone to dig into that, but I wanted to know whether QA is<br>
>> alone experiencing that or if there are more of us.<br>
>><br>
>> Radim<br>
>><br>
>> [1] <a href="https://issues.jboss.org/browse/JGRP-1675" target="_blank">https://issues.jboss.org/browse/JGRP-1675</a><br>
>><br>
>> --<br>
>> Radim Vansa <<a href="mailto:rvansa@redhat.com">rvansa@redhat.com</a>><br>
>> JBoss DataGrid QA<br>
>><br>
>> _______________________________________________<br>
>> infinispan-dev mailing list<br>
>> <a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
>> <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
>><br>
>> _______________________________________________<br>
>> infinispan-dev mailing list<br>
>> <a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
>> <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
>><br>
<br>
<br>
--<br>
Radim Vansa <<a href="mailto:rvansa@redhat.com">rvansa@redhat.com</a>><br>
JBoss DataGrid QA<br>
<br>
_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
</div></div></blockquote></div><br></div></div>