I was able to reproduce the deadlock reliably with a stress test:
https://issues.jboss.org/browse/JGRP-1675?focusedCommentId=12823031&p...
On Wed, Oct 16, 2013 at 6:13 PM, Radim Vansa <rvansa(a)redhat.com> wrote:
On 10/16/2013 04:08 PM, Bela Ban wrote:
> (1) Taking flow control out of a config is a *bad thing* unless
> * your message is small or
> * you make a blocking RPC. In this case you could use NO_FC,
> bypassing flow control. Not sure if Infinispan does this
Infinispan mostly uses dispatchers with blocking RPC - there are some
messages with ResponseMode.GET_NONE, but these are usually used only
after some blocking communication (I think that TxNotificationCommand is
sent this way). So, for Infinispan the FC may not be really needed.
Infinispan RPCs are indeed synchonous most of the time, but they aren't
necessarily small. Especially for state transfer, but also for Map/Reduce
(see
https://issues.jboss.org/browse/ISPN-3375) so I'm pretty sure we still
need flow control.
>
> (2) Use MFC/UFC rather than FC, which is deprecated
I think that FC is misleading acronym for FlowControl - as MFC and UFC
inherit from FlowControl, shortening it to FC is touting.
I'm also guilty of abbreviating UFC/MFC to FC many times. Bela, you can
rest assured that none of our configs use the actual FC protocol.
Radim
>
> On 10/11/13 3:22 PM, Erik Salter wrote:
>> Hi Radim,
>>
>> If you're using VMs, this may be of use. I actually experienced a
JGroups
>> lockup in production with FC in decrementCredits. What happened in my
case
>> is that virtualization tools had actually screwed with the system clock
--
>> it was reported that JBoss took ~1 year to start.
>>
>>
http://www.vmware.com/pdf/vmware_timekeeping.pdf
>>
>> The upshot is that I had to take FC out of my stack going forward.
>>
>> Erik
>>
>> -----Original Message-----
>> From: infinispan-dev-bounces(a)lists.jboss.org
>> [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Radim
Vansa
>> Sent: Friday, October 11, 2013 8:41 AM
>> To: infinispan-dev(a)lists.jboss.org
>> Subject: [infinispan-dev] Is anyone else experiencing JGRP-1675
>>
>> Hi,
>>
>> since Infinispan moved to JGroups 3.4, we're experiencing occassional
>> deadlocks in some tests - most of threads that send anything over
JGroups
>> are waiting in JGroups' FlowControl.decrementCredits. The problem
sometimes
>> goes away after several seconds, but it produces some ugly spikes in our
>> througput/response time charts. Originally this affected just some
RadarGun
>> tests but this is appearing in some client-server tests as well (we've
>> recently investigated an issue where this appeared in a regular soak
test).
>> I was looking into that [1] for some time but haven't really figured
out the
>> cause. The workaround is to set up MFC and UFC credits high enough (I
use
>> 10M) and stuff works then. I was trying to reproduce that on pure
JGroups,
>> but unsuccessfully.
>> I am not asking anyone to dig into that, but I wanted to know whether
QA is
>> alone experiencing that or if there are more of us.
>>
>> Radim
>>
>> [1]
https://issues.jboss.org/browse/JGRP-1675
>>
>> --
>> Radim Vansa <rvansa(a)redhat.com>
>> JBoss DataGrid QA
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
--
Radim Vansa <rvansa(a)redhat.com>
JBoss DataGrid QA
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev