[infinispan-dev] Is anyone else experiencing JGRP-1675

[Dan Berindei]

I was able to reproduce the deadlock reliably with a stress test:
https://issues.jboss.org/browse/JGRP-1675?focusedCommentId=12823031&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12823031



On Wed, Oct 16, 2013 at 6:13 PM, Radim Vansa <rvansa at redhat.com> wrote:

> On 10/16/2013 04:08 PM, Bela Ban wrote:
> > (1) Taking flow control out of a config is a *bad thing* unless
> >        * your message is small or
> >        * you make a blocking RPC. In this case you could use NO_FC,
> > bypassing flow control. Not sure if Infinispan does this
> Infinispan mostly uses dispatchers with blocking RPC - there are some
> messages with ResponseMode.GET_NONE, but these are usually used only
> after some blocking communication (I think that TxNotificationCommand is
> sent this way). So, for Infinispan the FC may not be really needed.
>

Infinispan RPCs are indeed synchonous most of the time, but they aren't
necessarily small. Especially for state transfer, but also for Map/Reduce
(see https://issues.jboss.org/browse/ISPN-3375) so I'm pretty sure we still
need flow control.


> >
> > (2) Use MFC/UFC rather than FC, which is deprecated
> I think that FC is misleading acronym for FlowControl - as MFC and UFC
> inherit from FlowControl, shortening it to FC is touting.
>

I'm also guilty of abbreviating UFC/MFC to FC many times. Bela, you can
rest assured that none of our configs use the actual FC protocol.


>
> Radim
>
> >
> > On 10/11/13 3:22 PM, Erik Salter wrote:
> >> Hi Radim,
> >>
> >> If you're using VMs, this may be of use.  I actually experienced a
> JGroups
> >> lockup in production with FC in decrementCredits.  What happened in my
> case
> >> is that virtualization tools had actually screwed with the system clock
> --
> >> it was reported that JBoss took ~1 year to start.
> >>
> >> http://www.vmware.com/pdf/vmware_timekeeping.pdf
> >>
> >> The upshot is that I had to take FC out of my stack going forward.
> >>
> >> Erik
> >>
> >> -----Original Message-----
> >> From: infinispan-dev-bounces at lists.jboss.org
> >> [mailto:infinispan-dev-bounces at lists.jboss.org] On Behalf Of Radim
> Vansa
> >> Sent: Friday, October 11, 2013 8:41 AM
> >> To: infinispan-dev at lists.jboss.org
> >> Subject: [infinispan-dev] Is anyone else experiencing JGRP-1675
> >>
> >> Hi,
> >>
> >> since Infinispan moved to JGroups 3.4, we're experiencing occassional
> >> deadlocks in some tests - most of threads that send anything over
> JGroups
> >> are waiting in JGroups' FlowControl.decrementCredits. The problem
> sometimes
> >> goes away after several seconds, but it produces some ugly spikes in our
> >> througput/response time charts. Originally this affected just some
> RadarGun
> >> tests but this is appearing in some client-server tests as well (we've
> >> recently investigated an issue where this appeared in a regular soak
> test).
> >> I was looking into that [1] for some time but haven't really figured
> out the
> >> cause. The workaround is to set up MFC and UFC credits high enough (I
> use
> >> 10M) and stuff works then. I was trying to reproduce that on pure
> JGroups,
> >> but unsuccessfully.
> >> I am not asking anyone to dig into that, but I wanted to know whether
> QA is
> >> alone experiencing that or if there are more of us.
> >>
> >> Radim
> >>
> >> [1] https://issues.jboss.org/browse/JGRP-1675
> >>
> >> --
> >> Radim Vansa <rvansa at redhat.com>
> >> JBoss DataGrid QA
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
>
>
> --
> Radim Vansa <rvansa at redhat.com>
> JBoss DataGrid QA
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20131018/3bd62701/attachment.html