2:40 p.m.
Hi,
since Infinispan moved to JGroups 3.4, we're experiencing occassional
deadlocks in some tests - most of threads that send anything over
JGroups are waiting in JGroups' FlowControl.decrementCredits. The
problem sometimes goes away after several seconds, but it produces some
ugly spikes in our througput/response time charts. Originally this
affected just some RadarGun tests but this is appearing in some
client-server tests as well (we've recently investigated an issue where
this appeared in a regular soak test).
I was looking into that [1] for some time but haven't really figured out
the cause. The workaround is to set up MFC and UFC credits high enough
(I use 10M) and stuff works then. I was trying to reproduce that on pure
JGroups, but unsuccessfully.
I am not asking anyone to dig into that, but I wanted to know whether QA
is alone experiencing that or if there are more of us.
Radim
[1] https://issues.jboss.org/browse/JGRP-1675
--
Radim Vansa <rvansa(a)redhat.com>
JBoss DataGrid QA
2:51 p.m.
Udp or tcp?
On Oct 11, 2013, at 8:41, Radim Vansa <rvansa(a)redhat.com> wrote:
Hi,
since Infinispan moved to JGroups 3.4, we're experiencing occassional
deadlocks in some tests - most of threads that send anything over
JGroups are waiting in JGroups' FlowControl.decrementCredits. The
problem sometimes goes away after several seconds, but it produces some
ugly spikes in our througput/response time charts. Originally this
affected just some RadarGun tests but this is appearing in some
client-server tests as well (we've recently investigated an issue where
this appeared in a regular soak test).
I was looking into that [1] for some time but haven't really figured out
the cause. The workaround is to set up MFC and UFC credits high enough
(I use 10M) and stuff works then. I was trying to reproduce that on pure
JGroups, but unsuccessfully.
I am not asking anyone to dig into that, but I wanted to know whether QA
is alone experiencing that or if there are more of us.
Radim
[1] https://issues.jboss.org/browse/JGRP-1675
--
Radim Vansa <rvansa(a)redhat.com>
JBoss DataGrid QA
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
3:12 p.m.
I remember seeing that mostly on UDP, but I've had an issue with this
over TCP as well on the cross-site link with multiple site masters
(synchronous 1pc xs replication).
Radim
On 10/11/2013 02:51 PM, Ray Tsang wrote:
Udp or tcp?
On Oct 11, 2013, at 8:41, Radim Vansa <rvansa(a)redhat.com> wrote:
> Hi,
>
> since Infinispan moved to JGroups 3.4, we're experiencing occassional
> deadlocks in some tests - most of threads that send anything over
> JGroups are waiting in JGroups' FlowControl.decrementCredits. The
> problem sometimes goes away after several seconds, but it produces some
> ugly spikes in our througput/response time charts. Originally this
> affected just some RadarGun tests but this is appearing in some
> client-server tests as well (we've recently investigated an issue where
> this appeared in a regular soak test).
> I was looking into that [1] for some time but haven't really figured out
> the cause. The workaround is to set up MFC and UFC credits high enough
> (I use 10M) and stuff works then. I was trying to reproduce that on pure
> JGroups, but unsuccessfully.
> I am not asking anyone to dig into that, but I wanted to know whether QA
> is alone experiencing that or if there are more of us.
>
> Radim
>
> [1] https://issues.jboss.org/browse/JGRP-1675
>
> --
> Radim Vansa <rvansa(a)redhat.com>
> JBoss DataGrid QA
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Radim Vansa <rvansa(a)redhat.com>
JBoss DataGrid QA
3:21 p.m.
I've seen StateTransferLargeObjectTest hang on my machine with all OOB
threads waiting in FlowControl$Credit.decrementIfEnoughCredits, but I
thought that was because the JGroups internal thread pool wasn't enabled in
the test configuration. Now that I've seen it's enabled by default, I think
it could be a JGroups problem after all.
I'll enable TRACE logging for org.jgroups and hopefully I'll reproduce it
again. For the record, the test suite is running with TCP, and the internal
thread pool has queueing enabled by default, so I don't think any REPLENISH
message could be lost.
Cheers
Dan
On Fri, Oct 11, 2013 at 4:12 PM, Radim Vansa <rvansa(a)redhat.com> wrote:
I remember seeing that mostly on UDP, but I've had an issue with
this
over TCP as well on the cross-site link with multiple site masters
(synchronous 1pc xs replication).
Radim
On 10/11/2013 02:51 PM, Ray Tsang wrote:
> Udp or tcp?
>
> On Oct 11, 2013, at 8:41, Radim Vansa <rvansa(a)redhat.com> wrote:
>
>> Hi,
>>
>> since Infinispan moved to JGroups 3.4, we're experiencing occassional
>> deadlocks in some tests - most of threads that send anything over
>> JGroups are waiting in JGroups' FlowControl.decrementCredits. The
>> problem sometimes goes away after several seconds, but it produces some
>> ugly spikes in our througput/response time charts. Originally this
>> affected just some RadarGun tests but this is appearing in some
>> client-server tests as well (we've recently investigated an issue where
>> this appeared in a regular soak test).
>> I was looking into that [1] for some time but haven't really figured out
>> the cause. The workaround is to set up MFC and UFC credits high enough
>> (I use 10M) and stuff works then. I was trying to reproduce that on pure
>> JGroups, but unsuccessfully.
>> I am not asking anyone to dig into that, but I wanted to know whether QA
>> is alone experiencing that or if there are more of us.
>>
>> Radim
>>
>> [1] https://issues.jboss.org/browse/JGRP-1675
>>
>> --
>> Radim Vansa <rvansa(a)redhat.com>
>> JBoss DataGrid QA
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Radim Vansa <rvansa(a)redhat.com>
JBoss DataGrid QA
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
4:26 p.m.
On 11 October 2013 14:21, Dan Berindei <dan.berindei(a)gmail.com> wrote:
I've seen StateTransferLargeObjectTest hang on my machine with
all OOB
threads waiting in FlowControl$Credit.decrementIfEnoughCredits, but I
thought that was because the JGroups internal thread pool wasn't enabled in
the test configuration. Now that I've seen it's enabled by default, I think
it could be a JGroups problem after all.
Seems you think that the testsuite setup could make some tests behave
differently than expected, that's quite a bad sign on the testsuite
quality.
I would argue then that these threadpools should not be disabled, or
that these tests should not be end-to-end tests including a JGroups
stack, but probably replace the channel with an in-memory copy of the
buffers.
Sanne
1:18 p.m.
On Fri, Oct 11, 2013 at 5:26 PM, Sanne Grinovero <sanne(a)infinispan.org>wrote:
On 11 October 2013 14:21, Dan Berindei <dan.berindei(a)gmail.com>
wrote:
> I've seen StateTransferLargeObjectTest hang on my machine with all OOB
> threads waiting in FlowControl$Credit.decrementIfEnoughCredits, but I
> thought that was because the JGroups internal thread pool wasn't enabled
in
> the test configuration. Now that I've seen it's enabled by default, I
think
> it could be a JGroups problem after all.
Seems you think that the testsuite setup could make some tests behave
differently than expected, that's quite a bad sign on the testsuite
quality.
The JGroups configuration, in particular the number of
OOB/INT/remote-executor threads certainly makes a difference for a lot of
tests, because they rely on being able to run n commands in parallel, in
order to achieve a certain sequence of events. I don't see that as a bad
thing, the only problem is that we don't know exactly how many threads each
test needs.
I would argue then that these threadpools should not be disabled, or
that these tests should not be end-to-end tests including a JGroups
stack, but probably replace the channel with an in-memory copy of the
buffers.
To clarify, the internal thread pool was introduced in JGroups specifically
to address this deadlock: a node can't send any message because it doesn't
have enough credits, yet it can't receive more credits because all of its
OOB threads are blocked sending a message. It was never disabled, we just
never enabled it explicitly in the test suite and I wasn't aware that it
was enabled by default. But it seems the deadlock can appear even when the
internal thread pool is enabled.
StateTransferLargeObjectTest is a stress test that checks how state
transfer and then remote reads are handled when the amount of data is very
large. I'm not sure how useful it would be if we didn't run it with a real
JGroups stack. Perhaps there are other tests that would benefit from being
isolated from the peculiarities of JGroups, but I don't think this is it.
True, the test suite's JGroups stack uses TCP so it doesn't really need
flow control, but if a test hanging help us solve a real issue then I see
it as a positive thing, not a negative one. So unless we find that dropping
UFC and/or TCP speeds up the test suite considerably, I'd keep it just as
it is.
Cheers
Dan
4:11 p.m.
On 10/11/13 3:21 PM, Dan Berindei wrote:
I've seen StateTransferLargeObjectTest hang on my machine with
all OOB
threads waiting in FlowControl$Credit.decrementIfEnoughCredits, but I
thought that was because the JGroups internal thread pool wasn't enabled
in the test configuration. Now that I've seen it's enabled by default, I
think it could be a JGroups problem after all.
Would be good if you could reproduce it
I'll enable TRACE logging for org.jgroups and hopefully I'll
reproduce
it again. For the record, the test suite is running with TCP, and the
internal thread pool has queueing enabled by default, so I don't think
any REPLENISH message could be lost.
No, but it could be *stuck* behind other messages, and so delivery could
be stuck or heavily delayed....
Cheers
Dan
On Fri, Oct 11, 2013 at 4:12 PM, Radim Vansa <rvansa(a)redhat.com
<mailto:rvansa@redhat.com>> wrote:
I remember seeing that mostly on UDP, but I've had an issue with this
over TCP as well on the cross-site link with multiple site masters
(synchronous 1pc xs replication).
Radim
On 10/11/2013 02:51 PM, Ray Tsang wrote:
> Udp or tcp?
>
> On Oct 11, 2013, at 8:41, Radim Vansa <rvansa(a)redhat.com
<mailto:rvansa@redhat.com>> wrote:
>
>> Hi,
>>
>> since Infinispan moved to JGroups 3.4, we're experiencing
occassional
>> deadlocks in some tests - most of threads that send anything over
>> JGroups are waiting in JGroups' FlowControl.decrementCredits. The
>> problem sometimes goes away after several seconds, but it
produces some
>> ugly spikes in our througput/response time charts. Originally this
>> affected just some RadarGun tests but this is appearing in some
>> client-server tests as well (we've recently investigated an
issue where
>> this appeared in a regular soak test).
>> I was looking into that [1] for some time but haven't really
figured out
>> the cause. The workaround is to set up MFC and UFC credits high
enough
>> (I use 10M) and stuff works then. I was trying to reproduce that
on pure
>> JGroups, but unsuccessfully.
>> I am not asking anyone to dig into that, but I wanted to know
whether QA
>> is alone experiencing that or if there are more of us.
>>
>> Radim
>>
>> [1] https://issues.jboss.org/browse/JGRP-1675
>>
>> --
>> Radim Vansa <rvansa(a)redhat.com <mailto:rvansa@redhat.com>>
>> JBoss DataGrid QA
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Radim Vansa <rvansa(a)redhat.com <mailto:rvansa@redhat.com>>
JBoss DataGrid QA
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org <mailto:infinispan-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Bela Ban, JGroups lead (http://www.jgroups.org)
3:22 p.m.
Hi Radim,
If you're using VMs, this may be of use. I actually experienced a JGroups
lockup in production with FC in decrementCredits. What happened in my case
is that virtualization tools had actually screwed with the system clock --
it was reported that JBoss took ~1 year to start.
http://www.vmware.com/pdf/vmware_timekeeping.pdf
The upshot is that I had to take FC out of my stack going forward.
Erik
-----Original Message-----
From: infinispan-dev-bounces(a)lists.jboss.org
[mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Radim Vansa
Sent: Friday, October 11, 2013 8:41 AM
To: infinispan-dev(a)lists.jboss.org
Subject: [infinispan-dev] Is anyone else experiencing JGRP-1675
Hi,
since Infinispan moved to JGroups 3.4, we're experiencing occassional
deadlocks in some tests - most of threads that send anything over JGroups
are waiting in JGroups' FlowControl.decrementCredits. The problem sometimes
goes away after several seconds, but it produces some ugly spikes in our
througput/response time charts. Originally this affected just some RadarGun
tests but this is appearing in some client-server tests as well (we've
recently investigated an issue where this appeared in a regular soak test).
I was looking into that [1] for some time but haven't really figured out the
cause. The workaround is to set up MFC and UFC credits high enough (I use
10M) and stuff works then. I was trying to reproduce that on pure JGroups,
but unsuccessfully.
I am not asking anyone to dig into that, but I wanted to know whether QA is
alone experiencing that or if there are more of us.
Radim
[1] https://issues.jboss.org/browse/JGRP-1675
--
Radim Vansa <rvansa(a)redhat.com>
JBoss DataGrid QA
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
3:35 p.m.
also, don't forget this one: https://issues.jboss.org/browse/ISPN-3334
On Fri, Oct 11, 2013 at 9:22 AM, Erik Salter <an1310(a)hotmail.com> wrote:
Hi Radim,
If you're using VMs, this may be of use. I actually experienced a JGroups
lockup in production with FC in decrementCredits. What happened in my case
is that virtualization tools had actually screwed with the system clock --
it was reported that JBoss took ~1 year to start.
http://www.vmware.com/pdf/vmware_timekeeping.pdf
The upshot is that I had to take FC out of my stack going forward.
Erik
-----Original Message-----
From: infinispan-dev-bounces(a)lists.jboss.org
[mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Radim Vansa
Sent: Friday, October 11, 2013 8:41 AM
To: infinispan-dev(a)lists.jboss.org
Subject: [infinispan-dev] Is anyone else experiencing JGRP-1675
Hi,
since Infinispan moved to JGroups 3.4, we're experiencing occassional
deadlocks in some tests - most of threads that send anything over JGroups
are waiting in JGroups' FlowControl.decrementCredits. The problem sometimes
goes away after several seconds, but it produces some ugly spikes in our
througput/response time charts. Originally this affected just some RadarGun
tests but this is appearing in some client-server tests as well (we've
recently investigated an issue where this appeared in a regular soak test).
I was looking into that [1] for some time but haven't really figured out
the
cause. The workaround is to set up MFC and UFC credits high enough (I use
10M) and stuff works then. I was trying to reproduce that on pure JGroups,
but unsuccessfully.
I am not asking anyone to dig into that, but I wanted to know whether QA is
alone experiencing that or if there are more of us.
Radim
[1] https://issues.jboss.org/browse/JGRP-1675
--
Radim Vansa <rvansa(a)redhat.com>
JBoss DataGrid QA
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
3:54 p.m.
I changed the default config some time ago, but I forgot to close ISPN-3334.
On Fri, Oct 11, 2013 at 4:35 PM, Ray Tsang <saturnism(a)gmail.com> wrote:
also, don't forget this one:
https://issues.jboss.org/browse/ISPN-3334
On Fri, Oct 11, 2013 at 9:22 AM, Erik Salter <an1310(a)hotmail.com> wrote:
> Hi Radim,
>
> If you're using VMs, this may be of use. I actually experienced a JGroups
> lockup in production with FC in decrementCredits. What happened in my
> case
> is that virtualization tools had actually screwed with the system clock --
> it was reported that JBoss took ~1 year to start.
>
> http://www.vmware.com/pdf/vmware_timekeeping.pdf
>
> The upshot is that I had to take FC out of my stack going forward.
>
> Erik
>
> -----Original Message-----
> From: infinispan-dev-bounces(a)lists.jboss.org
> [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Radim Vansa
> Sent: Friday, October 11, 2013 8:41 AM
> To: infinispan-dev(a)lists.jboss.org
> Subject: [infinispan-dev] Is anyone else experiencing JGRP-1675
>
> Hi,
>
> since Infinispan moved to JGroups 3.4, we're experiencing occassional
> deadlocks in some tests - most of threads that send anything over JGroups
> are waiting in JGroups' FlowControl.decrementCredits. The problem
> sometimes
> goes away after several seconds, but it produces some ugly spikes in our
> througput/response time charts. Originally this affected just some
> RadarGun
> tests but this is appearing in some client-server tests as well (we've
> recently investigated an issue where this appeared in a regular soak
> test).
> I was looking into that [1] for some time but haven't really figured out
> the
> cause. The workaround is to set up MFC and UFC credits high enough (I use
> 10M) and stuff works then. I was trying to reproduce that on pure JGroups,
> but unsuccessfully.
> I am not asking anyone to dig into that, but I wanted to know whether QA
> is
> alone experiencing that or if there are more of us.
>
> Radim
>
> [1] https://issues.jboss.org/browse/JGRP-1675
>
> --
> Radim Vansa <rvansa(a)redhat.com>
> JBoss DataGrid QA
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
9:01 a.m.
Huh, I've hit this independently sometime in the early spring and since
then I have a comment about that in our configs :) But this is certainly
not the issue now.
Radim
On 10/11/2013 03:54 PM, Dan Berindei wrote:
I changed the default config some time ago, but I forgot to close
ISPN-3334.
On Fri, Oct 11, 2013 at 4:35 PM, Ray Tsang <saturnism(a)gmail.com
<mailto:saturnism@gmail.com>> wrote:
also, don't forget this one: https://issues.jboss.org/browse/ISPN-3334
On Fri, Oct 11, 2013 at 9:22 AM, Erik Salter <an1310(a)hotmail.com
<mailto:an1310@hotmail.com>> wrote:
Hi Radim,
If you're using VMs, this may be of use. I actually
experienced a JGroups
lockup in production with FC in decrementCredits. What
happened in my case
is that virtualization tools had actually screwed with the
system clock --
it was reported that JBoss took ~1 year to start.
http://www.vmware.com/pdf/vmware_timekeeping.pdf
The upshot is that I had to take FC out of my stack going forward.
Erik
-----Original Message-----
From: infinispan-dev-bounces(a)lists.jboss.org
<mailto:infinispan-dev-bounces@lists.jboss.org>
[mailto:infinispan-dev-bounces@lists.jboss.org
<mailto:infinispan-dev-bounces@lists.jboss.org>] On Behalf Of
Radim Vansa
Sent: Friday, October 11, 2013 8:41 AM
To: infinispan-dev(a)lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>
Subject: [infinispan-dev] Is anyone else experiencing JGRP-1675
Hi,
since Infinispan moved to JGroups 3.4, we're experiencing
occassional
deadlocks in some tests - most of threads that send anything
over JGroups
are waiting in JGroups' FlowControl.decrementCredits. The
problem sometimes
goes away after several seconds, but it produces some ugly
spikes in our
througput/response time charts. Originally this affected just
some RadarGun
tests but this is appearing in some client-server tests as
well (we've
recently investigated an issue where this appeared in a
regular soak test).
I was looking into that [1] for some time but haven't really
figured out the
cause. The workaround is to set up MFC and UFC credits high
enough (I use
10M) and stuff works then. I was trying to reproduce that on
pure JGroups,
but unsuccessfully.
I am not asking anyone to dig into that, but I wanted to know
whether QA is
alone experiencing that or if there are more of us.
Radim
[1] https://issues.jboss.org/browse/JGRP-1675
--
Radim Vansa <rvansa(a)redhat.com <mailto:rvansa@redhat.com>>
JBoss DataGrid QA
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org <mailto:infinispan-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Radim Vansa <rvansa(a)redhat.com>
JBoss DataGrid QA
4:08 p.m.
(1) Taking flow control out of a config is a *bad thing* unless
* your message is small or
* you make a blocking RPC. In this case you could use NO_FC,
bypassing flow control. Not sure if Infinispan does this
(2) Use MFC/UFC rather than FC, which is deprecated
On 10/11/13 3:22 PM, Erik Salter wrote:
Hi Radim,
If you're using VMs, this may be of use. I actually experienced a JGroups
lockup in production with FC in decrementCredits. What happened in my case
is that virtualization tools had actually screwed with the system clock --
it was reported that JBoss took ~1 year to start.
http://www.vmware.com/pdf/vmware_timekeeping.pdf
The upshot is that I had to take FC out of my stack going forward.
Erik
-----Original Message-----
From: infinispan-dev-bounces(a)lists.jboss.org
[mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Radim Vansa
Sent: Friday, October 11, 2013 8:41 AM
To: infinispan-dev(a)lists.jboss.org
Subject: [infinispan-dev] Is anyone else experiencing JGRP-1675
Hi,
since Infinispan moved to JGroups 3.4, we're experiencing occassional
deadlocks in some tests - most of threads that send anything over JGroups
are waiting in JGroups' FlowControl.decrementCredits. The problem sometimes
goes away after several seconds, but it produces some ugly spikes in our
througput/response time charts. Originally this affected just some RadarGun
tests but this is appearing in some client-server tests as well (we've
recently investigated an issue where this appeared in a regular soak test).
I was looking into that [1] for some time but haven't really figured out the
cause. The workaround is to set up MFC and UFC credits high enough (I use
10M) and stuff works then. I was trying to reproduce that on pure JGroups,
but unsuccessfully.
I am not asking anyone to dig into that, but I wanted to know whether QA is
alone experiencing that or if there are more of us.
Radim
[1] https://issues.jboss.org/browse/JGRP-1675
--
Radim Vansa <rvansa(a)redhat.com>
JBoss DataGrid QA
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Bela Ban, JGroups lead (http://www.jgroups.org)
5:13 p.m.
On 10/16/2013 04:08 PM, Bela Ban wrote:
(1) Taking flow control out of a config is a *bad thing* unless
* your message is small or
* you make a blocking RPC. In this case you could use NO_FC,
bypassing flow control. Not sure if Infinispan does this
Infinispan mostly uses
dispatchers with blocking RPC - there are some
messages with ResponseMode.GET_NONE, but these are usually used only
after some blocking communication (I think that TxNotificationCommand is
sent this way). So, for Infinispan the FC may not be really needed.
(2) Use MFC/UFC rather than FC, which is deprecated
I think that FC is misleading
acronym for FlowControl - as MFC and UFC
inherit from FlowControl, shortening it to FC is touting.
Radim
On 10/11/13 3:22 PM, Erik Salter wrote:
> Hi Radim,
>
> If you're using VMs, this may be of use. I actually experienced a JGroups
> lockup in production with FC in decrementCredits. What happened in my case
> is that virtualization tools had actually screwed with the system clock --
> it was reported that JBoss took ~1 year to start.
>
> http://www.vmware.com/pdf/vmware_timekeeping.pdf
>
> The upshot is that I had to take FC out of my stack going forward.
>
> Erik
>
> -----Original Message-----
> From: infinispan-dev-bounces(a)lists.jboss.org
> [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Radim Vansa
> Sent: Friday, October 11, 2013 8:41 AM
> To: infinispan-dev(a)lists.jboss.org
> Subject: [infinispan-dev] Is anyone else experiencing JGRP-1675
>
> Hi,
>
> since Infinispan moved to JGroups 3.4, we're experiencing occassional
> deadlocks in some tests - most of threads that send anything over JGroups
> are waiting in JGroups' FlowControl.decrementCredits. The problem sometimes
> goes away after several seconds, but it produces some ugly spikes in our
> througput/response time charts. Originally this affected just some RadarGun
> tests but this is appearing in some client-server tests as well (we've
> recently investigated an issue where this appeared in a regular soak test).
> I was looking into that [1] for some time but haven't really figured out the
> cause. The workaround is to set up MFC and UFC credits high enough (I use
> 10M) and stuff works then. I was trying to reproduce that on pure JGroups,
> but unsuccessfully.
> I am not asking anyone to dig into that, but I wanted to know whether QA is
> alone experiencing that or if there are more of us.
>
> Radim
>
> [1] https://issues.jboss.org/browse/JGRP-1675
>
> --
> Radim Vansa <rvansa(a)redhat.com>
> JBoss DataGrid QA
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
--
Radim Vansa <rvansa(a)redhat.com>
JBoss DataGrid QA
12:49 p.m.
I was able to reproduce the deadlock reliably with a stress test:
https://issues.jboss.org/browse/JGRP-1675?focusedCommentId=12823031&p...
On Wed, Oct 16, 2013 at 6:13 PM, Radim Vansa <rvansa(a)redhat.com> wrote:
On 10/16/2013 04:08 PM, Bela Ban wrote:
> (1) Taking flow control out of a config is a *bad thing* unless
> * your message is small or
> * you make a blocking RPC. In this case you could use NO_FC,
> bypassing flow control. Not sure if Infinispan does this
Infinispan mostly uses dispatchers with blocking RPC - there are some
messages with ResponseMode.GET_NONE, but these are usually used only
after some blocking communication (I think that TxNotificationCommand is
sent this way). So, for Infinispan the FC may not be really needed.
Infinispan RPCs are indeed synchonous most of the time, but they aren't
necessarily small. Especially for state transfer, but also for Map/Reduce
(see https://issues.jboss.org/browse/ISPN-3375) so I'm pretty sure we still
need flow control.
>
> (2) Use MFC/UFC rather than FC, which is deprecated
I think that FC is misleading acronym for FlowControl - as MFC and UFC
inherit from FlowControl, shortening it to FC is touting.
I'm also guilty of abbreviating UFC/MFC to FC many times. Bela, you can
rest assured that none of our configs use the actual FC protocol.
Radim
>
> On 10/11/13 3:22 PM, Erik Salter wrote:
>> Hi Radim,
>>
>> If you're using VMs, this may be of use. I actually experienced a
JGroups
>> lockup in production with FC in decrementCredits. What happened in my
case
>> is that virtualization tools had actually screwed with the system clock
--
>> it was reported that JBoss took ~1 year to start.
>>
>> http://www.vmware.com/pdf/vmware_timekeeping.pdf
>>
>> The upshot is that I had to take FC out of my stack going forward.
>>
>> Erik
>>
>> -----Original Message-----
>> From: infinispan-dev-bounces(a)lists.jboss.org
>> [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Radim
Vansa
>> Sent: Friday, October 11, 2013 8:41 AM
>> To: infinispan-dev(a)lists.jboss.org
>> Subject: [infinispan-dev] Is anyone else experiencing JGRP-1675
>>
>> Hi,
>>
>> since Infinispan moved to JGroups 3.4, we're experiencing occassional
>> deadlocks in some tests - most of threads that send anything over
JGroups
>> are waiting in JGroups' FlowControl.decrementCredits. The problem
sometimes
>> goes away after several seconds, but it produces some ugly spikes in our
>> througput/response time charts. Originally this affected just some
RadarGun
>> tests but this is appearing in some client-server tests as well (we've
>> recently investigated an issue where this appeared in a regular soak
test).
>> I was looking into that [1] for some time but haven't really figured
out the
>> cause. The workaround is to set up MFC and UFC credits high enough (I
use
>> 10M) and stuff works then. I was trying to reproduce that on pure
JGroups,
>> but unsuccessfully.
>> I am not asking anyone to dig into that, but I wanted to know whether
QA is
>> alone experiencing that or if there are more of us.
>>
>> Radim
>>
>> [1] https://issues.jboss.org/browse/JGRP-1675
>>
>> --
>> Radim Vansa <rvansa(a)redhat.com>
>> JBoss DataGrid QA
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
--
Radim Vansa <rvansa(a)redhat.com>
JBoss DataGrid QA
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
1:14 p.m.
On Oct 18, 2013, at 11:49 AM, Dan Berindei <dan.berindei(a)gmail.com> wrote:
I was able to reproduce the deadlock reliably with a stress test:
https://issues.jboss.org/browse/JGRP-1675?focusedCommentId=12823031&p...
Nice one!
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
4:05 p.m.
On 10/11/13 2:40 PM, Radim Vansa wrote:
Hi,
since Infinispan moved to JGroups 3.4, we're experiencing occassional
deadlocks in some tests - most of threads that send anything over
JGroups are waiting in JGroups' FlowControl.decrementCredits.
Are those real deadlocks ? Meaning, the system never recovers ? If not,
then it's just flow control doing its job and preventing a fast sender
from overrunning a busy receiver.
For example, if the receiver is busy processing something or locked on a
lock acquisition, then it may not be able to send back credits and
blocks the sender until it is done. Also, if the receiver's thread pool
drops the message because the pool is full, no credits will be sent,
thus blocking the sender. This is a *good thing*, or else the pool would
be exhausted even more.
We really need to make a clear distinction between these 2 modes.
Naturally, if a receiver performs some blocking (as is done in
Infinispan), the sender should stop sending at some point, or the
receiver would simply drop all messages and cause a lot of retransmissions.
The
problem sometimes goes away after several seconds, but it produces some
ugly spikes in our througput/response time charts.
OK, good, so it's the latter: temp blocking caused by flow control.
Flow control *can* cause some hiccups every now and then, especially if
the receiver can block processing a message. The 200K credits is pretty
low (unless you send very small messages), but the 10M mentioned here
might be too much, I'd suggest a middle ground, e.g. 2-4 MB (default).
With the new INTERNAL thread pool, these blockings will *not* go away,
as credits (even sent INTERNAL) won't get sent in the first place until
the receiver(s) process the messages...
Originally this
affected just some RadarGun tests but this is appearing in some
client-server tests as well (we've recently investigated an issue where
this appeared in a regular soak test).
I was looking into that [1] for some time but haven't really figured out
the cause. The workaround is to set up MFC and UFC credits high enough
(I use 10M) and stuff works then. I was trying to reproduce that on pure
JGroups, but unsuccessfully.
I am not asking anyone to dig into that, but I wanted to know whether QA
is alone experiencing that or if there are more of us.
Radim
[1] https://issues.jboss.org/browse/JGRP-1675
--
Bela Ban, JGroups lead (http://www.jgroups.org)
4096
days inactive
4103
days old
infinispan-dev@lists.jboss.org
15 comments
7 participants
participants (7)
-
Bela Ban -
Dan Berindei -
Erik Salter -
Mircea Markus -
Radim Vansa -
Ray Tsang -
Sanne Grinovero