4:35 a.m.
On 12 Nov 2008, at 10:25, Vladimir Blagojevic wrote:
Manik Surtani wrote:
>
> Yes, this was always an issue with the way we used FLUSH - that
> someone in the group could initiate a FLUSH and then die leaving
> other members' flushBlockGates closed. TBH, apart from adding
> timeouts to the flushBlockGate, I can't see how we would get around
> this.
Me too. I am confused how all these issues started to pop out now.
How come they slipped for so long?
Yes, because in theory they could have happened with JBC 2.x, 1.x, etc.
> Vladimir/Bela - in the scenario described (node initiates a FLUSH
> and then dies) would other nodes still see a view change relating
> to the node dying?
They would. However, I will add this test case to verify it.
Ok - so we could add an extra check into the view change listener to
force an unblock if a member who initiated a FLUSH dies. We would
also have to record the address of the member initiating the FLUSH in
the flushBlockGate.
Cheers
--
Manik Surtani
Lead, JBoss Cache
manik(a)jboss.org
4:55 a.m.
New subject: Cache unable to write to cluster
Manik Surtani wrote:
Ok - so we could add an extra check into the view change listener to
force an unblock if a member who initiated a FLUSH dies. We would
also have to record the address of the member initiating the FLUSH in
the flushBlockGate.
+1.
I'm also puzzled as to why this could evade detection for so long...
Have we started to use channels in a different way now ? E.g. concurrent
startup ?
--
Bela Ban
Lead JGroups / Clustering Team
JBoss - a division of Red Hat
9:01 a.m.
New subject: Cache unable to write to cluster
Bela Ban wrote:
Manik Surtani wrote:
> Ok - so we could add an extra check into the view change listener to
> force an unblock if a member who initiated a FLUSH dies. We would
> also have to record the address of the member initiating the FLUSH in
> the flushBlockGate.
+1.
I'm also puzzled as to why this could evade detection for so long...
Have we started to use channels in a different way now ? E.g. concurrent
startup ?
3 possible factors:
1) The AS now creates/starts a cache when it needs it, i.e. as part of
deploy of a clustered webapp or SFSB, rather than at AS start. Effect
of this is during a testsuite run there is a lot more starting/stopping
of caches and associated channels than there was back in the day. So
intermittent failures will happen more. This is a fairly old change
though. But for sure it increases odds of these failures vs. say the
first half of this year.
2) AS upgraded to 2.6.5 on October 15. I saw and reported an
intermittent flush failure on October 18. Maybe a relationship; don't know.
3) In our second week in Brno, I changed the protocol stacks to add > 1
min_threads to the pools. Clebert seems to feel the JBM failures he
started reported popped up when he changed the stack JBM is testing
against to match the AS stack; making localhost=true and min_threads > 1
were the most significant changes.
--
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberry(a)redhat.com
2:33 p.m.
New subject: Cache unable to write to cluster
Bela Ban wrote:
Manik Surtani wrote:
> Ok - so we could add an extra check into the view change listener to
> force an unblock if a member who initiated a FLUSH dies. We would
> also have to record the address of the member initiating the FLUSH in
> the flushBlockGate.
+1.
Shouldn't this be handled by FLUSH itself? MembershipListener.block()
doesn't provide any information on who the flush coordinator is, so JBC
has no way to implement Manik's suggestion. But FLUSH itself should know
enough to detect the view change and trigger an unblock().
BTW, I see that the AS ClusterPartition has the same weakness.
--
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberry(a)redhat.com
12:57 a.m.
New subject: Cache unable to write to cluster
That's what I meant: FLUSH itself already listens for view changes...
Brian Stansberry wrote:
Bela Ban wrote:
>
>
> Manik Surtani wrote:
>> Ok - so we could add an extra check into the view change listener to
>> force an unblock if a member who initiated a FLUSH dies. We would
>> also have to record the address of the member initiating the FLUSH
>> in the flushBlockGate.
>
> +1.
>
Shouldn't this be handled by FLUSH itself? MembershipListener.block()
doesn't provide any information on who the flush coordinator is, so
JBC has no way to implement Manik's suggestion. But FLUSH itself
should know enough to detect the view change and trigger an unblock().
BTW, I see that the AS ClusterPartition has the same weakness.
--
Bela Ban
Lead JGroups / Clustering Team
JBoss - a division of Red Hat
7:04 a.m.
New subject: Cache unable to write to cluster
Yeah, I was talking about doing this in JBC - but that doesn't work
really. It would have to be in FLUSH.
On 14 Nov 2008, at 06:57, Bela Ban wrote:
That's what I meant: FLUSH itself already listens for view
changes...
Brian Stansberry wrote:
> Bela Ban wrote:
>>
>>
>> Manik Surtani wrote:
>>> Ok - so we could add an extra check into the view change listener
>>> to force an unblock if a member who initiated a FLUSH dies. We
>>> would also have to record the address of the member initiating
>>> the FLUSH in the flushBlockGate.
>>
>> +1.
>>
>
> Shouldn't this be handled by FLUSH itself?
> MembershipListener.block() doesn't provide any information on who
> the flush coordinator is, so JBC has no way to implement Manik's
> suggestion. But FLUSH itself should know enough to detect the view
> change and trigger an unblock().
>
> BTW, I see that the AS ClusterPartition has the same weakness.
>
>
--
Bela Ban
Lead JGroups / Clustering Team
JBoss - a division of Red Hat
--
Manik Surtani
Lead, JBoss Cache
manik(a)jboss.org
6:07 a.m.
New subject: Cache unable to write to cluster
On 13 Nov 2008, at 20:33, Brian Stansberry wrote:
Bela Ban wrote:
> Manik Surtani wrote:
>> Ok - so we could add an extra check into the view change listener
>> to force an unblock if a member who initiated a FLUSH dies. We
>> would also have to record the address of the member initiating the
>> FLUSH in the flushBlockGate.
> +1.
Shouldn't this be handled by FLUSH itself?
MembershipListener.block() doesn't provide any information on who
the flush coordinator is, so JBC has no way to implement Manik's
suggestion. But FLUSH itself should know enough to detect the view
change and trigger an unblock().
Good point.
BTW, I see that the AS ClusterPartition has the same weakness.
--
Manik Surtani
Lead, JBoss Cache
manik(a)jboss.org
5852
days inactive
5854
days old
jbosscache-dev@lists.jboss.org
6 comments
3 participants
participants (3)
-
Bela Ban -
Brian Stansberry -
Manik Surtani