Is solution D out of the picture already? I mean if we really can't find a good enough
solution to solve it, why not just accpet it. State transfer, IMO, should not happen that
often anyway, if it is just a new node joining. I think the important thing is keep the
If it is because of network instability, then we will see lots of tx problems anyway. Are
we throwing more fuel into the fire?
[mailto:email@example.com] On Behalf Of Bela Ban
Sent: Wednesday, September 27, 2006 4:37 PM
To: Brian Stansberry
Subject: Re: [jbosscache-dev] Issues with FLUSH and JBC
What's the consensus as to how wwe should proceed ?
1. Solution A with not blocking unicasts during a flush or
2. Solution B where we block later (on FLUSH_COMPLETED) rather than
Vladimir is for #2.
How about adding of the unblock() callback in a separate listener interface ? I'd
rather add this sooner than later. We would, however, also have to make sure that we
actually do call this method.
Vladimir: let's have a call on this today, so we can see how to proceed...
Brian Stansberry wrote:
Bela Ban wrote:
> Okay, my comments will be available in book form at Prentice hall
> this fall... :-)
LOL. I'll try to reform. At least my overly long messages are on
e-mail and so don't kill trees. :)
> Just kidding, here are some comments:
> * I don't want to change the entire implementation of FLUSH this
> late, 2.4 is overdue for a final release. So option B doesn't like
> that appealing to me o OTOH: if we can resolve the issue, why not...
> * A: what if we block only **multicast** messages, but not
> **unicast** messages ? This would solve issue A, but maybe there are
> use cases that it won't solve... We can assume that unicast messages
> are always responses to multicasts, so they should be allowed to
> complete. If this solution flies, then we have a quickfix for our
> problem and can *really* cleanly fix it in the next release...
We'd need to be sure JBC didn't make any unicast calls (besides RPC
responses) during the state transfer. Possible unicast calls I can
think of are:
1) Request for partial state transfer (with the current RPC-based
mechanism). E.g. 3 node cluster, node B redeploys a webapp and asks
for partial state transfer while node C is doing an initial state transfer.
This would be an odd case though; typically you disable initial state
transfer if you're going to use the activate/inactivateRegion API.
2) Calls related to buddy group assignments. Need to think about this
a bit. But if they are using BR they won't be using initial state
transfer, so probably not an issue.
> * B: okay, but if my proposed solution above works, we can do this in
> * C: this is essentially implementing the flush protocol at the
> application level, which is not a bad idea because the app always has
> more information than JGroups. However, it is probably a bit too
> redundant, and also requires quite a number of changes, which is also
> later for JBC 1.4 (SP?)...
Yeah, it is a lot for 1.4. IMHO definitely moves it beyond the realm
of an SP2, into 1.4.1.
> * I might have to add an additional callback blockCompleted() or
> unblock() to JGroups, to notify members that the FLUSH phase has
> completed and everybody can resume sending messages.
> I'm currently
> investigating this... Downside: an API change, so possibly a new
> ExtendedXXX interface which would get merged in JGroups 3.0
This would be needed with B if our current algorithm for JBC is going
>> A downside of this idea is it changes the semantics of flush and
>> requires JGroups changes. We'd definitely like input from Bela on
>> this. Also, since we initially rejecting it, we haven't fully
>> thought it through. (As I'm editing this to send out I see there is
>> no way to tell JBC after it returns from block() to not let any
>> "new" activity through -- big hole. I'm back to rejecting this
> Here, we might have to introduce additional callbacks, e.g.
> - block(): stop sending messages. FLUSH doesn't block yet though, so
> if an app ignores the convention and keeps sending messages it will
> - No callback when FLUSH actually does block sending of messages
> - unblock(): called when the app can resume sending messages.
> FLUSH does not block sending of messages anymore
Yep. Our current algorithm does the following during the block() call:
1) Create a latch or something that prevents new transactions
acquiring locks or existing transactions proceeding into the 2PC (i.e.
2) Give transactions already in the 2PC time to complete. If they
don't, eventually roll them back.
3) Release the latch.
4) Immediately return from block(). (Vladimir -- problem here; there's
a race condition between threads released in #3 and the return from
block(). We need to figure out how to deal with that.)
We count on FLUSH preventing the threads released in #3 sending any
prepare() calls until the state transfer is done. Solution B breaks
this for the period until FLUSH_COMPLETED is sent.
An unblock() callback would help here, as we'd release the latch then.
> I don't think the semantic changes are that big, actually you could
> argue there are *no* semantic changes as block() is an inidication
> that message sending will block, here we're just saying it will block
> some time in the (near) future.
Lead JGroups / Manager JBoss Clustering Group JBoss - a division of Red Hat
jbosscache-dev mailing list