Following up discussion we had on jbosscache-dev. In summary, Brian found a fundemental problem in FLUSH that needed to be resolved before attacking JBCACHE-315. Problem is described below. FLUSH was retrofitted in JGroups 2.4 final to include solution Brian describes in last paragraph. Brian said on jbosscache-dev: We have a problem in that the FLUSH protocol makes the decision to shut off the ability to pass messages down the channel independently at each node. The protocol doesn't include anything at the JGroups level to readily support coordination between nodes as to when to shut off down messages. But, JBC needs coordination since it needs to make RPC calls around the cluster (e.g. commit()) as part of how it handles FLUSH. Basically, when the FLUSH protocol on a node receives a message telling it to START_FLUSH, it calls block() on the JBC instance. JBC does what it needs to do, then returns from block(). Following the return from block() the FLUSH protocol in that channel then begins blocking any further down() messages. Problem is as follows. 2 node REPL_SYNC cluster, A B where A is just starting up and thus initiates a FLUSH: 1) JBC on B has tx in progress, just starting the 2PC. Sends out the prepare(). 2) A sends out a START_FLUSH message. 3) A gets START_FLUSH, calls block() on JBC. 4) JBC on A is new, doesn't have much going on, very quickly returns from block(). A will no longer pass *down* any messages below FLUSH. 5) A gets the prepare() (no problem, FLUSH doesn't block up messages, just down messages.) 6) A executes the prepare(), but can't send the response to B because FLUSH is blocking the channel. 7) B gets the START_FLUSH, calls block() on JBC. 8) JBC B doesn't immediately return from block() as it is giving the prepare() some time to complete (avoid unnecessary tx rollback). But prepare() won't complete because A's channel is blocking the RPC response!! Eventually JBC B's block() impl will have to roll back the tx. Basically you have a race condition between calls to block() and prepare() calls, and can have different winners on different nodes. A solution we discussed, rejected and then came back to this evening (please read FLUSH.txt to understand the change we're discussing): Channel does not block down messages when block() returns. Rather it just sends out a FLUSH_OK message (see FLUSH.txt). It shouldn't initiate any new cluster activity (e.g. a prepare()) after sending FLUSH_OK, but it can respond to RPC calls. When it gets a FLUSH_OK from all the other members, it then blocks down messages and multicasts a FLUSH_COMPLETED to the cluster. View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3991384#... Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&a...