I see this in the code base:

which has come in as a part of this commit:

This is no good, since throwing an IllegalStateException *after* a transaction has prepared and is attempting to commit will leave the transaction in an inconsistent state.  This is why I see heuristic exceptions when trying to stress a lot of transactions while having nodes join/leave.

A few points:

1) Mircea, are you aware of this, and is this a part of the transaction refactoring you are working on?

2) If not, it probably should be.  ;)

3) Vladimir, why is this check necessary in the first place?  If prepares have happened on nodes A and B, and the commit should now go to nodes B and C due to a rehash, can't we send the commit to a union of A, B and C, and let the nodes that don't care about the tx anymore just ignore the commit?

Cheers

Manik