[infinispan-dev] rehashing hazard

[Mircea Markus]

Hi,

ISPN-180 requires some significant code changes around re-hashing, so I've decided to make the rehashing-related test suite more stable before changing the code.
I've fixed some, but transactional rehash tests are still failing. Here is why:

{A,B,C, D}, num owners= 2. A's state is {(k,v)} and a transactions is prepared with {(k, otherValue)} on A.
B crashes.
two things happen with A's state: 
1. on C an thread pulls state from A
2. A sends transaction state to C on another thread

Whit the right timing:
- 2 executes first (tx log is drained)
- tx commits - this applies the changes on C as well 
- 1 executes last 
Data ends up inconsistent:
A's states is {(k, otherValue)} while C's state is  {(k,v)} 

State transfer and tx log needs to be sequential (not parallel). The easy way to achieve this is  to only do push, i.e. A sends the state and then drains tx log.
This can be reproduce by running ConcurrentNonOverlappingLeaveTest.testTransactional (fails aprox 1/5)

Cheers,
Mircea