On 21 Oct 2010, at 22:54, Mircea Markus wrote: > Hi, > > ISPN-180 requires some significant code changes around re-hashing, so I've decided to make the rehashing-related test suite more stable before changing the code. > I've fixed some, but transactional rehash tests are still failing. Here is why: > > {A,B,C, D}, num owners= 2. A's state is {(k,v)} and a transactions is prepared with {(k, otherValue)} on A. > B crashes. > two things happen with A's state: > 1. on C an thread pulls state from A > 2. A sends transaction state to C on another thread > > Whit the right timing: > - 2 executes first (tx log is drained) > - tx commits - this applies the changes on C as well > - 1 executes last > Data ends up inconsistent: > A's states is {(k, otherValue)} while C's state is {(k,v)} > > State transfer and tx log needs to be sequential (not parallel). But it is sequential. http://fisheye.jboss.org/browse/Infinispan/branches/4.2.x/core/src/main/j...

Or is this not the case anymore, since Vladimir inverted the LeaveTask? Vladimir, can you confirm that these 2 actions (providing state and flushing/tx logs) happen in the same thread? > The easy way to achieve this is to only do push, i.e. A sends the state and then drains tx log. > This can be reproduce by running ConcurrentNonOverlappingLeaveTest.testTransactional (fails aprox 1/5) > > Cheers, > Mircea > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

> > The operations that need to be sequential are state transfer and tx log transfer. > State is being *pulled* by C (thread running on C) and tx log is *pushed* by A (thread running on A). Two threads running in different VMs. Ok, this would be the issue then - this must have changed when Vladimir inverted the leave task to be a pull for the main state. They both used to be pushes and sequential. Perhaps the sequentiality can be re-established by a lock on the receiver?

On 22 Oct 2010, at 15:17, Vladimir Blagojevic wrote: > >>> >>> The operations that need to be sequential are state transfer and tx log transfer. >>> State is being *pulled* by C (thread running on C) and tx log is *pushed* by A (thread running on A). Two threads running in different VMs. >> >> Ok, this would be the issue then - this must have changed when Vladimir inverted the leave task to be a pull for the main state. They both used to be pushes and sequential. >> >> Perhaps the sequentiality can be re-established by a lock on the receiver? > > Interested. What if C thread announce that it pulled state and upon reception of this message A drains the logs? Is there any work that can be done in parallel? State transfer and log draining need to happen in sequence. If not what's the point of using two threads?

One thread approach would be more maintainable and simple. I emphasise _maintainable_ because I've just debugged logs of state transfer and it's not the nicest thing to do. _______________________________________________ > > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 25 Oct 2010, at 11:52, Manik Surtani wrote: > > On 22 Oct 2010, at 17:01, Mircea Markus wrote: > >> >> On 22 Oct 2010, at 15:17, Vladimir Blagojevic wrote: >>> >>>>> >>>>> The operations that need to be sequential are state transfer and tx log transfer. >>>>> State is being *pulled* by C (thread running on C) and tx log is *pushed* by A (thread running on A). Two threads running in different VMs. >>>> >>>> Ok, this would be the issue then - this must have changed when Vladimir inverted the leave task to be a pull for the main state. They both used to be pushes and sequential. >>>> >>>> Perhaps the sequentiality can be re-established by a lock on the receiver? >>> >>> Interested. What if C thread announce that it pulled state and upon reception of this message A drains the logs? >> >> Is there any work that can be done in parallel? State transfer and log draining need to happen in sequence. >> If not what's the point of using two threads? > > They are not just 2 threads - they are on 2 separate nodes. State tfr is initiated by the receiver; tx log training by the sender. Why can't the sender initiate state transfer and then drain tx log? I.e. a thread on the sender write state and then write tx log. No sync needed then.

On 25 Oct 2010, at 12:00, Manik Surtani wrote: > > On 25 Oct 2010, at 11:56, Mircea Markus wrote: > >> >> On 25 Oct 2010, at 11:52, Manik Surtani wrote: >> >>> >>> On 22 Oct 2010, at 17:01, Mircea Markus wrote: >>> >>>> >>>> On 22 Oct 2010, at 15:17, Vladimir Blagojevic wrote: >>>>> >>>>>>> >>>>>>> The operations that need to be sequential are state transfer and tx log transfer. >>>>>>> State is being *pulled* by C (thread running on C) and tx log is *pushed* by A (thread running on A). Two threads running in different VMs. >>>>>> >>>>>> Ok, this would be the issue then - this must have changed when Vladimir inverted the leave task to be a pull for the main state. They both used to be pushes and sequential. >>>>>> >>>>>> Perhaps the sequentiality can be re-established by a lock on the receiver? >>>>> >>>>> Interested. What if C thread announce that it pulled state and upon reception of this message A drains the logs? >>>> >>>> Is there any work that can be done in parallel? State transfer and log draining need to happen in sequence. >>>> If not what's the point of using two threads? >>> >>> They are not just 2 threads - they are on 2 separate nodes. State tfr is initiated by the receiver; tx log training by the sender. >> Why can't the sender initiate state transfer and then drain tx log? I.e. a thread on the sender write state and then write tx log. No sync needed then. > > That was the original design. But that was even harder to debug/maintain since the Join and Leave processes were logically similar but implemented completely differently. > With the current approach the state transfer is the same for both Join and Leave. The tx log handling is the only bit that is different. Can't we factorize the code common Leave and Join code and still use a single thread to do all the action? There's no point in using two threads that run in sequence (unless they do some parallel processing?).

I've created https://jira.jboss.org/browse/ISPN-731 Vladimir, mind taking a look into this one as you're familiar with the code re:state transfer? >> Can't we factorize the code common Leave and Join code and still use a single thread to do all the action? There's no point in using two threads that run in sequence (unless they do some parallel processing?). > > Umm, nope. :) One is a push and one is a pull. Very different. Can't we just do push and no push(xor pull)? I guess my basic question is: what do we get by doing both push and pull at the same time? Any operation happening in parallel which would make things faster?

>> > > Mircea, Have a look at the old LeaveTask.java. Pushing state on leave is complicated! Hence InvertedLeaveTask.java which pulls state just like join task. We share a lot of code with JoinTask, it is easier to maintain, comprehend etc etc. I see. So the only reason to have both push and pull is code reuse? > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

They are not just 2 threads - they are on 2 separate nodes. State tfr is initiated by the receiver; tx log training by the sender. Hence the different threads. Modifying the protocol to allow for a state tfr ack message, as Vlad suggested, would work fine.