Manik Surtani wrote:

Hi guys

I might have found a potential flaw in the NBST impl that we have.

The way it works, transient state is written to the stream, and when this is done, the tx log is flushed.  When the tx log is small, a FLUSH is requested and then the remaining tx log is written.

But the flush isn't lifted here - the flush is only lifted once the state transfer process is complete, and in between these 2 events is the generation, streaming and storing of persistent state.  Which is potentially much bigger than transient state, and could mean that the flush is in place for much longer than expected.  Any thoughts here, Jason?  Perhaps a solution would be to stream persistent state before the tx log is flushed?  Could lead to wacky stream multiplexing on the receiver's end though.

That is how it is already implemented:

transient state
persistent state
tx log
 - log write
 - flush
 - log write
 - end flush