[jbosscache-dev] Re: Non Blocking State Transfer Status (& Integration with JGroups)

[Jason T. Greene]

Bela Ban wrote:
> 
> 
> Jason T. Greene wrote:
>> Hi All,
>>
>> I wanted to summarize my initial research into NBST. The planned 
>> design (outlined in the wiki: 
>> http://www.jboss.org/community/docs/DOC-10275) only needs to block 
>> transactional activity once, at the end of the process when sending 
>> the tx log. Unfortunately it appears that flush and partial flush can 
>> not be used for this, since the application needs the ability to send 
>> state (tx log) during the flush.
> 
> This is possible because FLUSH blocks only *multicast* messages, not 
> *unicast* messages. This is guaranteed because virtual synchrony applies 
> only to group messages.

Ah, ok then I can use flush for the final block.

> 
>> I.e. transactions need to be paused by only 2 nodes, while the 
>> transfer state.
> 
> What happens when A and D pause transactions and C wants to commit TX-5 
> ? Would, assuming the pause takes longer than the commit, TX-5 get 
> rolled back ? If that's the case then I think a regular flush would not 
> hurt.

It depends on whether C is async or not, and if A and D care about the 
TX C is committing (e.g. to a partition A and D are not a participate 
in). We don't really want to block C unless C somehow relies on A or D.

> 
>> This however is not a big deal because we can just do this in JBoss 
>> Cache using a normal RPC message that flips a gate.
>>
>> In addition, the state transfer and streaming state transfer 
>> facilities in jgroups can not be used (since they are designed around 
>> blocking the entire group).
> 
> No, both state transfer protocols are completely independent from FLUSH: 
> they work with or without FLUSH being present.

I could have phrased that better. The problem is that NBST requires a 
dialog beyond a single getState/setState. The receiver needs to find a 
sender that is available, and it needs to receive main state, then N tx 
log transfers, then coordinate a flush, then another tx log transfer.

>> This means JBoss Cache needs to stream state itself. Ideally this 
>> would be a separate point-to-point connection, since we don't want to 
>> pollute multicast traffic with potentially huge volumes of noise. 
>> Currently jgroups does not yet support a streaming API like this:
>> https://jira.jboss.org/jira/browse/JGRP-653
>>
>> IMO This leaves us with 3 options:
>>
>> 1. Wait on JGRP-653 (upping its priority), also add requirements for a 
>> p2p connection.
> 
> On JGRP-653, I argued against JGRP-653 :-) I don't think it is 
> necessary, because this can be done by application code, e.g. similar to 
> what Brian has implemented with farming: the application chunks up data 
> from an input stream and sends the chunks (governed by flow control). 
> The receiving side reads the chunks and feeds them to the recipient 
> (which e.g. reads them from an input stream).
>

The advantage of supporting this is that you can optimize a number of 
aspects including the transport, how much of the protocol stack is used, 
etc when a stream is used. Basically, when someone uses this, they 
really want a tcp connection.

> 
>> 2. Implement our own p2p connection using tcp (probably using xnio)
> 
> Yuck !
> 
>> 3. Somehow enhance state transfer / partial flush to meet our needs
> 
> Which enhancements do you have in mind ? FLUSH between 2 participant 
> *is* implemented unless I'm completely mistaken, Vladimir, can you 
> comment on this ? Not sure though it is needed in the first place...

Yes partial flush is implemented, and since it doesn't block unicast, it 
seems exactly what we need. I am not sure the remaining state sending 
aspects make sense though.

-- 
Jason T. Greene
JBoss, a division of Red Hat