[jbosscache-dev] Re: Non Blocking State Transfer Status (& Integration with JGroups)

[Bela Ban]

Jason T. Greene wrote:
> Hi All,
>
> I wanted to summarize my initial research into NBST. The planned 
> design (outlined in the wiki: 
> http://www.jboss.org/community/docs/DOC-10275) only needs to block 
> transactional activity once, at the end of the process when sending 
> the tx log. Unfortunately it appears that flush and partial flush can 
> not be used for this, since the application needs the ability to send 
> state (tx log) during the flush.

This is possible because FLUSH blocks only *multicast* messages, not 
*unicast* messages. This is guaranteed because virtual synchrony applies 
only to group messages.


> I.e. transactions need to be paused by only 2 nodes, while the 
> transfer state.

What happens when A and D pause transactions and C wants to commit TX-5 
? Would, assuming the pause takes longer than the commit, TX-5 get 
rolled back ? If that's the case then I think a regular flush would not 
hurt.


> This however is not a big deal because we can just do this in JBoss 
> Cache using a normal RPC message that flips a gate.
>
> In addition, the state transfer and streaming state transfer 
> facilities in jgroups can not be used (since they are designed around 
> blocking the entire group).

No, both state transfer protocols are completely independent from FLUSH: 
they work with or without FLUSH being present.

> This means JBoss Cache needs to stream state itself. Ideally this 
> would be a separate point-to-point connection, since we don't want to 
> pollute multicast traffic with potentially huge volumes of noise. 
> Currently jgroups does not yet support a streaming API like this:
> https://jira.jboss.org/jira/browse/JGRP-653
>
> IMO This leaves us with 3 options:
>
> 1. Wait on JGRP-653 (upping its priority), also add requirements for a 
> p2p connection.

On JGRP-653, I argued against JGRP-653 :-) I don't think it is 
necessary, because this can be done by application code, e.g. similar to 
what Brian has implemented with farming: the application chunks up data 
from an input stream and sends the chunks (governed by flow control). 
The receiving side reads the chunks and feeds them to the recipient 
(which e.g. reads them from an input stream).


> 2. Implement our own p2p connection using tcp (probably using xnio)

Yuck !

> 3. Somehow enhance state transfer / partial flush to meet our needs

Which enhancements do you have in mind ? FLUSH between 2 participant 
*is* implemented unless I'm completely mistaken, Vladimir, can you 
comment on this ? Not sure though it is needed in the first place...

> Option 1 seems to be a useful feature for other applications. Although 
> we need feedback from Bela and Vladimir about that

See my comment above

> Option 2 would give us more flexibility in the implementation, however 
> care has to be taken to ensure that communication can only happen 
> between group members (for security reasons), and that the network 
> address configurations are somehow reused.
>
> Option 3 I am less found of, since we would likely end up adding a 
> bunch of JBoss Cache specific code to JGroups that no one else would use.

Agreed. But if it *is* generic, then we can take a look

-- 
Bela Ban
Lead JGroups / Clustering Team
JBoss - a division of Red Hat