Re: [jbosscache-dev] Re: Non Blocking State Transfer Status (& Integration with JGroups)
Bela Ban wrote:
Jason T. Greene wrote:
> Hi All,
>
> I wanted to summarize my initial research into NBST. The planned
> design (outlined in the wiki:
> http://www.jboss.org/community/docs/DOC-10275) only needs to block
> transactional activity once, at the end of the process when sending
> the tx log. Unfortunately it appears that flush and partial flush can
> not be used for this, since the application needs the ability to send
> state (tx log) during the flush.
This is possible because FLUSH blocks only *multicast* messages, not
*unicast* messages. This is guaranteed because virtual synchrony applies
only to group messages.
Ah, ok then I can use flush for the final block.
> I.e. transactions need to be paused by only 2 nodes, while the
> transfer state.
What happens when A and D pause transactions and C wants to commit TX-5
? Would, assuming the pause takes longer than the commit, TX-5 get
rolled back ? If that's the case then I think a regular flush would not
hurt.
It depends on whether C is async or not, and if A and D care about the
TX C is committing (e.g. to a partition A and D are not a participate
in). We don't really want to block C unless C somehow relies on A or D.
> This however is not a big deal because we can just do this in JBoss
> Cache using a normal RPC message that flips a gate.
>
> In addition, the state transfer and streaming state transfer
> facilities in jgroups can not be used (since they are designed around
> blocking the entire group).
No, both state transfer protocols are completely independent from FLUSH:
they work with or without FLUSH being present.
I could have phrased that better. The problem is that NBST requires a
dialog beyond a single getState/setState. The receiver needs to find a
sender that is available, and it needs to receive main state, then N tx
log transfers, then coordinate a flush, then another tx log transfer.
> This means JBoss Cache needs to stream state itself. Ideally this
> would be a separate point-to-point connection, since we don't want to
> pollute multicast traffic with potentially huge volumes of noise.
> Currently jgroups does not yet support a streaming API like this:
> https://jira.jboss.org/jira/browse/JGRP-653
>
> IMO This leaves us with 3 options:
>
> 1. Wait on JGRP-653 (upping its priority), also add requirements for a
> p2p connection.
On JGRP-653, I argued against JGRP-653 :-) I don't think it is
necessary, because this can be done by application code, e.g. similar to
what Brian has implemented with farming: the application chunks up data
from an input stream and sends the chunks (governed by flow control).
The receiving side reads the chunks and feeds them to the recipient
(which e.g. reads them from an input stream).
The advantage of supporting this is that you can optimize a number of
aspects including the transport, how much of the protocol stack is used,
etc when a stream is used. Basically, when someone uses this, they
really want a tcp connection.
> 2. Implement our own p2p connection using tcp (probably using xnio)
Yuck !
> 3. Somehow enhance state transfer / partial flush to meet our needs
Which enhancements do you have in mind ? FLUSH between 2 participant
*is* implemented unless I'm completely mistaken, Vladimir, can you
comment on this ? Not sure though it is needed in the first place...
Yes partial flush is implemented, and since it doesn't block unicast, it
seems exactly what we need. I am not sure the remaining state sending
aspects make sense though.
--
Jason T. Greene
JBoss, a division of Red Hat