[jbosscache-dev] Re: Non Blocking State Transfer Status (& Integration with JGroups)

[Jason T. Greene]

Brian Stansberry wrote:
> Jason T. Greene wrote:
>> Hi All,
>>
>> I wanted to summarize my initial research into NBST. The planned 
>> design (outlined in the wiki: 
>> http://www.jboss.org/community/docs/DOC-10275) only needs to block 
>> transactional activity once, at the end of the process when sending 
>> the tx log. Unfortunately it appears that flush and partial flush can 
>> not be used for this, since the application needs the ability to send 
>> state (tx log) during the flush. I.e. transactions need to be paused 
>> by only 2 nodes, while the transfer state. This however is not a big 
>> deal because we can just do this in JBoss Cache using a normal RPC 
>> message that flips a gate.
>>
>> In addition, the state transfer and streaming state transfer 
>> facilities in jgroups can not be used (since they are designed around 
>> blocking the entire group). This means JBoss Cache needs to stream 
>> state itself. Ideally this would be a separate point-to-point 
>> connection, since we don't want to pollute multicast traffic with 
>> potentially huge volumes of noise. Currently jgroups does not yet 
>> support a streaming API like this:
>> https://jira.jboss.org/jira/browse/JGRP-653
>>
>> IMO This leaves us with 3 options:
>>
>> 1. Wait on JGRP-653 (upping its priority), also add requirements for a 
>> p2p connection.
>> 2. Implement our own p2p connection using tcp (probably using xnio).
>> 3. Somehow enhance state transfer / partial flush to meet our needs
>>
>> Option 1 seems to be a useful feature for other applications. Although 
>> we need feedback from Bela and Vladimir about that.
>>
>> Option 2 would give us more flexibility in the implementation, however 
>> care has to be taken to ensure that communication can only happen 
>> between group members (for security reasons), and that the network 
>> address configurations are somehow reused.
>>
>> Option 3 I am less found of, since we would likely end up adding a 
>> bunch of JBoss Cache specific code to JGroups that no one else would use.
>>
> 
> Option 2 makes me nervous. Two separate communication frameworks, added 
> dependencies, opening new sockets etc. Sounds like integration hassles 
> for sure.
> 

Yes there are definitely integration hassles that make this option less 
desirable than the first.

 From a dependency perspective, we are already using non-jgroups p2p 
with TCPCacheServer (currently Java sockets based), although I believe 
Manik was evaluating xnio for it since it would simplify development. 
While it is an added dep for JBC, it will eventually be part of AS, 
since Remoting 3 depends on it.
-- 
Jason T. Greene
JBoss, a division of Red Hat