[jbosscache-dev] Re: Non Blocking State Transfer Status (& Integration with JGroups)

[Jason T. Greene]

Jason T. Greene wrote:
> Brian Stansberry wrote:
>> Jason T. Greene wrote:
>>> Brian Stansberry wrote:
>>>> Jason T. Greene wrote:
>>>>> Hi All,
>>>>>
>>>>> I wanted to summarize my initial research into NBST. The planned 
>>>>> design (outlined in the wiki: 
>>>>> http://www.jboss.org/community/docs/DOC-10275) only needs to block 
>>>>> transactional activity once, at the end of the process when sending 
>>>>> the tx log. Unfortunately it appears that flush and partial flush 
>>>>> can not be used for this, since the application needs the ability 
>>>>> to send state (tx log) during the flush. I.e. transactions need to 
>>>>> be paused by only 2 nodes, while the transfer state. This however 
>>>>> is not a big deal because we can just do this in JBoss Cache using 
>>>>> a normal RPC message that flips a gate.
>>>>>
>>>>> In addition, the state transfer and streaming state transfer 
>>>>> facilities in jgroups can not be used (since they are designed 
>>>>> around blocking the entire group). This means JBoss Cache needs to 
>>>>> stream state itself. Ideally this would be a separate 
>>>>> point-to-point connection, since we don't want to pollute multicast 
>>>>> traffic with potentially huge volumes of noise. Currently jgroups 
>>>>> does not yet support a streaming API like this:
>>>>> https://jira.jboss.org/jira/browse/JGRP-653
>>>>>
>>>>> IMO This leaves us with 3 options:
>>>>>
>>>>> 1. Wait on JGRP-653 (upping its priority), also add requirements 
>>>>> for a p2p connection.
>>>>> 2. Implement our own p2p connection using tcp (probably using xnio).
>>>>> 3. Somehow enhance state transfer / partial flush to meet our needs
>>>>>
>>>>> Option 1 seems to be a useful feature for other applications. 
>>>>> Although we need feedback from Bela and Vladimir about that.
>>>>>
>>>>> Option 2 would give us more flexibility in the implementation, 
>>>>> however care has to be taken to ensure that communication can only 
>>>>> happen between group members (for security reasons), and that the 
>>>>> network address configurations are somehow reused.
>>>>>
>>>>> Option 3 I am less found of, since we would likely end up adding a 
>>>>> bunch of JBoss Cache specific code to JGroups that no one else 
>>>>> would use.
>>>>>
>>>>
>>>> Option 2 makes me nervous. Two separate communication frameworks, 
>>>> added dependencies, opening new sockets etc. Sounds like integration 
>>>> hassles for sure.
>>>>
>>>
>>> Yes there are definitely integration hassles that make this option 
>>> less desirable than the first.
>>>
>>>  From a dependency perspective, we are already using non-jgroups p2p 
>>> with TCPCacheServer (currently Java sockets based), although I 
>>> believe Manik was evaluating xnio for it since it would simplify 
>>> development. While it is an added dep for JBC, it will eventually be 
>>> part of AS, since Remoting 3 depends on it.
>>
>> Yeah, that's part of the concern. Two otherwise independent projects 
>> using an underlying library for a critical function and both have to 
>> play nice in the AS.  In my self-centered viewpoint TcpCacheServer 
>> isn't a critical function since I don't use it. ;) (Mostly kidding 
>> here; I recognize that AS users may use it so it would need to work in 
>> the AS.)
>>
>> BTW, is a *JGroups* streaming API necessary here?  The old AS Farm 
>> service passed arbitrary sized files by sending byte[] chunks via 
>> RpcDispatcher calls. Worked fine. That's not quite what JBC would 
>> need, since FarmService read a chunk from a FileInputStream and passed 
>> it to JGroups; you'd want an OutputStream impl that would pass a chunk 
>> to JGroups when it's internal buffer reached size X.
> 
> Their could be an abstraction in JBC that does this, however we are 
> still missing the piece that says this message needs to go across a 
> separate p2p tcp connection. Otherwise state transfer traffic will 
> compete with live standard operations. This problem is amplified when we 
> have multiple state transfers going on.
> 
> I suppose we could dynamically create a new JGroups channel for this, 
> but it seems like a lot of overhead for a single p2p connection.
> 

Thinking again, building this over JGroups would probably better with 
two static channels. One would be TCP + MPING, the other whatever the 
user defines.

-- 
Jason T. Greene
JBoss, a division of Red Hat