[jbosscache-dev] Re: Non Blocking State Transfer Status (& Integration with JGroups)

Jason T. Greene wrote: > Brian Stansberry wrote: >> Jason T. Greene wrote: >>> Brian Stansberry wrote: >>>> Jason T. Greene wrote: >>>>> Hi All, >>>>> >>>>> I wanted to summarize my initial research into NBST. The planned >>>>> design (outlined in the wiki: >>>>> http://www.jboss.org/community/docs/DOC-10275) only needs to block >>>>> transactional activity once, at the end of the process when >>>>> sending the tx log. Unfortunately it appears that flush and >>>>> partial flush can not be used for this, since the application >>>>> needs the ability to send state (tx log) during the flush. I.e. >>>>> transactions need to be paused by only 2 nodes, while the transfer >>>>> state. This however is not a big deal because we can just do this >>>>> in JBoss Cache using a normal RPC message that flips a gate. >>>>> >>>>> In addition, the state transfer and streaming state transfer >>>>> facilities in jgroups can not be used (since they are designed >>>>> around blocking the entire group). This means JBoss Cache needs to >>>>> stream state itself. Ideally this would be a separate >>>>> point-to-point connection, since we don't want to pollute >>>>> multicast traffic with potentially huge volumes of noise. >>>>> Currently jgroups does not yet support a streaming API like this: >>>>> https://jira.jboss.org/jira/browse/JGRP-653 >>>>> >>>>> IMO This leaves us with 3 options: >>>>> >>>>> 1. Wait on JGRP-653 (upping its priority), also add requirements >>>>> for a p2p connection. >>>>> 2. Implement our own p2p connection using tcp (probably using xnio). >>>>> 3. Somehow enhance state transfer / partial flush to meet our needs >>>>> >>>>> Option 1 seems to be a useful feature for other applications. >>>>> Although we need feedback from Bela and Vladimir about that. >>>>> >>>>> Option 2 would give us more flexibility in the implementation, >>>>> however care has to be taken to ensure that communication can only >>>>> happen between group members (for security reasons), and that the >>>>> network address configurations are somehow reused. >>>>> >>>>> Option 3 I am less found of, since we would likely end up adding a >>>>> bunch of JBoss Cache specific code to JGroups that no one else >>>>> would use. >>>>> >>>> >>>> Option 2 makes me nervous. Two separate communication frameworks, >>>> added dependencies, opening new sockets etc. Sounds like >>>> integration hassles for sure. >>>> >>> >>> Yes there are definitely integration hassles that make this option >>> less desirable than the first. >>> >>> From a dependency perspective, we are already using non-jgroups p2p >>> with TCPCacheServer (currently Java sockets based), although I >>> believe Manik was evaluating xnio for it since it would simplify >>> development. While it is an added dep for JBC, it will eventually be >>> part of AS, since Remoting 3 depends on it. >> >> Yeah, that's part of the concern. Two otherwise independent projects >> using an underlying library for a critical function and both have to >> play nice in the AS. In my self-centered viewpoint TcpCacheServer >> isn't a critical function since I don't use it. ;) (Mostly kidding >> here; I recognize that AS users may use it so it would need to work >> in the AS.) >> >> BTW, is a *JGroups* streaming API necessary here? The old AS Farm >> service passed arbitrary sized files by sending byte[] chunks via >> RpcDispatcher calls. Worked fine. That's not quite what JBC would >> need, since FarmService read a chunk from a FileInputStream and >> passed it to JGroups; you'd want an OutputStream impl that would pass >> a chunk to JGroups when it's internal buffer reached size X. > > Their could be an abstraction in JBC that does this, however we are > still missing the piece that says this message needs to go across a > separate p2p tcp connection. Otherwise state transfer traffic will > compete with live standard operations. This problem is amplified when > we have multiple state transfers going on. > > I suppose we could dynamically create a new JGroups channel for this, > but it seems like a lot of overhead for a single p2p connection. > Thinking again, building this over JGroups would probably better with two static channels. One would be TCP + MPING, the other whatever the user defines.