Hi All, I wanted to summarize my initial research into NBST. The planned design (outlined in the wiki: http://www.jboss.org/community/docs/DOC-10275) only needs to block transactional activity once, at the end of the process when sending the tx log. Unfortunately it appears that flush and partial flush can not be used for this, since the application needs the ability to send state (tx log) during the flush. I.e. transactions need to be paused by only 2 nodes, while the transfer state. This however is not a big deal because we can just do this in JBoss Cache using a normal RPC message that flips a gate. In addition, the state transfer and streaming state transfer facilities in jgroups can not be used (since they are designed around blocking the entire group). This means JBoss Cache needs to stream state itself. Ideally this would be a separate point-to-point connection, since we don't want to pollute multicast traffic with potentially huge volumes of noise. Currently jgroups does not yet support a streaming API like this: https://jira.jboss.org/jira/browse/JGRP-653 IMO This leaves us with 3 options: 1. Wait on JGRP-653 (upping its priority), also add requirements for a p2p connection. 2. Implement our own p2p connection using tcp (probably using xnio). 3. Somehow enhance state transfer / partial flush to meet our needs Option 1 seems to be a useful feature for other applications. Although we need feedback from Bela and Vladimir about that. Option 2 would give us more flexibility in the implementation, however care has to be taken to ensure that communication can only happen between group members (for security reasons), and that the network address configurations are somehow reused. Option 3 I am less found of, since we would likely end up adding a bunch of JBoss Cache specific code to JGroups that no one else would use.

Brian Stansberry wrote: > Jason T. Greene wrote: >> Hi All, >> >> I wanted to summarize my initial research into NBST. The planned >> design (outlined in the wiki: >> http://www.jboss.org/community/docs/DOC-10275) only needs to block >> transactional activity once, at the end of the process when sending >> the tx log. Unfortunately it appears that flush and partial flush can >> not be used for this, since the application needs the ability to send >> state (tx log) during the flush. I.e. transactions need to be paused >> by only 2 nodes, while the transfer state. This however is not a big >> deal because we can just do this in JBoss Cache using a normal RPC >> message that flips a gate. >> >> In addition, the state transfer and streaming state transfer >> facilities in jgroups can not be used (since they are designed around >> blocking the entire group). This means JBoss Cache needs to stream >> state itself. Ideally this would be a separate point-to-point >> connection, since we don't want to pollute multicast traffic with >> potentially huge volumes of noise. Currently jgroups does not yet >> support a streaming API like this: >> https://jira.jboss.org/jira/browse/JGRP-653 >> >> IMO This leaves us with 3 options: >> >> 1. Wait on JGRP-653 (upping its priority), also add requirements for >> a p2p connection. >> 2. Implement our own p2p connection using tcp (probably using xnio). >> 3. Somehow enhance state transfer / partial flush to meet our needs >> >> Option 1 seems to be a useful feature for other applications. >> Although we need feedback from Bela and Vladimir about that. >> >> Option 2 would give us more flexibility in the implementation, >> however care has to be taken to ensure that communication can only >> happen between group members (for security reasons), and that the >> network address configurations are somehow reused. >> >> Option 3 I am less found of, since we would likely end up adding a >> bunch of JBoss Cache specific code to JGroups that no one else would >> use. >> > > Option 2 makes me nervous. Two separate communication frameworks, > added dependencies, opening new sockets etc. Sounds like integration > hassles for sure. > Yes there are definitely integration hassles that make this option less desirable than the first. From a dependency perspective, we are already using non-jgroups p2p with TCPCacheServer (currently Java sockets based), although I believe Manik was evaluating xnio for it since it would simplify development. While it is an added dep for JBC, it will eventually be part of AS, since Remoting 3 depends on it.

Brian Stansberry wrote: > Jason T. Greene wrote: >> Brian Stansberry wrote: >>> Jason T. Greene wrote: >>>> Hi All, >>>> >>>> I wanted to summarize my initial research into NBST. The planned >>>> design (outlined in the wiki: >>>> http://www.jboss.org/community/docs/DOC-10275) only needs to block >>>> transactional activity once, at the end of the process when sending >>>> the tx log. Unfortunately it appears that flush and partial flush >>>> can not be used for this, since the application needs the ability >>>> to send state (tx log) during the flush. I.e. transactions need to >>>> be paused by only 2 nodes, while the transfer state. This however >>>> is not a big deal because we can just do this in JBoss Cache using >>>> a normal RPC message that flips a gate. >>>> >>>> In addition, the state transfer and streaming state transfer >>>> facilities in jgroups can not be used (since they are designed >>>> around blocking the entire group). This means JBoss Cache needs to >>>> stream state itself. Ideally this would be a separate >>>> point-to-point connection, since we don't want to pollute multicast >>>> traffic with potentially huge volumes of noise. Currently jgroups >>>> does not yet support a streaming API like this: >>>> https://jira.jboss.org/jira/browse/JGRP-653 >>>> >>>> IMO This leaves us with 3 options: >>>> >>>> 1. Wait on JGRP-653 (upping its priority), also add requirements >>>> for a p2p connection. >>>> 2. Implement our own p2p connection using tcp (probably using xnio). >>>> 3. Somehow enhance state transfer / partial flush to meet our needs >>>> >>>> Option 1 seems to be a useful feature for other applications. >>>> Although we need feedback from Bela and Vladimir about that. >>>> >>>> Option 2 would give us more flexibility in the implementation, >>>> however care has to be taken to ensure that communication can only >>>> happen between group members (for security reasons), and that the >>>> network address configurations are somehow reused. >>>> >>>> Option 3 I am less found of, since we would likely end up adding a >>>> bunch of JBoss Cache specific code to JGroups that no one else >>>> would use. >>>> >>> >>> Option 2 makes me nervous. Two separate communication frameworks, >>> added dependencies, opening new sockets etc. Sounds like integration >>> hassles for sure. >>> >> >> Yes there are definitely integration hassles that make this option >> less desirable than the first. >> >> From a dependency perspective, we are already using non-jgroups p2p >> with TCPCacheServer (currently Java sockets based), although I >> believe Manik was evaluating xnio for it since it would simplify >> development. While it is an added dep for JBC, it will eventually be >> part of AS, since Remoting 3 depends on it. > > Yeah, that's part of the concern. Two otherwise independent projects > using an underlying library for a critical function and both have to > play nice in the AS. In my self-centered viewpoint TcpCacheServer > isn't a critical function since I don't use it. ;) (Mostly kidding > here; I recognize that AS users may use it so it would need to work in > the AS.) > > BTW, is a *JGroups* streaming API necessary here? The old AS Farm > service passed arbitrary sized files by sending byte[] chunks via > RpcDispatcher calls. Worked fine. That's not quite what JBC would > need, since FarmService read a chunk from a FileInputStream and passed > it to JGroups; you'd want an OutputStream impl that would pass a chunk > to JGroups when it's internal buffer reached size X. Their could be an abstraction in JBC that does this, however we are still missing the piece that says this message needs to go across a separate p2p tcp connection. Otherwise state transfer traffic will compete with live standard operations. This problem is amplified when we have multiple state transfers going on. I suppose we could dynamically create a new JGroups channel for this, but it seems like a lot of overhead for a single p2p connection.

Jason T. Greene wrote: > Jason T. Greene wrote: >> Brian Stansberry wrote: >>> Jason T. Greene wrote: >>>> Brian Stansberry wrote: >>>>> Jason T. Greene wrote: >>>>>> Hi All, >>>>>> >>>>>> I wanted to summarize my initial research into NBST. The planned >>>>>> design (outlined in the wiki: >>>>>> http://www.jboss.org/community/docs/DOC-10275) only needs to >>>>>> block transactional activity once, at the end of the process when >>>>>> sending the tx log. Unfortunately it appears that flush and >>>>>> partial flush can not be used for this, since the application >>>>>> needs the ability to send state (tx log) during the flush. I.e. >>>>>> transactions need to be paused by only 2 nodes, while the >>>>>> transfer state. This however is not a big deal because we can >>>>>> just do this in JBoss Cache using a normal RPC message that flips >>>>>> a gate. >>>>>> >>>>>> In addition, the state transfer and streaming state transfer >>>>>> facilities in jgroups can not be used (since they are designed >>>>>> around blocking the entire group). This means JBoss Cache needs >>>>>> to stream state itself. Ideally this would be a separate >>>>>> point-to-point connection, since we don't want to pollute >>>>>> multicast traffic with potentially huge volumes of noise. >>>>>> Currently jgroups does not yet support a streaming API like this: >>>>>> https://jira.jboss.org/jira/browse/JGRP-653 >>>>>> >>>>>> IMO This leaves us with 3 options: >>>>>> >>>>>> 1. Wait on JGRP-653 (upping its priority), also add requirements >>>>>> for a p2p connection. >>>>>> 2. Implement our own p2p connection using tcp (probably using xnio). >>>>>> 3. Somehow enhance state transfer / partial flush to meet our needs >>>>>> >>>>>> Option 1 seems to be a useful feature for other applications. >>>>>> Although we need feedback from Bela and Vladimir about that. >>>>>> >>>>>> Option 2 would give us more flexibility in the implementation, >>>>>> however care has to be taken to ensure that communication can >>>>>> only happen between group members (for security reasons), and >>>>>> that the network address configurations are somehow reused. >>>>>> >>>>>> Option 3 I am less found of, since we would likely end up adding >>>>>> a bunch of JBoss Cache specific code to JGroups that no one else >>>>>> would use. >>>>>> >>>>> >>>>> Option 2 makes me nervous. Two separate communication frameworks, >>>>> added dependencies, opening new sockets etc. Sounds like >>>>> integration hassles for sure. >>>>> >>>> >>>> Yes there are definitely integration hassles that make this option >>>> less desirable than the first. >>>> >>>> From a dependency perspective, we are already using non-jgroups >>>> p2p with TCPCacheServer (currently Java sockets based), although I >>>> believe Manik was evaluating xnio for it since it would simplify >>>> development. While it is an added dep for JBC, it will eventually >>>> be part of AS, since Remoting 3 depends on it. >>> >>> Yeah, that's part of the concern. Two otherwise independent projects >>> using an underlying library for a critical function and both have to >>> play nice in the AS. In my self-centered viewpoint TcpCacheServer >>> isn't a critical function since I don't use it. ;) (Mostly kidding >>> here; I recognize that AS users may use it so it would need to work >>> in the AS.) >>> >>> BTW, is a *JGroups* streaming API necessary here? The old AS Farm >>> service passed arbitrary sized files by sending byte[] chunks via >>> RpcDispatcher calls. Worked fine. That's not quite what JBC would >>> need, since FarmService read a chunk from a FileInputStream and >>> passed it to JGroups; you'd want an OutputStream impl that would >>> pass a chunk to JGroups when it's internal buffer reached size X. >> >> Their could be an abstraction in JBC that does this, however we are >> still missing the piece that says this message needs to go across a >> separate p2p tcp connection. Otherwise state transfer traffic will >> compete with live standard operations. This problem is amplified when >> we have multiple state transfers going on. >> >> I suppose we could dynamically create a new JGroups channel for this, >> but it seems like a lot of overhead for a single p2p connection. >> > > Thinking again, building this over JGroups would probably better with > two static channels. One would be TCP + MPING, the other whatever the > user defines. > I don't understand why two channels are needed. You can easily unicast over the regular JGroups channel. Is your concern that the state transfer messages between the sender and the recipient will get held up by FC or UNICAST or something due to the regular traffic? I don't see why that would be a big problem. The recipient until it completes the state transfer isn't going to be doing anything (e.g. holding locks) that will cause significant delays in processing state transfer messages. (Also, I may be wrong but I believe the UNICAST and NAKACK locks are separate, so a thread carrying a unicast state transfer message from node A will not block on recipient B because a multicast replication message from A is being handled by another thread.)

Jason T. Greene wrote: > >> Also, I may be wrong but I believe the UNICAST and NAKACK locks are >> separate, so a thread carrying a unicast state transfer message from >> node A will not block on recipient B because a multicast replication >> message from A is being handled by another thread.) >> > > I am not too worried about the locking, although that is a factor, but > more about a multicast configuration, where you have state transfer > traffic for N transfers hitting the multicast address. Even if > everyone discards the traffic, it still has to be processed in java > land, and it will be alot of traffic. Why the multicast address ? State transfer should be between 2 nodes, a provider and a requester, so it should always be unicast. If we bring partial flush into the mix, where we block only the 2 participants in the state transfer, then we should be OK. Not sure if we should do a partial or a full flush though...

Jason T. Greene wrote: <snip/> > Although IMO a bare tcp connection would still be the most optimal > form of transferring large state to a single node since the kernel and > the hardware is doing all the work. That can be saved for a future > optimization though ;) > Semi-OT: The week before the holidays I largely spent helping with some security certifications for the AS. Had to explain every clustering-related port the AS used, how it was bound, secured, could be controlled by an admin etc. Now I'm updating the AS Clustering Guide, trying to write user-understandable docs about how clustering works. A big conceptual barrier is always the multiple JGroups channels. The shared transport helps some with this (although AS 5 is still not optimal in this regard, so it uses 2 separate transports). What I'm driving at is both of these efforts leave me with a queasy feeling about adding new and different ways of opening sockets and communicating around a cluster. OK if needed as an advanced optimization for the real hard core cases, but IMHO there's a lot of value in keeping it simple by default.

> Why the multicast address ? State transfer should be between 2 nodes, > a provider and a requester, so it should always be unicast. If we > bring partial flush into the mix, where we block only the 2 > participants in the state transfer, then we should be OK. Not sure if > we should do a partial or a full flush though... Ah! There is my disconnect. I somehow had it in my head that the jgroups UDP transport used a multicast socket for everything, and that UNICAST packets where just emulated on top of that. I should have looked at the code. That pretty much eliminates my concern. Although IMO a bare tcp connection would still be the most optimal form of transferring large state to a single node since the kernel and the hardware is doing all the work. That can be saved for a future optimization though ;)

> BTW, is a *JGroups* streaming API necessary here? The old AS Farm > service passed arbitrary sized files by sending byte[] chunks via > RpcDispatcher calls. Worked fine. That's not quite what JBC would > need, since FarmService read a chunk from a FileInputStream and > passed it to JGroups; you'd want an OutputStream impl that would pass > a chunk to JGroups when it's internal buffer reached size X. Their could be an abstraction in JBC that does this, however we are still missing the piece that says this message needs to go across a separate p2p tcp connection. Otherwise state transfer traffic will compete with live standard operations.

This problem is amplified when we have multiple state transfers going on. I suppose we could dynamically create a new JGroups channel for this, but it seems like a lot of overhead for a single p2p connection.

BTW, is a *JGroups* streaming API necessary here? The old AS Farm service passed arbitrary sized files by sending byte[] chunks via RpcDispatcher calls. Worked fine. That's not quite what JBC would need, since FarmService read a chunk from a FileInputStream and passed it to JGroups; you'd want an OutputStream impl that would pass a chunk to JGroups when it's internal buffer reached size X.

Bela Ban wrote: > > > Brian Stansberry wrote: >> BTW, is a *JGroups* streaming API necessary here? The old AS Farm >> service passed arbitrary sized files by sending byte[] chunks via >> RpcDispatcher calls. Worked fine. That's not quite what JBC would >> need, since FarmService read a chunk from a FileInputStream and >> passed it to JGroups; you'd want an OutputStream impl that would pass >> a chunk to JGroups when it's internal buffer reached size X. > > +1. > > On the side: I know a <big unnamed company> uses streaming state > transfer to do this. Not really the way it was designed for, but it > works well for them. > Here's the link: > http://www.nabble.com/Streaming-Message-Transfer---td14476343.html. > The 3rd posting from the top describes their approach. > > I still think though that JBC implementing this via chunks would be > the better solution. > > So there does appear to be a common need for sending a large stream to members in the group. JBC needs it, Streaming State Transfer needs it, Farming needs it, and at least one customer needs. The question is, should jgroups support the capability at its core, or should such use cases use a p2p framework. Chunking over a messaging api will work, but it is definitely not the optimal approach. It is probably good enough. However, i suspect it becomes a problem when people have gigs of data to send.

> On the side: I know a <big unnamed company> uses streaming state > transfer to do this. Not really the way it was designed for, but it > works well for them. > Here's the link: > http://www.nabble.com/Streaming-Message-Transfer---td14476343.html. > The 3rd posting from the top describes their approach. > > I still think though that JBC implementing this via chunks would be > the better solution. > > So there does appear to be a common need for sending a large stream to members in the group. JBC needs it, Streaming State Transfer needs it, Farming needs it, and at least one customer needs. The question is, should jgroups support the capability at its core, or should such use cases use a p2p framework. Chunking over a messaging api will work, but it is definitely not the optimal approach. It is probably good enough. However, i suspect it becomes a problem when people have gigs of data to send.

On 6 Jan 2009, at 22:05, Brian Stansberry wrote: > > BTW, is a *JGroups* streaming API necessary here? The old AS Farm > service passed arbitrary sized files by sending byte[] chunks via > RpcDispatcher calls. Worked fine. That's not quite what JBC would > need, since FarmService read a chunk from a FileInputStream and > passed it to JGroups; you'd want an OutputStream impl that would pass > a chunk to JGroups when it's internal buffer reached size X. This could be done but this would involve excessive copying of buffers. The whole purpose behind the JGroups streaming JIRA is to optimise away such unnecessary buffer copying between layers. -- Manik Surtani Lead, JBoss Cache http://www.jbosscache.org manik(a)jboss.org <mailto:manik@jboss.org> ------------------------------------------------------------------------ _______________________________________________ jbosscache-dev mailing list jbosscache-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jbosscache-dev

Jason T. Greene wrote: > Hi All, > > I wanted to summarize my initial research into NBST. The planned > design (outlined in the wiki: > http://www.jboss.org/community/docs/DOC-10275) only needs to block > transactional activity once, at the end of the process when sending > the tx log. Unfortunately it appears that flush and partial flush can > not be used for this, since the application needs the ability to send > state (tx log) during the flush. This is possible because FLUSH blocks only *multicast* messages, not *unicast* messages. This is guaranteed because virtual synchrony applies only to group messages.

> I.e. transactions need to be paused by only 2 nodes, while the > transfer state. What happens when A and D pause transactions and C wants to commit TX-5 ? Would, assuming the pause takes longer than the commit, TX-5 get rolled back ? If that's the case then I think a regular flush would not hurt.

> This however is not a big deal because we can just do this in JBoss > Cache using a normal RPC message that flips a gate. > > In addition, the state transfer and streaming state transfer > facilities in jgroups can not be used (since they are designed around > blocking the entire group). No, both state transfer protocols are completely independent from FLUSH: they work with or without FLUSH being present.

> This means JBoss Cache needs to stream state itself. Ideally this > would be a separate point-to-point connection, since we don't want to > pollute multicast traffic with potentially huge volumes of noise. > Currently jgroups does not yet support a streaming API like this: > https://jira.jboss.org/jira/browse/JGRP-653 > > IMO This leaves us with 3 options: > > 1. Wait on JGRP-653 (upping its priority), also add requirements for a > p2p connection. On JGRP-653, I argued against JGRP-653 :-) I don't think it is necessary, because this can be done by application code, e.g. similar to what Brian has implemented with farming: the application chunks up data from an input stream and sends the chunks (governed by flow control). The receiving side reads the chunks and feeds them to the recipient (which e.g. reads them from an input stream).

> 2. Implement our own p2p connection using tcp (probably using xnio) Yuck ! > 3. Somehow enhance state transfer / partial flush to meet our needs Which enhancements do you have in mind ? FLUSH between 2 participant *is* implemented unless I'm completely mistaken, Vladimir, can you comment on this ? Not sure though it is needed in the first place...

Bela Ban wrote: > Jason T. Greene wrote: >> Hi All, >> >> I wanted to summarize my initial research into NBST. The planned >> design (outlined in the wiki: http://www.jboss.org/community/docs/DOC-10275) >> only needs to block transactional activity once, at the end of >> the process when sending the tx log. Unfortunately it appears that >> flush and partial flush can not be used for this, since the >> application needs the ability to send state (tx log) during the >> flush. > This is possible because FLUSH blocks only *multicast* messages, > not *unicast* messages. This is guaranteed because virtual > synchrony applies only to group messages. Ah, ok then I can use flush for the final block.

>> >> 1. Wait on JGRP-653 (upping its priority), also add requirements >> for a p2p connection. > On JGRP-653, I argued against JGRP-653 :-) I don't think it is > necessary, because this can be done by application code, e.g. > similar to what Brian has implemented with farming: the application > chunks up data from an input stream and sends the chunks (governed > by flow control). The receiving side reads the chunks and feeds > them to the recipient (which e.g. reads them from an input stream). > The advantage of supporting this is that you can optimize a number of aspects including the transport, how much of the protocol stack is used, etc when a stream is used. Basically, when someone uses this, they really want a tcp connection.