[jboss-jira] [JBoss JIRA] Commented: (JGRP-653) Streaming API for large messages

[Bela Ban (JIRA)]

    [ http://jira.jboss.com/jira/browse/JGRP-653?page=comments#action_12419573 ] 
            
Bela Ban commented on JGRP-653:
-------------------------------

I'm wondering if a streaming API is needed after all: when a user needs to transfer a large object (say a DVD ISO image of 4 GB size), then he would do the following:
- Open an input stream to the DVD's file
- Get a JGroupsOutputStream
- Allocate a small buffer (say 8K)
- While(end of file not reached on input stream)
  - Read 8K from input stream into buffer
  - Write buffer to output stream

On the receiver's side:
- Get JGroupsInputStream
- Create an output stream (file)
- Allocate a small buffer (say 8K)
- While(not end of file)
  - Read data from input stream into buffer
  - Write data to output stream (file)
- Close output stream

If we didn't have JGroups{Input,Output}Streams, we could still achieve the same:

- Open an input stream to the DVD's file
- Allocate a small buffer (say 8K)
- Send CREATE(filename) message
- While(end of file not reached on input stream)
  - Read 8K from input stream into buffer
  - Send 8K WRITE message (include filename in message)
- Send CLOSE(filename) message

On the receiver's side:
- On reception of CREATE(filename) message:
  - Create an output stream (file) for filename

- On reception of WRITE message:
  - Append the received byte buffer to the output stream (file)

- On reception of CLOSE message (filename)
  - Close output stream for filename

> Streaming API for large messages
> --------------------------------
>
>                 Key: JGRP-653
>                 URL: http://jira.jboss.com/jira/browse/JGRP-653
>             Project: JGroups
>          Issue Type: Feature Request
>            Reporter: Bela Ban
>         Assigned To: Bela Ban
>             Fix For: 2.7
>
>         Attachments: JGroupsInputStream.java, JGroupsOutputStream.java, StreamTest.java
>
>
> For large messages, to load the entire payload into memory might be bad because the payload might be bigger than the max memory available. It would be useful to have an API which allows for use of input and output streams, so that large payloads can be read iteratively by a user and streamed out to the cluster via the underlying channel breaking the data in the input stream into chunks, which are fed into the input stream on the receivers side.
> Issues: we have to have 1 input stream per sender on the receiver side, because a stream is always defined between 2 parties (sender, receiver). Maybe something like NIO, where we register interest in a stream, are notified of new streams ('accept()') and get notified when data on any of the stream is available, would be beneficial.
> Demo is attached

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira