[jboss-jira] [JBoss JIRA] Commented: (JGRP-653) Streaming API for large messages
[ http://jira.jboss.com/jira/browse/JGRP-653?page=comments#action_12419573 ]
Bela Ban commented on JGRP-653:
-------------------------------
I'm wondering if a streaming API is needed after all: when a user needs to transfer a
large object (say a DVD ISO image of 4 GB size), then he would do the following:
- Open an input stream to the DVD's file
- Get a JGroupsOutputStream
- Allocate a small buffer (say 8K)
- While(end of file not reached on input stream)
- Read 8K from input stream into buffer
- Write buffer to output stream
On the receiver's side:
- Get JGroupsInputStream
- Create an output stream (file)
- Allocate a small buffer (say 8K)
- While(not end of file)
- Read data from input stream into buffer
- Write data to output stream (file)
- Close output stream
If we didn't have JGroups{Input,Output}Streams, we could still achieve the same:
- Open an input stream to the DVD's file
- Allocate a small buffer (say 8K)
- Send CREATE(filename) message
- While(end of file not reached on input stream)
- Read 8K from input stream into buffer
- Send 8K WRITE message (include filename in message)
- Send CLOSE(filename) message
On the receiver's side:
- On reception of CREATE(filename) message:
- Create an output stream (file) for filename
- On reception of WRITE message:
- Append the received byte buffer to the output stream (file)
- On reception of CLOSE message (filename)
- Close output stream for filename
Streaming API for large messages
--------------------------------
Key: JGRP-653
URL: http://jira.jboss.com/jira/browse/JGRP-653
Project: JGroups
Issue Type: Feature Request
Reporter: Bela Ban
Assigned To: Bela Ban
Fix For: 2.7
Attachments: JGroupsInputStream.java, JGroupsOutputStream.java, StreamTest.java
For large messages, to load the entire payload into memory might be bad because the
payload might be bigger than the max memory available. It would be useful to have an API
which allows for use of input and output streams, so that large payloads can be read
iteratively by a user and streamed out to the cluster via the underlying channel breaking
the data in the input stream into chunks, which are fed into the input stream on the
receivers side.
Issues: we have to have 1 input stream per sender on the receiver side, because a stream
is always defined between 2 parties (sender, receiver). Maybe something like NIO, where we
register interest in a stream, are notified of new streams ('accept()') and get
notified when data on any of the stream is available, would be beneficial.
Demo is attached
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira