[jboss-jira] [JBoss JIRA] Commented: (JGRP-653) Streaming API for large messages

[Manik Surtani (JIRA)]

    [ http://jira.jboss.com/jira/browse/JGRP-653?page=comments#action_12419585 ] 
            
Manik Surtani commented on JGRP-653:
------------------------------------

That makes sense in the case of a DVD which is treated as a byte array.  In the case of streaming a large Object (for example, a large XML DOM tree, but any arbitrary Object applies), it is a bit different:

Without using streams, this is what would happen, for example in the JGroups Marshaller:

Sender:
1.  Write entire Object to byte array using an ObjectOutputStream and a ByteArrayOutputStream
2.  Return byte array for transmission

// already we are consuming twice the memory space of the Object!

Receiver:
1.  Retrieve entire byte array
2.  Deserialize object from byte array using an ObjectInputStream and a ByteArrayInputStream

// again, consuming twice as much memory

Using a JGroups streaming API, we could potentially do something like:

Sender:
1.  Write Object using an ObjectOutputStream to wrap a JGroupsOutputStream.

Receiver:
1.  Deserialize Object using an ObjectInputStream to wrap a JGroupsInputStream

The JGroupsInput/OutputStreams could have a small internal byte buffer (perhaps based on message payload size) so as soon as enough bytes have been read the message can be transmitted, received and deserialized.

> Streaming API for large messages
> --------------------------------
>
>                 Key: JGRP-653
>                 URL: http://jira.jboss.com/jira/browse/JGRP-653
>             Project: JGroups
>          Issue Type: Feature Request
>            Reporter: Bela Ban
>         Assigned To: Bela Ban
>             Fix For: 2.7
>
>         Attachments: JGroupsInputStream.java, JGroupsOutputStream.java, StreamTest.java
>
>
> For large messages, to load the entire payload into memory might be bad because the payload might be bigger than the max memory available. It would be useful to have an API which allows for use of input and output streams, so that large payloads can be read iteratively by a user and streamed out to the cluster via the underlying channel breaking the data in the input stream into chunks, which are fed into the input stream on the receivers side.
> Issues: we have to have 1 input stream per sender on the receiver side, because a stream is always defined between 2 parties (sender, receiver). Maybe something like NIO, where we register interest in a stream, are notified of new streams ('accept()') and get notified when data on any of the stream is available, would be beneficial.
> Demo is attached

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira