5:18 a.m.
Streaming API for large messages
--------------------------------
Key: JGRP-653
URL: http://jira.jboss.com/jira/browse/JGRP-653
Project: JGroups
Issue Type: Feature Request
Reporter: Bela Ban
Assigned To: Bela Ban
Fix For: 2.x
For large messages, to load the entire payload into memory might be bad because the
payload might be bigger than the max memory available. It would be useful to have an API
which allows for use of input and output streams, so that large payloads can be read
iteratively by a user and streamed out to the cluster via the underlying channel breaking
the data in the input stream into chunks, which are fed into the input stream on the
receivers side.
Issues: we have to have 1 input stream per sender on the receiver side, because a stream
is always defined between 2 parties (sender, receiver). Maybe something like NIO, where we
register interest in a stream, are notified of new streams ('accept()') and get
notified when data on any of the stream is available, would be beneficial.
Demo is attached
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:20 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-653) Streaming API for large messages
[ http://jira.jboss.com/jira/browse/JGRP-653?page=all ]
Bela Ban updated JGRP-653:
--------------------------
Attachment: JGroupsOutputStream.java
Streaming API for large messages
--------------------------------
Key: JGRP-653
URL: http://jira.jboss.com/jira/browse/JGRP-653
Project: JGroups
Issue Type: Feature Request
Reporter: Bela Ban
Assigned To: Bela Ban
Fix For: 2.x
Attachments: JGroupsInputStream.java, JGroupsOutputStream.java, StreamTest.java
For large messages, to load the entire payload into memory might be bad because the
payload might be bigger than the max memory available. It would be useful to have an API
which allows for use of input and output streams, so that large payloads can be read
iteratively by a user and streamed out to the cluster via the underlying channel breaking
the data in the input stream into chunks, which are fed into the input stream on the
receivers side.
Issues: we have to have 1 input stream per sender on the receiver side, because a stream
is always defined between 2 parties (sender, receiver). Maybe something like NIO, where we
register interest in a stream, are notified of new streams ('accept()') and get
notified when data on any of the stream is available, would be beneficial.
Demo is attached
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:20 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-653) Streaming API for large messages
[ http://jira.jboss.com/jira/browse/JGRP-653?page=all ]
Bela Ban updated JGRP-653:
--------------------------
Attachment: JGroupsInputStream.java
Streaming API for large messages
--------------------------------
Key: JGRP-653
URL: http://jira.jboss.com/jira/browse/JGRP-653
Project: JGroups
Issue Type: Feature Request
Reporter: Bela Ban
Assigned To: Bela Ban
Fix For: 2.x
Attachments: JGroupsInputStream.java, JGroupsOutputStream.java, StreamTest.java
For large messages, to load the entire payload into memory might be bad because the
payload might be bigger than the max memory available. It would be useful to have an API
which allows for use of input and output streams, so that large payloads can be read
iteratively by a user and streamed out to the cluster via the underlying channel breaking
the data in the input stream into chunks, which are fed into the input stream on the
receivers side.
Issues: we have to have 1 input stream per sender on the receiver side, because a stream
is always defined between 2 parties (sender, receiver). Maybe something like NIO, where we
register interest in a stream, are notified of new streams ('accept()') and get
notified when data on any of the stream is available, would be beneficial.
Demo is attached
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:20 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-653) Streaming API for large messages
[ http://jira.jboss.com/jira/browse/JGRP-653?page=all ]
Bela Ban updated JGRP-653:
--------------------------
Attachment: StreamTest.java
Streaming API for large messages
--------------------------------
Key: JGRP-653
URL: http://jira.jboss.com/jira/browse/JGRP-653
Project: JGroups
Issue Type: Feature Request
Reporter: Bela Ban
Assigned To: Bela Ban
Fix For: 2.x
Attachments: JGroupsInputStream.java, JGroupsOutputStream.java, StreamTest.java
For large messages, to load the entire payload into memory might be bad because the
payload might be bigger than the max memory available. It would be useful to have an API
which allows for use of input and output streams, so that large payloads can be read
iteratively by a user and streamed out to the cluster via the underlying channel breaking
the data in the input stream into chunks, which are fed into the input stream on the
receivers side.
Issues: we have to have 1 input stream per sender on the receiver side, because a stream
is always defined between 2 parties (sender, receiver). Maybe something like NIO, where we
register interest in a stream, are notified of new streams ('accept()') and get
notified when data on any of the stream is available, would be beneficial.
Demo is attached
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:58 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-653) Streaming API for large messages
[ http://jira.jboss.com/jira/browse/JGRP-653?page=comments#action_12399223 ]
Bela Ban commented on JGRP-653:
-------------------------------
- Receiver gets an abstraction in the form of an NIO selectable channel. Handback object
needs to contain sender's and receiver's address. We don't need headers, as
the app can write/read additional data directly to the channel
- On the sender side, we have 1 output stream for the multicast address, and N for the
unicast destinations. There's an abstraction (StreamableJChannel ?) which allows us to
fetch the right output stream based on the destination
- Provide an implementation of blocking method calls based on StreamingJChannel
- Provide a new impl of RpcDispatcher/MessageDispatcher, based on an input and output
stream
- How do we send OOB messages (flags in general) ?
Streaming API for large messages
--------------------------------
Key: JGRP-653
URL: http://jira.jboss.com/jira/browse/JGRP-653
Project: JGroups
Issue Type: Feature Request
Reporter: Bela Ban
Assigned To: Bela Ban
Fix For: 2.7
Attachments: JGroupsInputStream.java, JGroupsOutputStream.java, StreamTest.java
For large messages, to load the entire payload into memory might be bad because the
payload might be bigger than the max memory available. It would be useful to have an API
which allows for use of input and output streams, so that large payloads can be read
iteratively by a user and streamed out to the cluster via the underlying channel breaking
the data in the input stream into chunks, which are fed into the input stream on the
receivers side.
Issues: we have to have 1 input stream per sender on the receiver side, because a stream
is always defined between 2 parties (sender, receiver). Maybe something like NIO, where we
register interest in a stream, are notified of new streams ('accept()') and get
notified when data on any of the stream is available, would be beneficial.
Demo is attached
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:58 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-653) Streaming API for large messages
[ http://jira.jboss.com/jira/browse/JGRP-653?page=all ]
Bela Ban updated JGRP-653:
--------------------------
Fix Version/s: 2.7
(was: 2.x)
Streaming API for large messages
--------------------------------
Key: JGRP-653
URL: http://jira.jboss.com/jira/browse/JGRP-653
Project: JGroups
Issue Type: Feature Request
Reporter: Bela Ban
Assigned To: Bela Ban
Fix For: 2.7
Attachments: JGroupsInputStream.java, JGroupsOutputStream.java, StreamTest.java
For large messages, to load the entire payload into memory might be bad because the
payload might be bigger than the max memory available. It would be useful to have an API
which allows for use of input and output streams, so that large payloads can be read
iteratively by a user and streamed out to the cluster via the underlying channel breaking
the data in the input stream into chunks, which are fed into the input stream on the
receivers side.
Issues: we have to have 1 input stream per sender on the receiver side, because a stream
is always defined between 2 parties (sender, receiver). Maybe something like NIO, where we
register interest in a stream, are notified of new streams ('accept()') and get
notified when data on any of the stream is available, would be beneficial.
Demo is attached
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:34 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-653) Streaming API for large messages
[ http://jira.jboss.com/jira/browse/JGRP-653?page=all ]
Bela Ban updated JGRP-653:
--------------------------
Priority: Critical (was: Major)
Streaming API for large messages
--------------------------------
Key: JGRP-653
URL: http://jira.jboss.com/jira/browse/JGRP-653
Project: JGroups
Issue Type: Feature Request
Reporter: Bela Ban
Assigned To: Bela Ban
Priority: Critical
Fix For: 2.7
Attachments: JGroupsInputStream.java, JGroupsOutputStream.java, StreamTest.java
For large messages, to load the entire payload into memory might be bad because the
payload might be bigger than the max memory available. It would be useful to have an API
which allows for use of input and output streams, so that large payloads can be read
iteratively by a user and streamed out to the cluster via the underlying channel breaking
the data in the input stream into chunks, which are fed into the input stream on the
receivers side.
Issues: we have to have 1 input stream per sender on the receiver side, because a stream
is always defined between 2 parties (sender, receiver). Maybe something like NIO, where we
register interest in a stream, are notified of new streams ('accept()') and get
notified when data on any of the stream is available, would be beneficial.
Demo is attached
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:29 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-653) Streaming API for large messages
[ http://jira.jboss.com/jira/browse/JGRP-653?page=all ]
Bela Ban updated JGRP-653:
--------------------------
Priority: Major (was: Critical)
Streaming API for large messages
--------------------------------
Key: JGRP-653
URL: http://jira.jboss.com/jira/browse/JGRP-653
Project: JGroups
Issue Type: Feature Request
Reporter: Bela Ban
Assigned To: Bela Ban
Fix For: 2.7
Attachments: JGroupsInputStream.java, JGroupsOutputStream.java, StreamTest.java
For large messages, to load the entire payload into memory might be bad because the
payload might be bigger than the max memory available. It would be useful to have an API
which allows for use of input and output streams, so that large payloads can be read
iteratively by a user and streamed out to the cluster via the underlying channel breaking
the data in the input stream into chunks, which are fed into the input stream on the
receivers side.
Issues: we have to have 1 input stream per sender on the receiver side, because a stream
is always defined between 2 parties (sender, receiver). Maybe something like NIO, where we
register interest in a stream, are notified of new streams ('accept()') and get
notified when data on any of the stream is available, would be beneficial.
Demo is attached
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:13 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-653) Streaming API for large messages
[ http://jira.jboss.com/jira/browse/JGRP-653?page=comments#action_12419573 ]
Bela Ban commented on JGRP-653:
-------------------------------
I'm wondering if a streaming API is needed after all: when a user needs to transfer a
large object (say a DVD ISO image of 4 GB size), then he would do the following:
- Open an input stream to the DVD's file
- Get a JGroupsOutputStream
- Allocate a small buffer (say 8K)
- While(end of file not reached on input stream)
- Read 8K from input stream into buffer
- Write buffer to output stream
On the receiver's side:
- Get JGroupsInputStream
- Create an output stream (file)
- Allocate a small buffer (say 8K)
- While(not end of file)
- Read data from input stream into buffer
- Write data to output stream (file)
- Close output stream
If we didn't have JGroups{Input,Output}Streams, we could still achieve the same:
- Open an input stream to the DVD's file
- Allocate a small buffer (say 8K)
- Send CREATE(filename) message
- While(end of file not reached on input stream)
- Read 8K from input stream into buffer
- Send 8K WRITE message (include filename in message)
- Send CLOSE(filename) message
On the receiver's side:
- On reception of CREATE(filename) message:
- Create an output stream (file) for filename
- On reception of WRITE message:
- Append the received byte buffer to the output stream (file)
- On reception of CLOSE message (filename)
- Close output stream for filename
Streaming API for large messages
--------------------------------
Key: JGRP-653
URL: http://jira.jboss.com/jira/browse/JGRP-653
Project: JGroups
Issue Type: Feature Request
Reporter: Bela Ban
Assigned To: Bela Ban
Fix For: 2.7
Attachments: JGroupsInputStream.java, JGroupsOutputStream.java, StreamTest.java
For large messages, to load the entire payload into memory might be bad because the
payload might be bigger than the max memory available. It would be useful to have an API
which allows for use of input and output streams, so that large payloads can be read
iteratively by a user and streamed out to the cluster via the underlying channel breaking
the data in the input stream into chunks, which are fed into the input stream on the
receivers side.
Issues: we have to have 1 input stream per sender on the receiver side, because a stream
is always defined between 2 parties (sender, receiver). Maybe something like NIO, where we
register interest in a stream, are notified of new streams ('accept()') and get
notified when data on any of the stream is available, would be beneficial.
Demo is attached
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:26 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-653) Streaming API for large messages
[ http://jira.jboss.com/jira/browse/JGRP-653?page=comments#action_12419585 ]
Manik Surtani commented on JGRP-653:
------------------------------------
That makes sense in the case of a DVD which is treated as a byte array. In the case of
streaming a large Object (for example, a large XML DOM tree, but any arbitrary Object
applies), it is a bit different:
Without using streams, this is what would happen, for example in the JGroups Marshaller:
Sender:
1. Write entire Object to byte array using an ObjectOutputStream and a
ByteArrayOutputStream
2. Return byte array for transmission
// already we are consuming twice the memory space of the Object!
Receiver:
1. Retrieve entire byte array
2. Deserialize object from byte array using an ObjectInputStream and a
ByteArrayInputStream
// again, consuming twice as much memory
Using a JGroups streaming API, we could potentially do something like:
Sender:
1. Write Object using an ObjectOutputStream to wrap a JGroupsOutputStream.
Receiver:
1. Deserialize Object using an ObjectInputStream to wrap a JGroupsInputStream
The JGroupsInput/OutputStreams could have a small internal byte buffer (perhaps based on
message payload size) so as soon as enough bytes have been read the message can be
transmitted, received and deserialized.
Streaming API for large messages
--------------------------------
Key: JGRP-653
URL: http://jira.jboss.com/jira/browse/JGRP-653
Project: JGroups
Issue Type: Feature Request
Reporter: Bela Ban
Assigned To: Bela Ban
Fix For: 2.7
Attachments: JGroupsInputStream.java, JGroupsOutputStream.java, StreamTest.java
For large messages, to load the entire payload into memory might be bad because the
payload might be bigger than the max memory available. It would be useful to have an API
which allows for use of input and output streams, so that large payloads can be read
iteratively by a user and streamed out to the cluster via the underlying channel breaking
the data in the input stream into chunks, which are fed into the input stream on the
receivers side.
Issues: we have to have 1 input stream per sender on the receiver side, because a stream
is always defined between 2 parties (sender, receiver). Maybe something like NIO, where we
register interest in a stream, are notified of new streams ('accept()') and get
notified when data on any of the stream is available, would be beneficial.
Demo is attached
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:22 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-653) Streaming API for large messages
[ http://jira.jboss.com/jira/browse/JGRP-653?page=comments#action_12419631 ]
Brian Stansberry commented on JGRP-653:
---------------------------------------
Bela, your DVD example is a very transparent attempt to steal the design from the
FarmService file transfer mechanism. ;-)
Agree w/ Manik that the benefit of a streaming API is for cases where the natural state of
the data being transferred isn't an array or stream of bytes.
Streaming API for large messages
--------------------------------
Key: JGRP-653
URL: http://jira.jboss.com/jira/browse/JGRP-653
Project: JGroups
Issue Type: Feature Request
Reporter: Bela Ban
Assigned To: Bela Ban
Fix For: 2.7
Attachments: JGroupsInputStream.java, JGroupsOutputStream.java, StreamTest.java
For large messages, to load the entire payload into memory might be bad because the
payload might be bigger than the max memory available. It would be useful to have an API
which allows for use of input and output streams, so that large payloads can be read
iteratively by a user and streamed out to the cluster via the underlying channel breaking
the data in the input stream into chunks, which are fed into the input stream on the
receivers side.
Issues: we have to have 1 input stream per sender on the receiver side, because a stream
is always defined between 2 parties (sender, receiver). Maybe something like NIO, where we
register interest in a stream, are notified of new streams ('accept()') and get
notified when data on any of the stream is available, would be beneficial.
Demo is attached
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:06 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-653) Streaming API for large messages
[
https://jira.jboss.org/jira/browse/JGRP-653?page=com.atlassian.jira.plugi...
]
Bela Ban updated JGRP-653:
--------------------------
Fix Version/s: 2.x
(was: 2.7)
Too many open questions, e.g.
- this solves an edge case in JBossCache, so the priority is not high
- Do we need changes in RpcDispatcher as well ?
Streaming API for large messages
--------------------------------
Key: JGRP-653
URL: https://jira.jboss.org/jira/browse/JGRP-653
Project: JGroups
Issue Type: Feature Request
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.x
Attachments: JGroupsInputStream.java, JGroupsOutputStream.java, StreamTest.java
For large messages, to load the entire payload into memory might be bad because the
payload might be bigger than the max memory available. It would be useful to have an API
which allows for use of input and output streams, so that large payloads can be read
iteratively by a user and streamed out to the cluster via the underlying channel breaking
the data in the input stream into chunks, which are fed into the input stream on the
receivers side.
Issues: we have to have 1 input stream per sender on the receiver side, because a stream
is always defined between 2 parties (sender, receiver). Maybe something like NIO, where we
register interest in a stream, are notified of new streams ('accept()') and get
notified when data on any of the stream is available, would be beneficial.
Demo is attached
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6017
days inactive
6223
days old
11 comments
3 participants
participants (3)
-
Bela Ban (JIRA) -
Brian Stansberry (JIRA)
-
Manik Surtani (JIRA)