blocking when sending lots of data

[mra]

thanks trustin.

i see that you use java.util.Queue.offer() in ChunkedWriteHandler to
account for the downstream message, which doesn't block if queue's
capacity is full. so it seems as though excess netty messages are
simply dropped (line 98 in 3.1.0.CR1).

so could you suggest an implementation that would actually block the
writer, if more data is being generated via netty than can be handled
by the underlying o/s's tcp/ip implementation?

best,
mike


On Mon, Jul 6, 2009 at 12:43 PM, Trustin Lee (via
Nabble)<ml-user+57722-393564278 at n2.nabble.com> wrote:
> On 07/06/2009 09:37 PM, mra wrote:
>> in my system i often want to send out LOTS of data in a burst, maybe 100's
>> or
>> 1000's of packets (google protobufs) of about 1,000,000 bytes each. now i
>> know in a typical use of OIO, writing to Socket.getOutputStream(), the O/S
>> will block if TCP and its buffers can't immediately absorb the data. but
>> with netty, i'm worried that an unlimited number of bytes could wind up
>> absorbed into the JVM even if the underlying TCP blocks, leading to
>> OutOfMemory exceptions? i think i may have observed this in practice with
>> netty (but i'm not 100% sure that's the root cause), and so i'm wondering
>> if
>> i should throttle  output manually instead.
>
> Channel has 'isWritable()' method, which returns false if the channels
> internal buffer is filled up with write request so that writing more
> data into the channel could lead to OOME.  So, if you are going to write
> a large amount of data, you need to check if isWritable() returns true
> before writing something.  If it returns false, you need to wait until
> channelInterestChanged event is fired.  In your handler's
> channelInterestChanged method, you can continue writing the remaining
> data (also making sure isWrite() returns true.)
>
> Actually, there's a built-in handler called ChunkedWriteHandler which
> allows you to write a stream or a file very easily without experiencing
> memory pressure.  Please refer to the HTTP file server example.  You
> might also want to review the source code of ChunkedWriteHandler to
> understand how I implemented it.
>
> HTH,
> Trustin
> _______________________________________________
> netty-users mailing list
> netty-users at ...
> https://lists.jboss.org/mailman/listinfo/netty-users
> — Trustin Lee, http://gleamynode.net/
>
> ________________________________
> View message @
> http://n2.nabble.com/blocking-when-sending-lots-of-data-tp3212806p3214053.html
> To unsubscribe from blocking when sending lots of data, click here.
>



-- 
permanent contact information at http://mikerandrews.com

-- 
View this message in context: http://n2.nabble.com/blocking-when-sending-lots-of-data-tp3212806p3214592.html
Sent from the Netty User Group mailing list archive at Nabble.com.