Coalesced Sending

[daviderickson]

Hi I was playing around with Netty over the weekend, specifically comparing
it to some C++ test code using Boost.ASIO.  My test Netty code is super
simple, I have an incoming binary protocol inside TCP sending messages of
size 82 bytes, Netty splits these using a LengthFieldBasedFrameDecoder, then
I have a class extending SimpleChannelUpstreamHandler that does the
following:

    @Override
    public void messageReceived(ChannelHandlerContext ctx, MessageEvent e)
            throws Exception {
        ChannelBuffer buf =
((ChannelBuffer)e.getMessage()).factory().getBuffer(24);
        buf.writeBytes(myBytes);
        e.getChannel().write(buf);
    }

Where myBytes is a static 24 byte sized byte[] just for testing.

Now I've been benchmarking this on my machine with a client program that
always keeps the incoming tcp window full of packets containing messages, on
loopback these packets grow all the way to 49220 bytes, ie containing 600
messages.  Now to get high throughput its important that the output of these
messages gets coalesced into equally large outgoing packets, otherwise the
work per packet isn't amortized and throughput drops.  I've noticed with
Boost that it is delivering huge 150k sized buffers to the kernel via
sendmsg, thus crossing the user/kernel barrier very few times, and
outputting 50k size packets even with nodelay turned on.  Whereas with
Netty/Java every channel write is causing a system write call with 24 bytes,
and as a result a huge amount of time is being wasted in crossing that
barrier with each write, rather than doing actual work.  Does Netty have any
solutions on how to deal with this?  It seems tricky because you have to
balance promptness of the send along with chance to coalesce and be more
efficient.

I also just ran a quick profile on the code and 47% of the time is spent in
org.jboss.netty.channel.socket.nio.SocketSendBufferPool$PooledSendBuffer.transferTo
and ultimately IOUtil.write, and 95% of the time is spent from
Channels.write() and downwward..  I'd really like to decrease this number if
possible.

Thanks,
David

--
View this message in context: http://netty-forums-and-mailing-lists.685743.n2.nabble.com/Coalesced-Sending-tp6343303p6343303.html
Sent from the Netty User Group mailing list archive at Nabble.com.