wget vs netty

[Trustin Lee (이희승)]

Hi David,

Just to make sure disk I/O is not a bottleneck, could you try to
configure wget and your Netty application to discard the received data
to /dev/null?

I also wonder if you are using Netty's built-in HTTP codec or your own one.

— Trustin Lee, http://gleamynode.net/

On Sat, Nov 7, 2009 at 7:02 PM, Hoyt, David <hoyt6 at llnl.gov> wrote:
> Our application does a lot of downloading -- I notice that when I test my netty app against wget for some basic non-encrypted http streams, wget outperforms my netty app. My basic test is I start 2 wget processes and have them download some content from our server. I then have netty download the same content. I give each 20 seconds to download as much as they can and then I cancel it (the file contents are produced on-the-fly at a constant rate of ~500 KB/s). wget consistently does better than my java app (by ~3MB per file with 2 concurrent downloads). It's a bit confusing, b/c my CPU is nowhere near maxed out using either wget or netty (java). The discrepancy gets much higher when I start doing 10+ concurrent downloads. wget will download an 11 MB file while my java app gets ~700-800 KB. Every message (http chunk) I get from netty is immediately flushed to disk and I'm using a FileChannel for what I figured was optimum performance. It should be able to do the full 11 MB eve!
>  n if it takes a little while for it to come in. 2 streams of 500 KB/s (so ~ 1 MB/s) should be easily handled by java/netty I would think.
>
> I don't think I'm IO-bound - the disk can easily write ~22 MB (the file is ~11 MB) and wget doesn't seem to have a problem writing to disk. I'm on a gigabit line and 22 MB over 20 seconds shouldn't be saturating the connection.
>
> I'm wondering if there's something I'm missing here. I'm following the http examples pretty closely. I use keepAlive, reuseAddress, and tcpNoDelay for the client bootstrap. I made the threads in my thread pool have MAX_PRIORITY to help rule out scheduler issues. Everything else is using the defaults.
>
> I did a quick old IO (OIO) implementation and it seemed to suffer more than netty. I thought perhaps that something I was doing may have cut off the content before it could be flushed to the disk, but using OIO and input streams should have gotten the full content eventually.
>
> I could really use some help or some pointers on what the community does for netty and/or network optimizations.
>
> I'm doing my testing on a Windows box but we'll be deploying for linux. The behavior appears on both platforms (Windows XP and CentOS).
>
> Thanks,
> - David Hoyt
>
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users
>