Netty/HTTP dropping connections with httpperf

[Sébastien Pierre]

Hi Trustin,

So I checked the number of sockets in *_WAIT state on both client and server
machines, using the following commands

server> watch -n1 'lsof -nl | egrep "TCP|UDP" | grep "java" | wc -l '
client> watch -n1 'lsof -nl | grep WAIT | wc -l'

the number of opened sockets on the server oscillates between 0 and 500,
while the number of *WAIT sockets on the client is always 0, so I doubt it's
related to an overflow of TIME_WAIT or CLOSE_WAIT.

I'll prepare more detailed tests with the HttpCore/NIO version.

 -- Sébastien

2010/1/6 "Trustin Lee (이희승)" <trustin at gmail.com>

> Hi Sébastien,
>
> You might be running out of available ports due to many TIME_WAIT either
> on the client or server side.  Could you confirm it?  If so, you need to
> wait until the TIME_WAIT state is cleared between each run.
>
> If HttpCore/NIO version works fine when Netty fails, I'd like to compare
> the two to see what the differences are.  Please paste or attach the
> source code.
>
> Thanks,
> Trustin
>
> Sébastien Pierre wrote:
> > Hi there !
> >
> > I just moved a test HTTP service from HttpCore/NIO to Netty based on the
> > HTTP snoop example. I experienced (despite good performance at first)
> that
> > the server is dropping connections when the load is heavy. To reproduce
> > this, simply start the snoop example:
> >
> > java -cp ./src/main/java/:./jar/netty-3.2.0.ALPHA2.jar
> > org.jboss.netty.example.http.snoop.HttpServer
> >
> > and then do
> >
> > python -c'for r in range(500,10500,500): import os ; os.system("httperf
> > --hog --timeout=60 --client=0/1 --server=127.0.0.1 --port=8080 --uri=/
> > --rate=%s --send-buffer=4096 --recv-buffer=16384 --num-conns=10000
> > --num-calls=1" % (r))'
> >
> > which is the same as running this command with --rate growing from 500 to
> > 10000 with 500 increases.
> >
> > httperf --hog --timeout=60 --client=0/1 --server=127.0.0.1 --port=8080
> > --uri=/ --rate=500 --send-buffer=4096 --recv-buffer=16384
> --num-conns=10000
> > --num-calls=1
> >
> > so in this example, Netty often drops connections between rates of 3000
> and
> > 5000, which you'll see in httpperf's log with the following lines:
> >
> > httperf: connection failed with unexpected error 98
> > ...
> > Errors: total 2236 client-timo 0 socket-timo 0 connrefused 0 connreset 0
> > Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 2236
> >
> > I attached a graph where you'll see that for the rate=4000 and rate=45000
> > the server just dropped all the connections (from httperf's perspective
> at
> > least). So do you know any way to prevent or at least detect that ?
> >
> > Thanks !
> >
> >  -- Sébastien
> >
> > PS: I should add that I've run the test a couple of times, and quite
> often
> > have trouble in the rate=3000-5000 range
> >
> >
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > netty-users mailing list
> > netty-users at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/netty-users
>
> --
> what we call human nature in actuality is human habit
> http://gleamynode.net/
>
>
>
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/netty-users/attachments/20100107/50e6e939/attachment.html