Netty/HTTP dropping connections with httpperf

[Sébastien Pierre]

Hi Trustin,

I've not made the HttpCore/NIO vs Netty benchmark yet, but I'm doing instead
some more benchmarks, using 'ab' this time. I also get quite often this
error, again when the -C parameter is over 2500:

>> ab -n10000 -c3000 'http://69.164.208.81:8000/'
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
*apr_socket_recv: Connection reset by peer (104)*
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 69.164.208.81 (be patient)
Total of 4123 requests completed

and at some point when -C=6000), all connections are properly processed. So
it's likely to be a sytem configuration problem, but then at some point the
Netty server becomes unresponsive.

Would you have some tips on how to find the reason for the connection reset
and also on how to detect (from the Netty server) that Netty becomes
unresponsive ?

Thanks,

 -- Sébastien

2010/1/6 "Trustin Lee (이희승)" <trustin at gmail.com>

> Hi Sébastien,
>
> You might be running out of available ports due to many TIME_WAIT either
> on the client or server side.  Could you confirm it?  If so, you need to
> wait until the TIME_WAIT state is cleared between each run.
>
> If HttpCore/NIO version works fine when Netty fails, I'd like to compare
> the two to see what the differences are.  Please paste or attach the
> source code.
>
> Thanks,
> Trustin
>
> Sébastien Pierre wrote:
> > Hi there !
> >
> > I just moved a test HTTP service from HttpCore/NIO to Netty based on the
> > HTTP snoop example. I experienced (despite good performance at first)
> that
> > the server is dropping connections when the load is heavy. To reproduce
> > this, simply start the snoop example:
> >
> > java -cp ./src/main/java/:./jar/netty-3.2.0.ALPHA2.jar
> > org.jboss.netty.example.http.snoop.HttpServer
> >
> > and then do
> >
> > python -c'for r in range(500,10500,500): import os ; os.system("httperf
> > --hog --timeout=60 --client=0/1 --server=127.0.0.1 --port=8080 --uri=/
> > --rate=%s --send-buffer=4096 --recv-buffer=16384 --num-conns=10000
> > --num-calls=1" % (r))'
> >
> > which is the same as running this command with --rate growing from 500 to
> > 10000 with 500 increases.
> >
> > httperf --hog --timeout=60 --client=0/1 --server=127.0.0.1 --port=8080
> > --uri=/ --rate=500 --send-buffer=4096 --recv-buffer=16384
> --num-conns=10000
> > --num-calls=1
> >
> > so in this example, Netty often drops connections between rates of 3000
> and
> > 5000, which you'll see in httpperf's log with the following lines:
> >
> > httperf: connection failed with unexpected error 98
> > ...
> > Errors: total 2236 client-timo 0 socket-timo 0 connrefused 0 connreset 0
> > Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 2236
> >
> > I attached a graph where you'll see that for the rate=4000 and rate=45000
> > the server just dropped all the connections (from httperf's perspective
> at
> > least). So do you know any way to prevent or at least detect that ?
> >
> > Thanks !
> >
> >  -- Sébastien
> >
> > PS: I should add that I've run the test a couple of times, and quite
> often
> > have trouble in the rate=3000-5000 range
> >
> >
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > netty-users mailing list
> > netty-users at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/netty-users
>
> --
> what we call human nature in actuality is human habit
> http://gleamynode.net/
>
>
>
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/netty-users/attachments/20100107/367ec748/attachment.html