Netty/HTTP dropping connections with httpperf
Sébastien Pierre
sebastien.pierre at gmail.com
Sat Jan 9 16:33:32 EST 2010
Hi Trustin,
Yes, ab does not stress the system as much as httperf, which I actually
compiled according to this article:
http://gom-jabbar.org/articles/2009/02/04/httperf-and-file-descriptors
httperf will often return error code "98", which is, according to errno.h is
#define EADDRINUSE 98 /* Address already in use */
which doesn't make much sense. I also run the same test on a linode slice,
theand did not experiment any problem around 3K connections, but there are
still conncetion resets.
-- Sébastien
2010/1/8 "Trustin Lee (이희승)" <trustin at gmail.com>
> Hi,
>
> Thank you very much for further investigation!
>
> It is interesting that you don't get any error with ab. ab might not be
> fast enough or httpperf has different behavior.
>
> In your previous message:
>
> >>>> Errors: total 2236 client-timo 0 socket-timo 0 connrefused 0
> connreset 0
> >>>> Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 2236
>
> could you figure out what the 'other' causes are? Are they simply
> 'connection reset by peer' error, or something else?
>
> I will run the test by myself soon and get back to you anyway.
>
> Thanks again,
> Trustin
>
> Sébastien Pierre wrote:
> > Hello !
> >
> > So I did the comparison between Netty and HttpCore/NIO Netty, and I can
> > confirm the behaviour I saw before:
> >
> > - *Netty produces more errors than HttpCore/NIO*, dropping connections
> when
> > concurency is between 3000 and 5000
> > - HttpCore/NIO is slighly less performant, but significantly less errors
> > overall
> >
> > I used the stock Netty HTTP snoop example (see previous message in this
> > thread) and a modified version of the NHttpServer example of HttpCore
> > (attached). To compiled it and run it, do the following in HttpCore's
> > repository:
> >
> > javac -cp
> >
> httpcore/target/httpcore-4.1-alpha2-SNAPSHOT.jar:./httpcore-nio/target/httpcore-nio-4.1-alpha2-SNAPSHOT.jar
> > NHttpServer.java
> > java -cp
> >
> .:httpcore/target/httpcore-4.1-alpha2-SNAPSHOT.jar:./httpcore-nio/target/httpcore-nio-4.1-alpha2-SNAPSHOT.jar
> > NHttpServer
> >
> > The test I run is the httperf-based benchmark. as presented earlier. I
> also
> > started by doing multiple 'ab' benchmarks to see if connections errors
> may
> > appear, but I didn't get any. I still don't know if the errors are
> related
> > to improper system configuration, but among the various tests I've run,
> > I've seen Netty drop connections and become non-reactive more often than
> > HttpCore/NIO.
> >
> > I also attached the HttpCore/NIO test case as well sa graphs illustrating
> > the test results or running httperf on both.
> >
> > Cheers,
> >
> > -- Sébastien
> >
> >
> > Le 7 janvier 2010 13:16, Sébastien Pierre <sebastien.pierre at gmail.com> a
> > écrit :
> >
> >> Hi Trustin,
> >>
> >> So I checked the number of sockets in *_WAIT state on both client and
> >> server machines, using the following commands
> >>
> >> server> watch -n1 'lsof -nl | egrep "TCP|UDP" | grep "java" | wc -l '
> >> client> watch -n1 'lsof -nl | grep WAIT | wc -l'
> >>
> >> the number of opened sockets on the server oscillates between 0 and 500,
> >> while the number of *WAIT sockets on the client is always 0, so I doubt
> it's
> >> related to an overflow of TIME_WAIT or CLOSE_WAIT.
> >>
> >> I'll prepare more detailed tests with the HttpCore/NIO version.
> >>
> >> -- Sébastien
> >>
> >> 2010/1/6 "Trustin Lee (이희승)" <trustin at gmail.com>
> >>
> >>> Hi Sébastien,
> >>>
> >>>
> >>> You might be running out of available ports due to many TIME_WAIT
> either
> >>> on the client or server side. Could you confirm it? If so, you need
> to
> >>> wait until the TIME_WAIT state is cleared between each run.
> >>>
> >>> If HttpCore/NIO version works fine when Netty fails, I'd like to
> compare
> >>> the two to see what the differences are. Please paste or attach the
> >>> source code.
> >>>
> >>> Thanks,
> >>> Trustin
> >>>
> >>> Sébastien Pierre wrote:
> >>>> Hi there !
> >>>>
> >>>> I just moved a test HTTP service from HttpCore/NIO to Netty based on
> the
> >>>> HTTP snoop example. I experienced (despite good performance at first)
> >>> that
> >>>> the server is dropping connections when the load is heavy. To
> reproduce
> >>>> this, simply start the snoop example:
> >>>>
> >>>> java -cp ./src/main/java/:./jar/netty-3.2.0.ALPHA2.jar
> >>>> org.jboss.netty.example.http.snoop.HttpServer
> >>>>
> >>>> and then do
> >>>>
> >>>> python -c'for r in range(500,10500,500): import os ;
> os.system("httperf
> >>>> --hog --timeout=60 --client=0/1 --server=127.0.0.1 --port=8080 --uri=/
> >>>> --rate=%s --send-buffer=4096 --recv-buffer=16384 --num-conns=10000
> >>>> --num-calls=1" % (r))'
> >>>>
> >>>> which is the same as running this command with --rate growing from 500
> >>> to
> >>>> 10000 with 500 increases.
> >>>>
> >>>> httperf --hog --timeout=60 --client=0/1 --server=127.0.0.1 --port=8080
> >>>> --uri=/ --rate=500 --send-buffer=4096 --recv-buffer=16384
> >>> --num-conns=10000
> >>>> --num-calls=1
> >>>>
> >>>> so in this example, Netty often drops connections between rates of
> 3000
> >>> and
> >>>> 5000, which you'll see in httpperf's log with the following lines:
> >>>>
> >>>> httperf: connection failed with unexpected error 98
> >>>> ...
> >>>> Errors: total 2236 client-timo 0 socket-timo 0 connrefused 0 connreset
> 0
> >>>> Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 2236
> >>>>
> >>>> I attached a graph where you'll see that for the rate=4000 and
> >>> rate=45000
> >>>> the server just dropped all the connections (from httperf's
> perspective
> >>> at
> >>>> least). So do you know any way to prevent or at least detect that ?
> >>>>
> >>>> Thanks !
> >>>>
> >>>> -- Sébastien
> >>>>
> >>>> PS: I should add that I've run the test a couple of times, and quite
> >>> often
> >>>> have trouble in the rate=3000-5000 range
> >>>>
> >>>>
> >>>>
> >>>>
> ------------------------------------------------------------------------
> >>>>
> >>>> _______________________________________________
> >>>> netty-users mailing list
> >>>> netty-users at lists.jboss.org
> >>>> https://lists.jboss.org/mailman/listinfo/netty-users
> >>> --
> >>> what we call human nature in actuality is human habit
> >>> http://gleamynode.net/
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> netty-users mailing list
> >>> netty-users at lists.jboss.org
> >>> https://lists.jboss.org/mailman/listinfo/netty-users
> >>>
> >>>
> >
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > netty-users mailing list
> > netty-users at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/netty-users
>
> --
> what we call human nature in actuality is human habit
> http://gleamynode.net/
>
>
>
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/netty-users/attachments/20100109/3538d795/attachment.html
More information about the netty-users
mailing list