Hi Trustin,<div><br></div><div>Yes, ab does not stress the system as much as httperf, which I actually compiled according to this article:</div><div><br></div><div> <a href="http://gom-jabbar.org/articles/2009/02/04/httperf-and-file-descriptors">http://gom-jabbar.org/articles/2009/02/04/httperf-and-file-descriptors</a></div>
<div><br></div><div>httperf will often return error code "98", which is, according to errno.h is</div><div><br></div><div><div> #define EADDRINUSE 98 /* Address already in use */</div><div><br></div>
<div>
which doesn't make much sense. I also run the same test on a linode slice, theand did not experiment any problem around 3K connections, but there are still conncetion resets.</div><div><br></div><div> -- Sébastien</div>
<div><br></div><br><div class="gmail_quote">2010/1/8 "Trustin Lee (이희승)" <span dir="ltr"><<a href="mailto:trustin@gmail.com">trustin@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Hi,<br>
<br>
Thank you very much for further investigation!<br>
<br>
It is interesting that you don't get any error with ab. ab might not be<br>
fast enough or httpperf has different behavior.<br>
<br>
In your previous message:<br>
<div class="im"><br>
>>>> Errors: total 2236 client-timo 0 socket-timo 0 connrefused 0<br>
connreset 0<br>
>>>> Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 2236<br>
<br>
</div>could you figure out what the 'other' causes are? Are they simply<br>
'connection reset by peer' error, or something else?<br>
<br>
I will run the test by myself soon and get back to you anyway.<br>
<br>
Thanks again,<br>
<font color="#888888">Trustin<br>
</font><div><div></div><div class="h5"><br>
Sébastien Pierre wrote:<br>
> Hello !<br>
><br>
> So I did the comparison between Netty and HttpCore/NIO Netty, and I can<br>
> confirm the behaviour I saw before:<br>
><br>
> - *Netty produces more errors than HttpCore/NIO*, dropping connections when<br>
> concurency is between 3000 and 5000<br>
> - HttpCore/NIO is slighly less performant, but significantly less errors<br>
> overall<br>
><br>
> I used the stock Netty HTTP snoop example (see previous message in this<br>
> thread) and a modified version of the NHttpServer example of HttpCore<br>
> (attached). To compiled it and run it, do the following in HttpCore's<br>
> repository:<br>
><br>
> javac -cp<br>
> httpcore/target/httpcore-4.1-alpha2-SNAPSHOT.jar:./httpcore-nio/target/httpcore-nio-4.1-alpha2-SNAPSHOT.jar<br>
> NHttpServer.java<br>
> java -cp<br>
> .:httpcore/target/httpcore-4.1-alpha2-SNAPSHOT.jar:./httpcore-nio/target/httpcore-nio-4.1-alpha2-SNAPSHOT.jar<br>
> NHttpServer<br>
><br>
> The test I run is the httperf-based benchmark. as presented earlier. I also<br>
> started by doing multiple 'ab' benchmarks to see if connections errors may<br>
> appear, but I didn't get any. I still don't know if the errors are related<br>
> to improper system configuration, but among the various tests I've run,<br>
> I've seen Netty drop connections and become non-reactive more often than<br>
> HttpCore/NIO.<br>
><br>
> I also attached the HttpCore/NIO test case as well sa graphs illustrating<br>
> the test results or running httperf on both.<br>
><br>
> Cheers,<br>
><br>
> -- Sébastien<br>
><br>
><br>
> Le 7 janvier 2010 13:16, Sébastien Pierre <<a href="mailto:sebastien.pierre@gmail.com">sebastien.pierre@gmail.com</a>> a<br>
> écrit :<br>
><br>
>> Hi Trustin,<br>
>><br>
>> So I checked the number of sockets in *_WAIT state on both client and<br>
>> server machines, using the following commands<br>
>><br>
>> server> watch -n1 'lsof -nl | egrep "TCP|UDP" | grep "java" | wc -l '<br>
>> client> watch -n1 'lsof -nl | grep WAIT | wc -l'<br>
>><br>
>> the number of opened sockets on the server oscillates between 0 and 500,<br>
>> while the number of *WAIT sockets on the client is always 0, so I doubt it's<br>
>> related to an overflow of TIME_WAIT or CLOSE_WAIT.<br>
>><br>
>> I'll prepare more detailed tests with the HttpCore/NIO version.<br>
>><br>
>> -- Sébastien<br>
>><br>
>> 2010/1/6 "Trustin Lee (이희승)" <<a href="mailto:trustin@gmail.com">trustin@gmail.com</a>><br>
>><br>
>>> Hi Sébastien,<br>
>>><br>
>>><br>
>>> You might be running out of available ports due to many TIME_WAIT either<br>
>>> on the client or server side. Could you confirm it? If so, you need to<br>
>>> wait until the TIME_WAIT state is cleared between each run.<br>
>>><br>
>>> If HttpCore/NIO version works fine when Netty fails, I'd like to compare<br>
>>> the two to see what the differences are. Please paste or attach the<br>
>>> source code.<br>
>>><br>
>>> Thanks,<br>
>>> Trustin<br>
>>><br>
>>> Sébastien Pierre wrote:<br>
>>>> Hi there !<br>
>>>><br>
>>>> I just moved a test HTTP service from HttpCore/NIO to Netty based on the<br>
>>>> HTTP snoop example. I experienced (despite good performance at first)<br>
>>> that<br>
>>>> the server is dropping connections when the load is heavy. To reproduce<br>
>>>> this, simply start the snoop example:<br>
>>>><br>
>>>> java -cp ./src/main/java/:./jar/netty-3.2.0.ALPHA2.jar<br>
>>>> org.jboss.netty.example.http.snoop.HttpServer<br>
>>>><br>
>>>> and then do<br>
>>>><br>
>>>> python -c'for r in range(500,10500,500): import os ; os.system("httperf<br>
>>>> --hog --timeout=60 --client=0/1 --server=127.0.0.1 --port=8080 --uri=/<br>
>>>> --rate=%s --send-buffer=4096 --recv-buffer=16384 --num-conns=10000<br>
>>>> --num-calls=1" % (r))'<br>
>>>><br>
>>>> which is the same as running this command with --rate growing from 500<br>
>>> to<br>
>>>> 10000 with 500 increases.<br>
>>>><br>
>>>> httperf --hog --timeout=60 --client=0/1 --server=127.0.0.1 --port=8080<br>
>>>> --uri=/ --rate=500 --send-buffer=4096 --recv-buffer=16384<br>
>>> --num-conns=10000<br>
>>>> --num-calls=1<br>
>>>><br>
>>>> so in this example, Netty often drops connections between rates of 3000<br>
>>> and<br>
>>>> 5000, which you'll see in httpperf's log with the following lines:<br>
>>>><br>
>>>> httperf: connection failed with unexpected error 98<br>
>>>> ...<br>
>>>> Errors: total 2236 client-timo 0 socket-timo 0 connrefused 0 connreset 0<br>
>>>> Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 2236<br>
>>>><br>
>>>> I attached a graph where you'll see that for the rate=4000 and<br>
>>> rate=45000<br>
>>>> the server just dropped all the connections (from httperf's perspective<br>
>>> at<br>
>>>> least). So do you know any way to prevent or at least detect that ?<br>
>>>><br>
>>>> Thanks !<br>
>>>><br>
>>>> -- Sébastien<br>
>>>><br>
>>>> PS: I should add that I've run the test a couple of times, and quite<br>
>>> often<br>
>>>> have trouble in the rate=3000-5000 range<br>
>>>><br>
>>>><br>
>>>><br>
>>>> ------------------------------------------------------------------------<br>
>>>><br>
>>>> _______________________________________________<br>
>>>> netty-users mailing list<br>
>>>> <a href="mailto:netty-users@lists.jboss.org">netty-users@lists.jboss.org</a><br>
>>>> <a href="https://lists.jboss.org/mailman/listinfo/netty-users" target="_blank">https://lists.jboss.org/mailman/listinfo/netty-users</a><br>
>>> --<br>
>>> what we call human nature in actuality is human habit<br>
>>> <a href="http://gleamynode.net/" target="_blank">http://gleamynode.net/</a><br>
>>><br>
>>><br>
>>><br>
>>> _______________________________________________<br>
>>> netty-users mailing list<br>
>>> <a href="mailto:netty-users@lists.jboss.org">netty-users@lists.jboss.org</a><br>
>>> <a href="https://lists.jboss.org/mailman/listinfo/netty-users" target="_blank">https://lists.jboss.org/mailman/listinfo/netty-users</a><br>
>>><br>
>>><br>
><br>
><br>
> ------------------------------------------------------------------------<br>
><br>
> _______________________________________________<br>
> netty-users mailing list<br>
> <a href="mailto:netty-users@lists.jboss.org">netty-users@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/netty-users" target="_blank">https://lists.jboss.org/mailman/listinfo/netty-users</a><br>
<br>
--<br>
what we call human nature in actuality is human habit<br>
<a href="http://gleamynode.net/" target="_blank">http://gleamynode.net/</a><br>
<br>
<br>
</div></div><br>_______________________________________________<br>
netty-users mailing list<br>
<a href="mailto:netty-users@lists.jboss.org">netty-users@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/netty-users" target="_blank">https://lists.jboss.org/mailman/listinfo/netty-users</a><br>
<br></blockquote></div><br></div>