<div>Hi Trustin,</div>
<div> </div>
<div>Thanks for looking into it. The problem most likely due to my local server not configured to handle this many HTTP requests. After I increased maximum threads for my Tomcat, the problem was gone. Maybe that's how your server is configured.</div>
<div> </div>
<div>As you advised, I'll look into reusing connections, or something like connection pooling/management.</div>
<div>For the lock-up issue, not sure what it is. Please give me more lead if you get a chance.</div>
<div>About the large thread pool in Requestor, if that's what you were referring, it is used to simulate concurrent traffic loads/requests into our servlet which could use Netty to retrieve online info from multiple Internet sources. The application itself does not necessarily need to do multi-threading.</div>
<div> </div>
<div>Thanks again,</div>
<div>Jason<br></div>
<div class="gmail_quote">On Mon, Sep 21, 2009 at 6:16 PM, Trustin Lee (ÀÌÈñ½Â) <span dir="ltr"><<a href="mailto:trustin@gmail.com">trustin@gmail.com</a>></span> wrote:<br>
<blockquote style="BORDER-LEFT: #ccc 1px solid; MARGIN: 0px 0px 0px 0.8ex; PADDING-LEFT: 1ex" class="gmail_quote">Hi Jason,<br><br>I ran your code by myself, and I was not able to reproduce the problem<br>even after increasing the parameters. I did not try public servers<br>
though as I don't want to overload them.<br><br>According to your code, you make a new connection attempt for each<br>request. You will have a lot of ports with 'TIME_WAIT' state, which<br>slow down the connection rate. I'd recommend to keep the connection<br>
alive and reuse it over and over (i.e. HTTP Keep-Ali<br><br>On the other hand, I see some lock-up in your application during the<br>test, which keeps me from completing the test, but I did not<br>investigate further because it seemed irrelevant to the original<br>
problem. Actually, you don't even need a large thread pool to run<br>this test. You can perform the same job with only one thread,<br>resulting higher throughput, because everything is asynchronous in<br>Netty. :)<br>
<br>HTH,<br>
<div class="im"><br>— Trustin Lee, <a href="http://gleamynode.net/" target="_blank">http://gleamynode.net/</a><br><br></div>
<div>
<div></div>
<div class="h5">On Fri, Sep 18, 2009 at 5:10 PM, J. Mi <<a href="mailto:jmi258@gmail.com">jmi258@gmail.com</a>> wrote:<br>> Trustin,<br>><br>> Yes, my problem can be reproduced consistently. Attached are 4 java files.<br>
> Among them, HttpClientPipelineFactory.java and HttpResponseHandler.java are<br>> almost unchanged from their example sibling of snoop, except that I enabled<br>> HttpChunkAggregator in HttpClientPipelineFactory and offered answer in<br>
> HttpResponseHandler to be retrieved.<br>><br>> I converted the original HttpClient to ConcurrentHttpClient to concurrently<br>> send http requests to any server. I tried to achieve maximum concurrency by<br>
> adding ChannelFutureListener when connecting (see line 70). It's supposed<br>> to work better for me. But if I used the commented-out code below instead. I<br>> can run more connections.<br>><br>> Finally, Requestor is the test driver (into ConcurrentHttpClient) with<br>
> multi-threading to observe how Netty scales under heavy loads.<br>> To run the program:<br>>>java -cp %CLASSPATH% myperfeval.http.Requestor 80 10 <a href="http://www.yahoo.com/" target="_blank">http://www.yahoo.com</a><br>
><br>> where 80 is for n threads spawn from Requestor and 10 is the number of Http<br>> servers that ConcurrentHttpClient is to hit for each thread. So in this<br>> case, it could reach 800 connections at a time. For the test, you can either<br>
> use a public server like yahoo, or a local test server available to you, as<br>> the thrid argument. I found that with yahoo, I tend to run into timeout<br>> trouble. And with a local test server, I'll get the connection exception as<br>
> in the original email. You can play with the first two arguments to narrow<br>> it down when it starts to trouble your system. Requestor can run a little<br>> more connections in Windows than VMware with Centso 5.3.<br>
><br>> I suspect that somehow ConcurrentHttpClient might not do things right.<br>><br>> Thanks a lot for your time!<br>> Jason<br>><br>><br>> On Thu, Sep 17, 2009 at 7:36 PM, Trustin Lee (ÀÌÈñ½Â) <<a href="mailto:trustin@gmail.com">trustin@gmail.com</a>><br>
> wrote:<br>>><br>>> Sorry about the misleading post. I was merely answering Mike's questions.<br>>><br>>> I have not experienced such a problem so far in Linux. I am using<br>>> Fedora at the moment but I also use RHEL for performance tests.<br>
>><br>>> Does the connection attempt succeed when you retry or fail forever once<br>>> refused?<br>>><br>>> If all subsequent connection attempts are refused, it means the server<br>>> has been terminated for some reason. Otherwise, it's a temporary<br>
>> problem related with your operating system.<br>>><br>>> I'd like to investigate this problem further if there's a way to<br>>> reproduce the problem reliably. Could you post a simple client-server<br>
>> test application so that I can run it by myself? A simple modification<br>>> of the official examples would be perfect.<br>>><br>>> Thanks<br>>><br>>> — Trustin Lee, <a href="http://gleamynode.net/" target="_blank">http://gleamynode.net/</a><br>
>><br>>> On Fri, Sep 18, 2009 at 10:21 AM, J. Mi <<a href="mailto:jmi258@gmail.com">jmi258@gmail.com</a>> wrote:<br>>> > Trustin,<br>>> ><br>>> > I already tried that. I tried to increase 'nofile' to 90000 both in<br>
>> > /etc/security/limits.conf (and restart my Centos in VMware) and on the<br>>> > fly<br>>> > for a perticular termal session by 'ulimit -n 90000' command. None of<br>>> > them<br>
>> > work for me.<br>>> ><br>>> > The exception I got was "java.net.ConnectException: Connection refused:<br>>> > no<br>>> > further information..", not "Too many open files..." althouth I seem to<br>
>> > hear<br>>> > that file descriptors, connections and thread might share this same<br>>> > limit.<br>>> ><br>>> > This problem blocked me for days. And it doesn't seem to be the limit on<br>
>> > http server side because I tried to use the same Netty http client to<br>>> > hit<br>>> > differnt server and had the same problem.<br>>> ><br>>> > Thanks for your help. Appreciated it.<br>
>> ><br>>> > Jason<br>>> ><br>>> > On Thu, Sep 17, 2009 at 5:06 PM, Trustin Lee (ÀÌÈñ½Â) <<a href="mailto:trustin@gmail.com">trustin@gmail.com</a>><br>>> > wrote:<br>>> >><br>
>> >> I hope this post answers the first question.<br>>> >><br>>> >> <a href="http://gleamynode.net/articles/1557/" target="_blank">http://gleamynode.net/articles/1557/</a><br>>> >><br>
>> >> I'm not sure about the other questions though. :)<br>>> >><br>>> >> — Trustin Lee, <a href="http://gleamynode.net/" target="_blank">http://gleamynode.net/</a><br>>> >><br>
>> >><br>>> >><br>>> >> On Fri, Sep 18, 2009 at 2:37 AM, Michael McGrady<br>>> >> <<a href="mailto:mmcgrady@topiatechnology.com">mmcgrady@topiatechnology.com</a>> wrote:<br>
>> >> > What is the impact of changing these settings? Why is the default<br>>> >> > low<br>>> >> > instead of hight? And, any other questions I should be asking before<br>>> >> > changing this configuration?<br>
>> >> > MIke<br>>> >> > On Sep 17, 2009, at 10:34 AM, J. Mi wrote:<br>>> >> ><br>>> >> > H Thomas,<br>>> >> ><br>>> >> > Thanks for the lead. To be able to have more connections before<br>
>> >> > seeing<br>>> >> > "java.net.ConnectException: Connection refused...", which<br>>> >> > configuration<br>>> >> > I<br>>> >> > need to change in /etc/security/limits.conf? I added<br>
>> >> > * soft nofile 20000<br>>> >> > * hard nofile 20000<br>>> >> ><br>>> >> > and first time just tried to re-login, the new limit didn't work for<br>>> >> > me<br>
>> >> > for<br>>> >> > more connections. I then tried to restart my VMware. It still didn't<br>>> >> > work<br>>> >> > for me<br>>> >> ><br>>> >> > Does 'nofile' (max number of open files) have anything to do with<br>
>> >> > socket<br>>> >> > connections?<br>>> >> ><br>>> >> > Appreciate if you or anyone else can guide me more on this.<br>>> >> > btw, if any of you know any equivalent OS settings on Windows, i'll<br>
>> >> > try<br>>> >> > that<br>>> >> > as well.<br>>> >> ><br>>> >> > Jason<br>>> >> ><br>>> >> ><br>>> >> > On Wed, Sep 16, 2009 at 3:38 PM, Thomas Bocek <<a href="mailto:bocek@ifi.uzh.ch">bocek@ifi.uzh.ch</a>><br>
>> >> > wrote:<br>>> >> >><br>>> >> >> Hi J.,<br>>> >> >><br>>> >> >> This might be a limitation of the OS. If you are using for example<br>
>> >> >> Linux, then you can only open about 1000 connections before seeing<br>>> >> >> "too<br>>> >> >> many open files in system" error messages. You can adjust the value<br>
>> >> >> in<br>>> >> >> /etc/security/limits.conf<br>>> >> >><br>>> >> >> Thomas<br>>> >> >><br>>> >> >> J. Mi wrote:<br>
>> >> >> > All,<br>>> >> >> ><br>>> >> >> > I get following exception pretty consistently when trying to<br>>> >> >> > concurrently<br>>> >> >> > connect about 800 connections.<br>
>> >> >> ><br>>> >> >> > Any idea? Is this something out of Netty's control? If so, is<br>>> >> >> > there<br>>> >> >> > some<br>>> >> >> > configuration I could do to JVM or operating system to increase<br>
>> >> >> > resource<br>>> >> >> > capacity?<br>>> >> >> ><br>>> >> >> > Thanks,<br>>> >> >> > Jason<br>>> >> >> ><br>
>> >> >> > java.net.ConnectException: Connection refused: no further<br>>> >> >> > information<br>>> >> >> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native<br>
>> >> >> > Method)<br>>> >> >> > at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown<br>>> >> >> > Source)<br>>> >> >> > at<br>
>> >> >> ><br>>> >> >> ><br>>> >> >> ><br>>> >> >> > org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:381)<br>
>> >> >> > at<br>>> >> >> ><br>>> >> >> ><br>>> >> >> ><br>>> >> >> > org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:351)<br>
>> >> >> > at<br>>> >> >> ><br>>> >> >> ><br>>> >> >> ><br>>> >> >> > org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:273)<br>
>> >> >> > at<br>>> >> >> ><br>>> >> >> ><br>>> >> >> ><br>>> >> >> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:113)<br>
>> >> >> > at<br>>> >> >> ><br>>> >> >> ><br>>> >> >> ><br>>> >> >> > org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:53)<br>
>> >> >> > at<br>>> >> >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown<br>>> >> >> > Source)<br>>> >> >> > at<br>
>> >> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown<br>>> >> >> > Source)<br>>> >> >> > at java.lang.Thread.run(Unknown Source)<br>>> >> >> ><br>
>> >> >> ><br>>> >> >> ><br>>> >> >> ><br>>> >> >> ><br>>> >> >> > ------------------------------------------------------------------------<br>
>> >> >> ><br>>> >> >> > _______________________________________________<br>>> >> >> > netty-dev mailing list<br>>> >> >> > <a href="mailto:netty-dev@lists.jboss.org">netty-dev@lists.jboss.org</a><br>
>> >> >> > <a href="https://lists.jboss.org/mailman/listinfo/netty-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/netty-dev</a><br>>> >> >> _______________________________________________<br>
>> >> >> netty-dev mailing list<br>>> >> >> <a href="mailto:netty-dev@lists.jboss.org">netty-dev@lists.jboss.org</a><br>>> >> >> <a href="https://lists.jboss.org/mailman/listinfo/netty-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/netty-dev</a><br>
>> >> ><br>>> >> > _______________________________________________<br>>> >> > netty-dev mailing list<br>>> >> > <a href="mailto:netty-dev@lists.jboss.org">netty-dev@lists.jboss.org</a><br>
>> >> > <a href="https://lists.jboss.org/mailman/listinfo/netty-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/netty-dev</a><br>>> >> ><br>>> >> > Mike McGrady<br>
>> >> > Principal Investigator AF081-028 AFRL SBIR<br>>> >> > Senior Engineer<br>>> >> > Topia Technology, Inc<br>>> >> > 1.253.720.3365<br>>> >> > <a href="mailto:mmcgrady@topiatechnology.com">mmcgrady@topiatechnology.com</a><br>
>> >> ><br>>> >> ><br>>> >> ><br>>> >> ><br>>> >> ><br>>> >> ><br>>> >> ><br>>> >> > _______________________________________________<br>
>> >> > netty-dev mailing list<br>>> >> > <a href="mailto:netty-dev@lists.jboss.org">netty-dev@lists.jboss.org</a><br>>> >> > <a href="https://lists.jboss.org/mailman/listinfo/netty-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/netty-dev</a><br>
>> >> ><br>>> >> ><br>>> >><br>>> >> _______________________________________________<br>>> >> netty-dev mailing list<br>>> >> <a href="mailto:netty-dev@lists.jboss.org">netty-dev@lists.jboss.org</a><br>
>> >> <a href="https://lists.jboss.org/mailman/listinfo/netty-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/netty-dev</a><br>>> ><br>>> ><br>>> > _______________________________________________<br>
>> > netty-dev mailing list<br>>> > <a href="mailto:netty-dev@lists.jboss.org">netty-dev@lists.jboss.org</a><br>>> > <a href="https://lists.jboss.org/mailman/listinfo/netty-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/netty-dev</a><br>
>> ><br>>> ><br>>><br>>> _______________________________________________<br>>> netty-dev mailing list<br>>> <a href="mailto:netty-dev@lists.jboss.org">netty-dev@lists.jboss.org</a><br>
>> <a href="https://lists.jboss.org/mailman/listinfo/netty-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/netty-dev</a><br>><br>><br>> _______________________________________________<br>> netty-dev mailing list<br>
> <a href="mailto:netty-dev@lists.jboss.org">netty-dev@lists.jboss.org</a><br>> <a href="https://lists.jboss.org/mailman/listinfo/netty-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/netty-dev</a><br>
><br>><br><br>_______________________________________________<br>netty-dev mailing list<br><a href="mailto:netty-dev@lists.jboss.org">netty-dev@lists.jboss.org</a><br><a href="https://lists.jboss.org/mailman/listinfo/netty-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/netty-dev</a><br>
</div></div></blockquote></div><br>