<div>Trustin,</div>
<div> </div>
<div>Yes, my problem can be reproduced consistently. Attached are 4 java files. Among them, HttpClientPipelineFactory.java and HttpResponseHandler.java are almost unchanged from their example sibling of snoop, except that I enabled HttpChunkAggregator in HttpClientPipelineFactory and offered answer in HttpResponseHandler to be retrieved.</div>
<div> </div>
<div>I converted the original HttpClient to ConcurrentHttpClient to concurrently send http requests to any server. I tried to achieve maximum concurrency by adding ChannelFutureListener when connecting (see line 70). It's supposed to work better for me. But if I used the commented-out code below instead. I can run more connections.</div>
<div> </div>
<div>Finally, Requestor is the test driver (into ConcurrentHttpClient) with multi-threading to observe how Netty scales under heavy loads.</div>
<div>To run the program:<br>>java -cp %CLASSPATH% myperfeval.http.Requestor 80 10 <a href="http://www.yahoo.com/">http://www.yahoo.com</a></div>
<div> </div>
<div>where 80 is for n threads spawn from Requestor and 10 is the number of Http servers that ConcurrentHttpClient is to hit for each thread. So in this case, it could reach 800 connections at a time. For the test, you can either use a public server like yahoo, or a local test server available to you, as the thrid argument. I found that with yahoo, I tend to run into timeout trouble. And with a local test server, I'll get the connection exception as in the original email. You can play with the first two arguments to narrow it down when it starts to trouble your system. Requestor can run a little more connections in Windows than VMware with Centso 5.3.</div>
<div> </div>
<div>I suspect that somehow ConcurrentHttpClient might not do things right.</div>
<div> </div>
<div>Thanks a lot for your time!</div>
<div>Jason</div>
<div> </div>
<div> </div>
<div class="gmail_quote">On Thu, Sep 17, 2009 at 7:36 PM, Trustin Lee (이희승) <span dir="ltr"><<a href="mailto:trustin@gmail.com">trustin@gmail.com</a>></span> wrote:<br>
<blockquote style="BORDER-LEFT: #ccc 1px solid; MARGIN: 0px 0px 0px 0.8ex; PADDING-LEFT: 1ex" class="gmail_quote">Sorry about the misleading post. I was merely answering Mike's questions.<br><br>I have not experienced such a problem so far in Linux. I am using<br>
Fedora at the moment but I also use RHEL for performance tests.<br><br>Does the connection attempt succeed when you retry or fail forever once refused?<br><br>If all subsequent connection attempts are refused, it means the server<br>
has been terminated for some reason. Otherwise, it's a temporary<br>problem related with your operating system.<br><br>I'd like to investigate this problem further if there's a way to<br>reproduce the problem reliably. Could you post a simple client-server<br>
test application so that I can run it by myself? A simple modification<br>of the official examples would be perfect.<br><br>Thanks<br>
<div class="im"><br>— Trustin Lee, <a href="http://gleamynode.net/" target="_blank">http://gleamynode.net/</a><br><br></div>
<div>
<div></div>
<div class="h5">On Fri, Sep 18, 2009 at 10:21 AM, J. Mi <<a href="mailto:jmi258@gmail.com">jmi258@gmail.com</a>> wrote:<br>> Trustin,<br>><br>> I already tried that. I tried to increase 'nofile' to 90000 both in<br>
> /etc/security/limits.conf (and restart my Centos in VMware) and on the fly<br>> for a perticular termal session by 'ulimit -n 90000' command. None of them<br>> work for me.<br>><br>> The exception I got was "java.net.ConnectException: Connection refused: no<br>
> further information..", not "Too many open files..." althouth I seem to hear<br>> that file descriptors, connections and thread might share this same limit.<br>><br>> This problem blocked me for days. And it doesn't seem to be the limit on<br>
> http server side because I tried to use the same Netty http client to hit<br>> differnt server and had the same problem.<br>><br>> Thanks for your help. Appreciated it.<br>><br>> Jason<br>><br>> On Thu, Sep 17, 2009 at 5:06 PM, Trustin Lee (이희승) <<a href="mailto:trustin@gmail.com">trustin@gmail.com</a>><br>
> wrote:<br>>><br>>> I hope this post answers the first question.<br>>><br>>> <a href="http://gleamynode.net/articles/1557/" target="_blank">http://gleamynode.net/articles/1557/</a><br>>><br>
>> I'm not sure about the other questions though. :)<br>>><br>>> — Trustin Lee, <a href="http://gleamynode.net/" target="_blank">http://gleamynode.net/</a><br>>><br>>><br>>><br>>> On Fri, Sep 18, 2009 at 2:37 AM, Michael McGrady<br>
>> <<a href="mailto:mmcgrady@topiatechnology.com">mmcgrady@topiatechnology.com</a>> wrote:<br>>> > What is the impact of changing these settings? Why is the default low<br>>> > instead of hight? And, any other questions I should be asking before<br>
>> > changing this configuration?<br>>> > MIke<br>>> > On Sep 17, 2009, at 10:34 AM, J. Mi wrote:<br>>> ><br>>> > H Thomas,<br>>> ><br>>> > Thanks for the lead. To be able to have more connections before seeing<br>
>> > "java.net.ConnectException: Connection refused...", which configuration<br>>> > I<br>>> > need to change in /etc/security/limits.conf? I added<br>>> > * soft nofile 20000<br>
>> > * hard nofile 20000<br>>> ><br>>> > and first time just tried to re-login, the new limit didn't work for me<br>>> > for<br>>> > more connections. I then tried to restart my VMware. It still didn't<br>
>> > work<br>>> > for me<br>>> ><br>>> > Does 'nofile' (max number of open files) have anything to do with socket<br>>> > connections?<br>>> ><br>>> > Appreciate if you or anyone else can guide me more on this.<br>
>> > btw, if any of you know any equivalent OS settings on Windows, i'll try<br>>> > that<br>>> > as well.<br>>> ><br>>> > Jason<br>>> ><br>>> ><br>>> > On Wed, Sep 16, 2009 at 3:38 PM, Thomas Bocek <<a href="mailto:bocek@ifi.uzh.ch">bocek@ifi.uzh.ch</a>> wrote:<br>
>> >><br>>> >> Hi J.,<br>>> >><br>>> >> This might be a limitation of the OS. If you are using for example<br>>> >> Linux, then you can only open about 1000 connections before seeing "too<br>
>> >> many open files in system" error messages. You can adjust the value in<br>>> >> /etc/security/limits.conf<br>>> >><br>>> >> Thomas<br>>> >><br>>> >> J. Mi wrote:<br>
>> >> > All,<br>>> >> ><br>>> >> > I get following exception pretty consistently when trying to<br>>> >> > concurrently<br>>> >> > connect about 800 connections.<br>
>> >> ><br>>> >> > Any idea? Is this something out of Netty's control? If so, is there<br>>> >> > some<br>>> >> > configuration I could do to JVM or operating system to increase<br>
>> >> > resource<br>>> >> > capacity?<br>>> >> ><br>>> >> > Thanks,<br>>> >> > Jason<br>>> >> ><br>>> >> > java.net.ConnectException: Connection refused: no further information<br>
>> >> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)<br>>> >> > at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)<br>>> >> > at<br>
>> >> ><br>>> >> ><br>>> >> > org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:381)<br>>> >> > at<br>
>> >> ><br>>> >> ><br>>> >> > org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:351)<br>>> >> > at<br>
>> >> ><br>>> >> ><br>>> >> > org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:273)<br>>> >> > at<br>
>> >> ><br>>> >> ><br>>> >> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:113)<br>>> >> > at<br>>> >> ><br>
>> >> ><br>>> >> > org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:53)<br>>> >> > at<br>>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown<br>
>> >> > Source)<br>>> >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown<br>>> >> > Source)<br>>> >> > at java.lang.Thread.run(Unknown Source)<br>
>> >> ><br>>> >> ><br>>> >> ><br>>> >> ><br>>> >> > ------------------------------------------------------------------------<br>>> >> ><br>
>> >> > _______________________________________________<br>>> >> > netty-dev mailing list<br>>> >> > <a href="mailto:netty-dev@lists.jboss.org">netty-dev@lists.jboss.org</a><br>
>> >> > <a href="https://lists.jboss.org/mailman/listinfo/netty-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/netty-dev</a><br>>> >> _______________________________________________<br>
>> >> netty-dev mailing list<br>>> >> <a href="mailto:netty-dev@lists.jboss.org">netty-dev@lists.jboss.org</a><br>>> >> <a href="https://lists.jboss.org/mailman/listinfo/netty-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/netty-dev</a><br>
>> ><br>>> > _______________________________________________<br>>> > netty-dev mailing list<br>>> > <a href="mailto:netty-dev@lists.jboss.org">netty-dev@lists.jboss.org</a><br>>> > <a href="https://lists.jboss.org/mailman/listinfo/netty-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/netty-dev</a><br>
>> ><br>>> > Mike McGrady<br>>> > Principal Investigator AF081-028 AFRL SBIR<br>>> > Senior Engineer<br>>> > Topia Technology, Inc<br>>> > 1.253.720.3365<br>>> > <a href="mailto:mmcgrady@topiatechnology.com">mmcgrady@topiatechnology.com</a><br>
>> ><br>>> ><br>>> ><br>>> ><br>>> ><br>>> ><br>>> ><br>>> > _______________________________________________<br>>> > netty-dev mailing list<br>
>> > <a href="mailto:netty-dev@lists.jboss.org">netty-dev@lists.jboss.org</a><br>>> > <a href="https://lists.jboss.org/mailman/listinfo/netty-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/netty-dev</a><br>
>> ><br>>> ><br>>><br>>> _______________________________________________<br>>> netty-dev mailing list<br>>> <a href="mailto:netty-dev@lists.jboss.org">netty-dev@lists.jboss.org</a><br>
>> <a href="https://lists.jboss.org/mailman/listinfo/netty-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/netty-dev</a><br>><br>><br>> _______________________________________________<br>> netty-dev mailing list<br>
> <a href="mailto:netty-dev@lists.jboss.org">netty-dev@lists.jboss.org</a><br>> <a href="https://lists.jboss.org/mailman/listinfo/netty-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/netty-dev</a><br>
><br>><br><br>_______________________________________________<br>netty-dev mailing list<br><a href="mailto:netty-dev@lists.jboss.org">netty-dev@lists.jboss.org</a><br><a href="https://lists.jboss.org/mailman/listinfo/netty-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/netty-dev</a><br>
</div></div></blockquote></div><br>