java.net.ConnectException thrown
Trustin Lee (이희승)
trustin at gmail.com
Mon Sep 21 21:16:28 EDT 2009
Hi Jason,
I ran your code by myself, and I was not able to reproduce the problem
even after increasing the parameters. I did not try public servers
though as I don't want to overload them.
According to your code, you make a new connection attempt for each
request. You will have a lot of ports with 'TIME_WAIT' state, which
slow down the connection rate. I'd recommend to keep the connection
alive and reuse it over and over (i.e. HTTP Keep-Alive)
On the other hand, I see some lock-up in your application during the
test, which keeps me from completing the test, but I did not
investigate further because it seemed irrelevant to the original
problem. Actually, you don't even need a large thread pool to run
this test. You can perform the same job with only one thread,
resulting higher throughput, because everything is asynchronous in
Netty. :)
HTH,
— Trustin Lee, http://gleamynode.net/
On Fri, Sep 18, 2009 at 5:10 PM, J. Mi <jmi258 at gmail.com> wrote:
> Trustin,
>
> Yes, my problem can be reproduced consistently. Attached are 4 java files.
> Among them, HttpClientPipelineFactory.java and HttpResponseHandler.java are
> almost unchanged from their example sibling of snoop, except that I enabled
> HttpChunkAggregator in HttpClientPipelineFactory and offered answer in
> HttpResponseHandler to be retrieved.
>
> I converted the original HttpClient to ConcurrentHttpClient to concurrently
> send http requests to any server. I tried to achieve maximum concurrency by
> adding ChannelFutureListener when connecting (see line 70). It's supposed
> to work better for me. But if I used the commented-out code below instead. I
> can run more connections.
>
> Finally, Requestor is the test driver (into ConcurrentHttpClient) with
> multi-threading to observe how Netty scales under heavy loads.
> To run the program:
>>java -cp %CLASSPATH% myperfeval.http.Requestor 80 10 http://www.yahoo.com
>
> where 80 is for n threads spawn from Requestor and 10 is the number of Http
> servers that ConcurrentHttpClient is to hit for each thread. So in this
> case, it could reach 800 connections at a time. For the test, you can either
> use a public server like yahoo, or a local test server available to you, as
> the thrid argument. I found that with yahoo, I tend to run into timeout
> trouble. And with a local test server, I'll get the connection exception as
> in the original email. You can play with the first two arguments to narrow
> it down when it starts to trouble your system. Requestor can run a little
> more connections in Windows than VMware with Centso 5.3.
>
> I suspect that somehow ConcurrentHttpClient might not do things right.
>
> Thanks a lot for your time!
> Jason
>
>
> On Thu, Sep 17, 2009 at 7:36 PM, Trustin Lee (이희승) <trustin at gmail.com>
> wrote:
>>
>> Sorry about the misleading post. I was merely answering Mike's questions.
>>
>> I have not experienced such a problem so far in Linux. I am using
>> Fedora at the moment but I also use RHEL for performance tests.
>>
>> Does the connection attempt succeed when you retry or fail forever once
>> refused?
>>
>> If all subsequent connection attempts are refused, it means the server
>> has been terminated for some reason. Otherwise, it's a temporary
>> problem related with your operating system.
>>
>> I'd like to investigate this problem further if there's a way to
>> reproduce the problem reliably. Could you post a simple client-server
>> test application so that I can run it by myself? A simple modification
>> of the official examples would be perfect.
>>
>> Thanks
>>
>> — Trustin Lee, http://gleamynode.net/
>>
>> On Fri, Sep 18, 2009 at 10:21 AM, J. Mi <jmi258 at gmail.com> wrote:
>> > Trustin,
>> >
>> > I already tried that. I tried to increase 'nofile' to 90000 both in
>> > /etc/security/limits.conf (and restart my Centos in VMware) and on the
>> > fly
>> > for a perticular termal session by 'ulimit -n 90000' command. None of
>> > them
>> > work for me.
>> >
>> > The exception I got was "java.net.ConnectException: Connection refused:
>> > no
>> > further information..", not "Too many open files..." althouth I seem to
>> > hear
>> > that file descriptors, connections and thread might share this same
>> > limit.
>> >
>> > This problem blocked me for days. And it doesn't seem to be the limit on
>> > http server side because I tried to use the same Netty http client to
>> > hit
>> > differnt server and had the same problem.
>> >
>> > Thanks for your help. Appreciated it.
>> >
>> > Jason
>> >
>> > On Thu, Sep 17, 2009 at 5:06 PM, Trustin Lee (이희승) <trustin at gmail.com>
>> > wrote:
>> >>
>> >> I hope this post answers the first question.
>> >>
>> >> http://gleamynode.net/articles/1557/
>> >>
>> >> I'm not sure about the other questions though. :)
>> >>
>> >> — Trustin Lee, http://gleamynode.net/
>> >>
>> >>
>> >>
>> >> On Fri, Sep 18, 2009 at 2:37 AM, Michael McGrady
>> >> <mmcgrady at topiatechnology.com> wrote:
>> >> > What is the impact of changing these settings? Why is the default
>> >> > low
>> >> > instead of hight? And, any other questions I should be asking before
>> >> > changing this configuration?
>> >> > MIke
>> >> > On Sep 17, 2009, at 10:34 AM, J. Mi wrote:
>> >> >
>> >> > H Thomas,
>> >> >
>> >> > Thanks for the lead. To be able to have more connections before
>> >> > seeing
>> >> > "java.net.ConnectException: Connection refused...", which
>> >> > configuration
>> >> > I
>> >> > need to change in /etc/security/limits.conf? I added
>> >> > * soft nofile 20000
>> >> > * hard nofile 20000
>> >> >
>> >> > and first time just tried to re-login, the new limit didn't work for
>> >> > me
>> >> > for
>> >> > more connections. I then tried to restart my VMware. It still didn't
>> >> > work
>> >> > for me
>> >> >
>> >> > Does 'nofile' (max number of open files) have anything to do with
>> >> > socket
>> >> > connections?
>> >> >
>> >> > Appreciate if you or anyone else can guide me more on this.
>> >> > btw, if any of you know any equivalent OS settings on Windows, i'll
>> >> > try
>> >> > that
>> >> > as well.
>> >> >
>> >> > Jason
>> >> >
>> >> >
>> >> > On Wed, Sep 16, 2009 at 3:38 PM, Thomas Bocek <bocek at ifi.uzh.ch>
>> >> > wrote:
>> >> >>
>> >> >> Hi J.,
>> >> >>
>> >> >> This might be a limitation of the OS. If you are using for example
>> >> >> Linux, then you can only open about 1000 connections before seeing
>> >> >> "too
>> >> >> many open files in system" error messages. You can adjust the value
>> >> >> in
>> >> >> /etc/security/limits.conf
>> >> >>
>> >> >> Thomas
>> >> >>
>> >> >> J. Mi wrote:
>> >> >> > All,
>> >> >> >
>> >> >> > I get following exception pretty consistently when trying to
>> >> >> > concurrently
>> >> >> > connect about 800 connections.
>> >> >> >
>> >> >> > Any idea? Is this something out of Netty's control? If so, is
>> >> >> > there
>> >> >> > some
>> >> >> > configuration I could do to JVM or operating system to increase
>> >> >> > resource
>> >> >> > capacity?
>> >> >> >
>> >> >> > Thanks,
>> >> >> > Jason
>> >> >> >
>> >> >> > java.net.ConnectException: Connection refused: no further
>> >> >> > information
>> >> >> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native
>> >> >> > Method)
>> >> >> > at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown
>> >> >> > Source)
>> >> >> > at
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:381)
>> >> >> > at
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:351)
>> >> >> > at
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:273)
>> >> >> > at
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:113)
>> >> >> > at
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:53)
>> >> >> > at
>> >> >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
>> >> >> > Source)
>> >> >> > at
>> >> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>> >> >> > Source)
>> >> >> > at java.lang.Thread.run(Unknown Source)
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > ------------------------------------------------------------------------
>> >> >> >
>> >> >> > _______________________________________________
>> >> >> > netty-dev mailing list
>> >> >> > netty-dev at lists.jboss.org
>> >> >> > https://lists.jboss.org/mailman/listinfo/netty-dev
>> >> >> _______________________________________________
>> >> >> netty-dev mailing list
>> >> >> netty-dev at lists.jboss.org
>> >> >> https://lists.jboss.org/mailman/listinfo/netty-dev
>> >> >
>> >> > _______________________________________________
>> >> > netty-dev mailing list
>> >> > netty-dev at lists.jboss.org
>> >> > https://lists.jboss.org/mailman/listinfo/netty-dev
>> >> >
>> >> > Mike McGrady
>> >> > Principal Investigator AF081-028 AFRL SBIR
>> >> > Senior Engineer
>> >> > Topia Technology, Inc
>> >> > 1.253.720.3365
>> >> > mmcgrady at topiatechnology.com
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > netty-dev mailing list
>> >> > netty-dev at lists.jboss.org
>> >> > https://lists.jboss.org/mailman/listinfo/netty-dev
>> >> >
>> >> >
>> >>
>> >> _______________________________________________
>> >> netty-dev mailing list
>> >> netty-dev at lists.jboss.org
>> >> https://lists.jboss.org/mailman/listinfo/netty-dev
>> >
>> >
>> > _______________________________________________
>> > netty-dev mailing list
>> > netty-dev at lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/netty-dev
>> >
>> >
>>
>> _______________________________________________
>> netty-dev mailing list
>> netty-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/netty-dev
>
>
> _______________________________________________
> netty-dev mailing list
> netty-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-dev
>
>
More information about the netty-dev
mailing list