java.net.ConnectException thrown

Tue Sep 22 01:52:39 EDT 2009

Hi Trustin,

Thanks for looking into it. The problem most likely due to my local server
not configured to handle this many HTTP requests. After I increased maximum
threads for my Tomcat, the problem was gone. Maybe that's how your server is
configured.

As you advised, I'll look into reusing connections, or something like
connection pooling/management.
For the lock-up issue, not sure what it is. Please give me more lead if you
get a chance.
About the large thread pool in Requestor, if that's what you were referring,
it is used to simulate concurrent traffic loads/requests into our servlet
which could use Netty to retrieve online info from multiple Internet
sources. The application itself does not necessarily need to do
multi-threading.

Thanks again,
Jason
On Mon, Sep 21, 2009 at 6:16 PM, Trustin Lee (이희승) <trustin at gmail.com>wrote:

> Hi Jason,
>
> I ran your code by myself, and I was not able to reproduce the problem
> even after increasing the parameters.  I did not try public servers
> though as I don't want to overload them.
>
> According to your code, you make a new connection attempt for each
> request.  You will have a lot of ports with 'TIME_WAIT' state, which
> slow down the connection rate.  I'd recommend to keep the connection
> alive and reuse it over and over (i.e. HTTP Keep-Ali
>
> On the other hand, I see some lock-up in your application during the
> test, which keeps me from completing the test, but I did not
> investigate further because it seemed irrelevant to the original
> problem.  Actually, you don't even need a large thread pool to run
> this test.  You can perform the same job with only one thread,
> resulting higher throughput, because everything is asynchronous in
> Netty. :)
>
> HTH,
>
> — Trustin Lee, http://gleamynode.net/
>
>  On Fri, Sep 18, 2009 at 5:10 PM, J. Mi <jmi258 at gmail.com> wrote:
> > Trustin,
> >
> > Yes, my problem can be reproduced consistently. Attached are 4 java
> files.
> > Among them, HttpClientPipelineFactory.java and HttpResponseHandler.java
> are
> > almost unchanged from their example sibling of snoop, except that I
> enabled
> > HttpChunkAggregator in HttpClientPipelineFactory and offered answer in
> > HttpResponseHandler to be retrieved.
> >
> > I converted the original HttpClient to ConcurrentHttpClient to
> concurrently
> > send http requests to any server. I tried to achieve maximum concurrency
> by
> > adding ChannelFutureListener when connecting (see line 70). It's supposed
> > to work better for me. But if I used the commented-out code below
> instead. I
> > can run more connections.
> >
> > Finally, Requestor is the test driver (into ConcurrentHttpClient) with
> > multi-threading to observe how Netty scales under heavy loads.
> > To run the program:
> >>java -cp %CLASSPATH% myperfeval.http.Requestor 80 10
> http://www.yahoo.com
> >
> > where 80 is for n threads spawn from Requestor and 10 is the number of
> Http
> > servers that ConcurrentHttpClient is to hit for each thread. So in this
> > case, it could reach 800 connections at a time. For the test, you can
> either
> > use a public server like yahoo, or a local test server available to
> you, as
> > the thrid argument. I found that with yahoo, I tend to run into timeout
> > trouble. And with a local test server, I'll get the connection exception
> as
> > in the original email. You can play with the first two arguments to
> narrow
> > it down when it starts to trouble your system. Requestor can run a little
> > more connections in Windows than VMware with Centso 5.3.
> >
> > I suspect that somehow ConcurrentHttpClient might not do things right.
> >
> > Thanks a lot for your time!
> > Jason
> >
> >
> > On Thu, Sep 17, 2009 at 7:36 PM, Trustin Lee (이희승) <trustin at gmail.com>
> > wrote:
> >>
> >> Sorry about the misleading post.  I was merely answering Mike's
> questions.
> >>
> >> I have not experienced such a problem so far in Linux.  I am using
> >> Fedora at the moment but I also use RHEL for performance tests.
> >>
> >> Does the connection attempt succeed when you retry or fail forever once
> >> refused?
> >>
> >> If all subsequent connection attempts are refused, it means the server
> >> has been terminated for some reason.  Otherwise, it's a temporary
> >> problem related with your operating system.
> >>
> >> I'd like to investigate this problem further if there's a way to
> >> reproduce the problem reliably.  Could you post a simple client-server
> >> test application so that I can run it by myself? A simple modification
> >> of the official examples would be perfect.
> >>
> >> Thanks
> >>
> >> — Trustin Lee, http://gleamynode.net/
> >>
> >> On Fri, Sep 18, 2009 at 10:21 AM, J. Mi <jmi258 at gmail.com> wrote:
> >> > Trustin,
> >> >
> >> > I already tried that. I tried to increase 'nofile' to 90000 both in
> >> > /etc/security/limits.conf (and restart my Centos in VMware) and on the
> >> > fly
> >> > for a perticular termal session by 'ulimit -n 90000' command. None of
> >> > them
> >> > work for me.
> >> >
> >> > The exception I got was "java.net.ConnectException: Connection
> refused:
> >> > no
> >> > further information..", not "Too many open files..." althouth I seem
> to
> >> > hear
> >> > that file descriptors, connections and thread might share this same
> >> > limit.
> >> >
> >> > This problem blocked me for days. And it doesn't seem to be the limit
> on
> >> > http server side because I tried to use the same Netty http client to
> >> > hit
> >> > differnt server and had the same problem.
> >> >
> >> > Thanks for your help. Appreciated it.
> >> >
> >> > Jason
> >> >
> >> > On Thu, Sep 17, 2009 at 5:06 PM, Trustin Lee (이희승) <trustin at gmail.com
> >
> >> > wrote:
> >> >>
> >> >> I hope this post answers the first question.
> >> >>
> >> >> http://gleamynode.net/articles/1557/
> >> >>
> >> >> I'm not sure about the other questions though. :)
> >> >>
> >> >> — Trustin Lee, http://gleamynode.net/
> >> >>
> >> >>
> >> >>
> >> >> On Fri, Sep 18, 2009 at 2:37 AM, Michael McGrady
> >> >> <mmcgrady at topiatechnology.com> wrote:
> >> >> > What is the impact of changing these settings?  Why is the default
> >> >> > low
> >> >> > instead of hight?  And, any other questions I should be asking
> before
> >> >> > changing this configuration?
> >> >> > MIke
> >> >> > On Sep 17, 2009, at 10:34 AM, J. Mi wrote:
> >> >> >
> >> >> > H Thomas,
> >> >> >
> >> >> > Thanks for the lead. To be able to have more connections before
> >> >> > seeing
> >> >> > "java.net.ConnectException: Connection refused...", which
> >> >> > configuration
> >> >> > I
> >> >> > need to change in /etc/security/limits.conf? I added
> >> >> > * soft nofile 20000
> >> >> > * hard nofile 20000
> >> >> >
> >> >> > and first time just tried to re-login, the new limit didn't work
> for
> >> >> > me
> >> >> > for
> >> >> > more connections. I then tried to restart my VMware. It still
> didn't
> >> >> > work
> >> >> > for me
> >> >> >
> >> >> > Does 'nofile' (max number of open files) have anything to do with
> >> >> > socket
> >> >> > connections?
> >> >> >
> >> >> > Appreciate if you or anyone else can guide me more on this.
> >> >> > btw, if any of you know any equivalent OS settings on Windows, i'll
> >> >> > try
> >> >> > that
> >> >> > as well.
> >> >> >
> >> >> > Jason
> >> >> >
> >> >> >
> >> >> > On Wed, Sep 16, 2009 at 3:38 PM, Thomas Bocek <bocek at ifi.uzh.ch>
> >> >> > wrote:
> >> >> >>
> >> >> >> Hi J.,
> >> >> >>
> >> >> >> This might be a limitation of the OS. If you are using for example
> >> >> >> Linux, then you can only open about 1000 connections before seeing
> >> >> >> "too
> >> >> >> many open files in system" error messages. You can adjust the
> value
> >> >> >> in
> >> >> >> /etc/security/limits.conf
> >> >> >>
> >> >> >> Thomas
> >> >> >>
> >> >> >> J. Mi wrote:
> >> >> >> > All,
> >> >> >> >
> >> >> >> > I get following exception pretty consistently when trying to
> >> >> >> > concurrently
> >> >> >> > connect about 800 connections.
> >> >> >> >
> >> >> >> > Any idea? Is this something out of Netty's control? If so, is
> >> >> >> > there
> >> >> >> > some
> >> >> >> > configuration I could do to JVM or operating system to increase
> >> >> >> > resource
> >> >> >> > capacity?
> >> >> >> >
> >> >> >> > Thanks,
> >> >> >> > Jason
> >> >> >> >
> >> >> >> > java.net.ConnectException: Connection refused: no further
> >> >> >> > information
> >> >> >> >         at sun.nio.ch.SocketChannelImpl.checkConnect(Native
> >> >> >> > Method)
> >> >> >> >         at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown
> >> >> >> > Source)
> >> >> >> >         at
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:381)
> >> >> >> >         at
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:351)
> >> >> >> >         at
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:273)
> >> >> >> >         at
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:113)
> >> >> >> >         at
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:53)
> >> >> >> >         at
> >> >> >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> >> >> >> > Source)
> >> >> >> >         at
> >> >> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> >> >> >> > Source)
> >> >> >> >         at java.lang.Thread.run(Unknown Source)
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> ------------------------------------------------------------------------
> >> >> >> >
> >> >> >> > _______________________________________________
> >> >> >> > netty-dev mailing list
> >> >> >> > netty-dev at lists.jboss.org
> >> >> >> > https://lists.jboss.org/mailman/listinfo/netty-dev
> >> >> >> _______________________________________________
> >> >> >> netty-dev mailing list
> >> >> >> netty-dev at lists.jboss.org
> >> >> >> https://lists.jboss.org/mailman/listinfo/netty-dev
> >> >> >
> >> >> > _______________________________________________
> >> >> > netty-dev mailing list
> >> >> > netty-dev at lists.jboss.org
> >> >> > https://lists.jboss.org/mailman/listinfo/netty-dev
> >> >> >
> >> >> > Mike McGrady
> >> >> > Principal Investigator AF081-028 AFRL SBIR
> >> >> > Senior Engineer
> >> >> > Topia Technology, Inc
> >> >> > 1.253.720.3365
> >> >> > mmcgrady at topiatechnology.com
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > _______________________________________________
> >> >> > netty-dev mailing list
> >> >> > netty-dev at lists.jboss.org
> >> >> > https://lists.jboss.org/mailman/listinfo/netty-dev
> >> >> >
> >> >> >
> >> >>
> >> >> _______________________________________________
> >> >> netty-dev mailing list
> >> >> netty-dev at lists.jboss.org
> >> >> https://lists.jboss.org/mailman/listinfo/netty-dev
> >> >
> >> >
> >> > _______________________________________________
> >> > netty-dev mailing list
> >> > netty-dev at lists.jboss.org
> >> > https://lists.jboss.org/mailman/listinfo/netty-dev
> >> >
> >> >
> >>
> >> _______________________________________________
> >> netty-dev mailing list
> >> netty-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/netty-dev
> >
> >
> > _______________________________________________
> > netty-dev mailing list
> > netty-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/netty-dev
> >
> >
>
> _______________________________________________
> netty-dev mailing list
> netty-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/netty-dev/attachments/20090921/a8d6a043/attachment-0001.html 