Connection timeout taking all cpu

[이희승 (Trustin Lee)]

Hi Nicolas,

Could you run your server in debug mode, reproduce the problem, and let me know if you observe any weird behavior? For example, you could determine if such many ConnectExceptions are expected or it's some sort of infinite loop.

Even better, you could share your debugger screencast so that I can see what's going on inside, like this one: 

http://screencast.com/t/cdIcloD0kfBt

-- 
Trustin Lee (http://gleamynode.net/)

On Saturday, August 13, 2011 at 9:21 AM, NicolasDesy wrote:

> Hello all,
> 
> I have been using netty successfully for quite a while but I started to get
> some issues lately... 
> 
> I use netty for an async TCP client to a 3rd party chat filtering service. 
> The filter is really advanced but not really fast if I sent to messages at
> once. 
> 
> On my side, I handle the read timeouts with the ReadTimeoutHandler +
> HashedWheelTimer. This is working well so far.
> 
> 
> A few months ago they upgraded the service and now use a limit of 80
> concurrent connections and put all the incoming ones in a queue.
> 
> The problem is I am not the only client of this service, there is also 20
> http servers in the cluster that shares the same service. So it happens from
> time to time that I also have a lot of connection timeouts.
> 
> The problem is there is so many connections timeout, it starts consuming all
> cpu until we reboot the game server. There is 6k conccurrent logout/login
> to process and that is killing us :-)
> 
> 
> Here is the beginning of our last "minute of hell" :
> 
> 
> 
> INFO | jvm 1 | 2011/08/12 13:58:53 | java.net.ConnectException:
> connection timed out
> INFO | jvm 1 | 2011/08/12 13:58:53 | at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processConnectTimeout(NioClientSocketPipelineSink.java:371)
> INFO | jvm 1 | 2011/08/12 13:58:53 | at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:283)
> INFO | jvm 1 | 2011/08/12 13:58:53 | at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> INFO | jvm 1 | 2011/08/12 13:58:53 | at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
> INFO | jvm 1 | 2011/08/12 13:58:53 | at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> INFO | jvm 1 | 2011/08/12 13:58:53 | at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> INFO | jvm 1 | 2011/08/12 13:58:53 | at
> java.lang.Thread.run(Unknown Source)
> INFO | jvm 1 | 2011/08/12 13:58:53 | java.net.ConnectException:
> connection timed out
> INFO | jvm 1 | 2011/08/12 13:58:53 | at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processConnectTimeout(NioClientSocketPipelineSink.java:371)
> INFO | jvm 1 | 2011/08/12 13:58:53 | at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:283)
> INFO | jvm 1 | 2011/08/12 13:58:53 | at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> INFO | jvm 1 | 2011/08/12 13:58:53 | at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> INFO | jvm 1 | 2011/08/12 13:58:53 | at
> java.lang.Thread.run(Unknown Source)
> 
> 
> 
> The seconds stacktrace is repeated continuously until server reboot. You
> also need to know that there was also a couple of read timeouts during the
> same second of the first stack trace.
> 
> I'm not sure what's going on, I'm looking throught netty sources right but I
> thought it would a good idea to ask your help.
> 
> 
> Thanks in advance,
> 
> -Nicolas
> 
> 
> --
> View this message in context: http://netty-forums-and-mailing-lists.685743.n2.nabble.com/Connection-timeout-taking-all-cpu-tp6682186p6682186.html
> Sent from the Netty User Group mailing list archive at Nabble.com (http://Nabble.com).
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org (mailto:netty-users at lists.jboss.org)
> https://lists.jboss.org/mailman/listinfo/netty-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/netty-users/attachments/20110816/4b14c33e/attachment.html