Connection timeout taking all cpu

[NicolasDesy]

Hello all,

I have been using netty successfully for quite a while but I started to get
some issues lately...  

I use netty for an async TCP client to a 3rd party chat filtering service. 
The filter is really advanced but not really fast if I sent to messages at
once.  

On my side, I handle the read timeouts with the ReadTimeoutHandler +
HashedWheelTimer.  This is working well so far.


A few months ago they upgraded the service and now use a limit of 80
concurrent connections and put all the incoming ones in a queue.

The problem is I am not the only client of this service, there is also 20
http servers in the cluster that shares the same service. So it happens from
time to time that I also have a lot of connection timeouts.

The problem is there is so many connections timeout, it starts consuming all
cpu until we reboot the game server.  There is 6k conccurrent logout/login
to process and that is killing us :-)


Here is the beginning of our last "minute of hell" :



INFO   | jvm 1    | 2011/08/12 13:58:53 | java.net.ConnectException:
connection timed out
INFO   | jvm 1    | 2011/08/12 13:58:53 |       at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processConnectTimeout(NioClientSocketPipelineSink.java:371)
INFO   | jvm 1    | 2011/08/12 13:58:53 |       at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:283)
INFO   | jvm 1    | 2011/08/12 13:58:53 |       at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
INFO   | jvm 1    | 2011/08/12 13:58:53 |       at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
INFO   | jvm 1    | 2011/08/12 13:58:53 |       at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
INFO   | jvm 1    | 2011/08/12 13:58:53 |       at
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
INFO   | jvm 1    | 2011/08/12 13:58:53 |       at
java.lang.Thread.run(Unknown Source)
INFO   | jvm 1    | 2011/08/12 13:58:53 | java.net.ConnectException:
connection timed out
INFO   | jvm 1    | 2011/08/12 13:58:53 |       at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processConnectTimeout(NioClientSocketPipelineSink.java:371)
INFO   | jvm 1    | 2011/08/12 13:58:53 |       at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:283)
INFO   | jvm 1    | 2011/08/12 13:58:53 |       at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
INFO   | jvm 1    | 2011/08/12 13:58:53 |       at
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
INFO   | jvm 1    | 2011/08/12 13:58:53 |       at
java.lang.Thread.run(Unknown Source)



The seconds stacktrace is repeated continuously until server reboot.   You
also need to know that there was also a couple of read timeouts during the
same second of the first stack trace.

I'm not sure what's going on, I'm looking throught netty sources right but I
thought it would a good idea to ask your help.


Thanks in advance,

-Nicolas


--
View this message in context: http://netty-forums-and-mailing-lists.685743.n2.nabble.com/Connection-timeout-taking-all-cpu-tp6682186p6682186.html
Sent from the Netty User Group mailing list archive at Nabble.com.