Connection timeout taking all cpu

Tue Aug 16 09:18:56 EDT 2011

Hi Trustin,
I can't provide you a debugger screencast since I can only reproduce the issue on a busy production server.   
However... I think I found the root of my issue and this is not related to netty! :-)   
I use Tanuki java service wrapper and apparently there is an issue when there is too much log output at the same time.  There was too many stacktraces for the connection timeout at the same time.    Now we handle the exceptions properly and we also need to update the service wrapper to the latest version during next maintenance.
Thanks anyway!
-Nicolas
Date: Tue, 16 Aug 2011 03:13:50 -0700
From: ml-node+6690763-635655155-74643 at n2.nabble.com
To: liguorien at hotmail.com
Subject: Re: Connection timeout taking all cpu

                Hi Nicolas,
Could you run your server in debug mode, reproduce the problem, and let me know if you observe any weird behavior?  For example, you could determine if such many ConnectExceptions are expected or it's some sort of infinite loop.
Even better, you could share your debugger screencast so that I can see what's going on inside, like this one:

    http://screencast.com/t/cdIcloD0kfBt
-- 
Trustin Lee

                On Saturday, August 13, 2011 at 9:21 AM, NicolasDesy wrote:
                    Hello all,

I have been using netty successfully for quite a while but I started to get
some issues lately...  

I use netty for an async TCP client to a 3rd party chat filtering service. 
The filter is really advanced but not really fast if I sent to messages at
once.  

On my side, I handle the read timeouts with the ReadTimeoutHandler +
HashedWheelTimer.  This is working well so far.

A few months ago they upgraded the service and now use a limit of 80
concurrent connections and put all the incoming ones in a queue.

The problem is I am not the only client of this service, there is also 20
http servers in the cluster that shares the same service. So it happens from
time to time that I also have a lot of connection timeouts.

The problem is there is so many connections timeout, it starts consuming all
cpu until we reboot the game server.  There is 6k conccurrent logout/login
to process and that is killing us :-)

Here is the beginning of our last "minute of hell" :

INFO   | jvm 1    | 2011/08/12 13:58:53 | java.net.ConnectException:
connection timed out
INFO   | jvm 1    | 2011/08/12 13:58:53 |       at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processConnectTimeout(NioClientSocketPipelineSink.java:371)
INFO   | jvm 1    | 2011/08/12 13:58:53 |       at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:283)
INFO   | jvm 1    | 2011/08/12 13:58:53 |       at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
INFO   | jvm 1    | 2011/08/12 13:58:53 |       at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
INFO   | jvm 1    | 2011/08/12 13:58:53 |       at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
INFO   | jvm 1    | 2011/08/12 13:58:53 |       at
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
INFO   | jvm 1    | 2011/08/12 13:58:53 |       at
java.lang.Thread.run(Unknown Source)
INFO   | jvm 1    | 2011/08/12 13:58:53 | java.net.ConnectException:
connection timed out
INFO   | jvm 1    | 2011/08/12 13:58:53 |       at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processConnectTimeout(NioClientSocketPipelineSink.java:371)
INFO   | jvm 1    | 2011/08/12 13:58:53 |       at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:283)
INFO   | jvm 1    | 2011/08/12 13:58:53 |       at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
INFO   | jvm 1    | 2011/08/12 13:58:53 |       at
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
INFO   | jvm 1    | 2011/08/12 13:58:53 |       at
java.lang.Thread.run(Unknown Source)

The seconds stacktrace is repeated continuously until server reboot.   You
also need to know that there was also a couple of read timeouts during the
same second of the first stack trace.

I'm not sure what's going on, I'm looking throught netty sources right but I
thought it would a good idea to ask your help.

Thanks in advance,

-Nicolas

--
View this message in context: http://netty-forums-and-mailing-lists.685743.n2.nabble.com/Connection-timeout-taking-all-cpu-tp6682186p6682186.html
Sent from the Netty User Group mailing list archive at Nabble.com.
_______________________________________________
netty-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/netty-users

_______________________________________________

netty-users mailing list

[hidden email]

https://lists.jboss.org/mailman/listinfo/netty-users
	what we call human nature in actuality is human habit

http://gleamynode.net/

		If you reply to this email, your message will be added to the discussion below:
		http://netty-forums-and-mailing-lists.685743.n2.nabble.com/Connection-timeout-taking-all-cpu-tp6682186p6690763.html

		To unsubscribe from Connection timeout taking all cpu, click here.

--
View this message in context: http://netty-forums-and-mailing-lists.685743.n2.nabble.com/Connection-timeout-taking-all-cpu-tp6682186p6691264.html
Sent from the Netty User Group mailing list archive at Nabble.com.