testing netty for 50k+ connections, odd problems

[Willi Schönborn]

On Jan 14, 2011, at 9:54 AM, Aaron Zinman wrote:

> Hi Netty community,
> 
> I'm trying to evaluate Netty to power the backend of a real-time high-concurrent connection game. I want to like it, but there are some issues I need to have resolved first.
> 
> My test/benchmark code is here: https://gist.github.com/779303
> The code is scala (not java), and I'm doing everything on localhost (client & server).
I am not an expert in that field but isnt the os limiting your files (or in that case socket connections)?
> 
> I'm getting some odd cpu/memory issues, and the occasional timeout problem. They seem to be inconsistent with each run. I also have two architecture questions.
> 
> My issues:
> 
> 1) High CPU at start
> I've noticed that when I start this server with 0 connections attempted, the CPU is around 8-9% and keeps dropping .1% every second or so. Why is that? I've also noticed the memory tends to keep increasing, but I haven't let it go long enough to see if its a memory leak or if the GC will kick in.
> 
> 2) Memory keeps creeping up, but my code is simple and I don't think I have a leak?
> But I have noticed that after I connect many clients, the memory does not seem to be well recovered (although could be same GC issue), and the CPU takes a really long time to slowly die off even after all clients have disconnected.
> 
> 3) Very high CPU / dropped connection at beginning, but after 'warming up' server does much better
> My last run at 20k connections caused the CPU to go to 60% with some dropped connections. Oddly enough I waited a while and ran it again (server still running) and now cpu is just at 11% with the same number of connections (and less connect time outs). After all the clients disconnect, it stabilized around ~7%. Does it need to 'warm up'?  Any why does the CPU never get to ~0 (related to #1)?
> 
> 4) java.net.ConnectException: connection timed out errors
> I often get Unexpected exception from downstream: java.net.ConnectException: connection timed out on the client, usually on about 5% of my connection tries (~1k/20k). Is this "normal" because I'm requesting so many at once? Am I doing something wrong?
> 
> I also have two execution questions:
> 
> 1) If I wanted to integrate db calls into this, I know that I should use the OrderedMemoryAwareThreadPoolExecutor as I am. But my question is, if I set that core thread size to 100, what happens if I have 150 I/O requests at the same time while 100 are blocking on db? Do the others just get queued up and eventually serviced?
> 
> 2) I need the ability for one channel to write to others, likely in reaction to something that came in (it's a game). That handler that learned that it needs to write to other channels is running on one particular thread. Will there be concurrency issues if it tries to write to those channels at the same time another thread is also trying to write to those channels? Do I need to queue up messages somehow per channel?
> 
> Thanks so much for taking the time to answer,
> Aaron
> 
> --
> aaron zinman :: mit media lab :: http://web.media.mit.edu/~azinman :: protocol.by/azinman
> 
> 
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/netty-users/attachments/20110114/e311c842/attachment.html