testing netty for 50k+ connections, odd problems

[Aaron Zinman]

Hi Netty community,

I'm trying to evaluate Netty to power the backend of a real-time
high-concurrent connection game. I want to like it, but there are some
issues I need to have resolved first.

My test/benchmark code is here: https://gist.github.com/779303
The code is scala (not java), and I'm doing everything on localhost (client
& server).

I'm getting some odd cpu/memory issues, and the occasional timeout problem.
They seem to be inconsistent with each run. I also have two architecture
questions.

My issues:

1) High CPU at start
I've noticed that when I start this server with 0 connections attempted, the
CPU is around 8-9% and keeps dropping .1% every second or so. Why is that?
I've also noticed the memory tends to keep increasing, but I haven't let it
go long enough to see if its a memory leak or if the GC will kick in.

2) Memory keeps creeping up, but my code is simple and I don't think I have
a leak?
But I have noticed that after I connect many clients, the memory does not
seem to be well recovered (although could be same GC issue), and the CPU
takes a really long time to slowly die off even after all clients have
disconnected.

3) Very high CPU / dropped connection at beginning, but after 'warming up'
server does much better
My last run at 20k connections caused the CPU to go to 60% with some dropped
connections. Oddly enough I waited a while and ran it again (server still
running) and now cpu is just at 11% with the same number of connections (and
less connect time outs). After all the clients disconnect, it stabilized
around ~7%. Does it need to 'warm up'?  Any why does the CPU never get to ~0
(related to #1)?

4) java.net.ConnectException: connection timed out errors
I often get Unexpected exception from downstream: java.net.ConnectException:
connection timed out on the client, usually on about 5% of my connection
tries (~1k/20k). Is this "normal" because I'm requesting so many at once? Am
I doing something wrong?

I also have two execution questions:

1) If I wanted to integrate db calls into this, I know that I should use
the OrderedMemoryAwareThreadPoolExecutor as I am. But my question is, if I
set that core thread size to 100, what happens if I have 150 I/O requests at
the same time while 100 are blocking on db? Do the others just get queued up
and eventually serviced?

2) I need the ability for one channel to write to others, likely in reaction
to something that came in (it's a game). That handler that learned that it
needs to write to other channels is running on one particular thread. Will
there be concurrency issues if it tries to write to those channels at the
same time another thread is also trying to write to those channels? Do I
need to queue up messages somehow per channel?

Thanks so much for taking the time to answer,
Aaron

--
aaron zinman :: mit media lab :: http://web.media.mit.edu/~azinman ::
protocol.by/azinman
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/netty-users/attachments/20110114/a54b93aa/attachment.html