testing netty for 50k+ connections, odd problems

["이희승 (Trustin Lee)"]

Hi Aaron,

On 01/14/2011 05:54 PM, Aaron Zinman wrote:
> Hi Netty community,
> 
> I'm trying to evaluate Netty to power the backend of a real-time
> high-concurrent connection game. I want to like it, but there are some
> issues I need to have resolved first.
> 
> My test/benchmark code is here: https://gist.github.com/779303
> The code is scala (not java), and I'm doing everything on localhost
> (client & server).
> 
> I'm getting some odd cpu/memory issues, and the occasional timeout
> problem. They seem to be inconsistent with each run. I also have two
> architecture questions.
> 
> My issues:
> 
> 1) High CPU at start
> I've noticed that when I start this server with 0 connections attempted,
> the CPU is around 8-9% and keeps dropping .1% every second or so. Why is
> that? I've also noticed the memory tends to keep increasing, but I
> haven't let it go long enough to see if its a memory leak or if the GC
> will kick in.

I guess it's because HotSpot is busy JIT-ing the byte code.

> 2) Memory keeps creeping up, but my code is simple and I don't think I
> have a leak?
> But I have noticed that after I connect many clients, the memory does
> not seem to be well recovered (although could be same GC issue), and the
> CPU takes a really long time to slowly die off even after all clients
> have disconnected.

JVM does not return unused memory to the OS very well as far as I know.
 To measure the actual amount of memory being used, you have to get the
values from Runtime.

> 3) Very high CPU / dropped connection at beginning, but after 'warming
> up' server does much better
> My last run at 20k connections caused the CPU to go to 60% with some
> dropped connections. Oddly enough I waited a while and ran it again
> (server still running) and now cpu is just at 11% with the same number
> of connections (and less connect time outs). After all the clients
> disconnect, it stabilized around ~7%. Does it need to 'warm up'?  Any
> why does the CPU never get to ~0 (related to #1)?

Yes, Java needs warm-up.  You can specify some HotSpot options when you
launch your VM to make compilation occur earlier or later, though.

I'm not sure why CPU doesn't go down to 0~1% when there's no connection
to manage though.  Let me try to reproduce the problem.  Please let me
know if you find something meanwhile.

> 4) java.net.ConnectException: connection timed out errors
> I often get Unexpected exception from downstream:
> java.net.ConnectException: connection timed out on the client, usually
> on about 5% of my connection tries (~1k/20k). Is this "normal" because
> I'm requesting so many at once? Am I doing something wrong?

I think so.  Practically, any connection attempt can fail, so the client
must be prepared to reconnect on failure.

> I also have two execution questions:
> 
> 1) If I wanted to integrate db calls into this, I know that I should use
> the OrderedMemoryAwareThreadPoolExecutor as I am. But my question is, if
> I set that core thread size to 100, what happens if I have 150 I/O
> requests at the same time while 100 are blocking on db? Do the others
> just get queued up and eventually serviced?

Yes.

> 2) I need the ability for one channel to write to others, likely in
> reaction to something that came in (it's a game). That handler that
> learned that it needs to write to other channels is running on one
> particular thread. Will there be concurrency issues if it tries to write
> to those channels at the same time another thread is also trying to
> write to those channels? Do I need to queue up messages somehow per channel?

No.  Netty channel is thread safe, and thus there's no need to
synchronized or manage event queue by yourself.

By the way, it would pre pretty interesting to see an MIT media lab
project uses Netty.  Please let us know when it happens! :-)

HTH,
Trustin

-- 
Trustin Lee, http://gleamynode.net/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 294 bytes
Desc: OpenPGP digital signature
Url : http://lists.jboss.org/pipermail/netty-users/attachments/20110115/a3c6a9c9/attachment.bin