testing netty for 50k+ connections, odd problems

[Aaron Zinman]

Thanks for getting back Tristan.

JVM does not return unused memory to the OS very well as far as I know.
>  To measure the actual amount of memory being used, you have to get the
> values from Runtime.
>

I need to setup VisualVM to see what is going on. I've been having some
issues with jstatd showing up remotely. I'll sort it out. The documentation
could be much better; I might have to go the JMX route :/



> Yes, Java needs warm-up.  You can specify some HotSpot options when you
> launch your VM to make compilation occur earlier or later, though.
>

Do you have particular recommendations for Netty? I would love to see the
appropriate GC, etc JVM flag recommendations to tune it for many concurrent
connections + executors. Especially if anyone has done any real significant
tuning and documented this, it would be great for the community.



I'm not sure why CPU doesn't go down to 0~1% when there's no connection
> to manage though.  Let me try to reproduce the problem.  Please let me
> know if you find something meanwhile.
>


It did eventually go down. Very odd that it takes SOO long to do. I need to
inspect in VisualVM to see what's happening. I've been going through your
code a bit and getting a slightly better understanding of how these
executors work.



> I think so.  Practically, any connection attempt can fail, so the client
> must be prepared to reconnect on failure.
>
>
This seems very strange to me. When I ssh into one of my boxes, or when I go
to Google's homepage, I do not get a connection drop (as far as I can tell).
I suppose I have some ignorance of the "ops" nature of high-end sites, but I
feel like this number should be well under 1%. While I haven't opened up
wireshark and tallied the TCP states and what's going on (yet), considering
this is localhost<-->localhost what could possibly be causing this? A
weakness in NIO? Could something be tuned/implemented differently?


By the way, it would pre pretty interesting to see an MIT media lab
> project uses Netty.  Please let us know when it happens! :-)
>
>
Hahah this is an outside contract, unfortunately. I'm actually using Jetty
for my main thesis project as it is REST-based, and Jetty has been good for
this. I'd be curious to know how Netty's performance compares in this
scenario, and if there are any frameworks to make this easier. I was coming
from Tornado in Python which is a dream webstack, and would love to see
something like this available in Java/Scala (ideally Scala). JAX-RS is quite
close, but it took a bit of fiddling to figure out how to get Jetty properly
embedded. All these WAR files, web.xml, etc is too much for the average
website / web developer. It should be much easier. Tornado's biggest
strength is it's small code base size, which is easily understandable.

Aaron

--
aaron zinman :: mit media lab :: http://web.media.mit.edu/~azinman ::
protocol.by/azinman
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/netty-users/attachments/20110115/119264d5/attachment.html