How to host a high performing Netty App in a fluctuating network environment?

[John D. Mitchell]

On Jul 27, 2011, at 12:59 , George wrote:
[...]
> I am using TCP NIO and the servers will be in different host environments across regions.

So you can do "client to different servers" testing (and then select which server a client should talk to based on that) but given that you're doing video (IIRC), it's the longer-term steady state throughput that matters rather than the quick and dirty connectivity latency & early throughput -- so that's tougher to figure out cheaply/quickly/accurately.

> I was looking into collectd to gather bandwidth throughput of the server and compare with the Netty app throughput to decide on the routing logic.

Depending on your networking setup (e.g. in your colos), it's not just the saturation of e.g., the local NIC but also any switches/up-links/etc. At a recent place I was working, they had full racks of machines with dual 1Gbps NICs on the front-end servers but all of those links were being run through rack switches that had (a) much, much less than N-servers * 1Gbps of switching fabric capacity and, even worse, (b) had only 1 or 2 * 1Gbps network drops into the backbone switch. So, looking at the performance on a single machine may confuse you when the congestion happens downstream, etc.

> But that didn't seem like a proper solution because of Netty app hitting OOM in case Channel.isWritable becomes false.

Why is Netty running out of memory just because a channel isn't writable? That back-pressure should be propagated to stall the upstream sources.

FWIW, you might also want to google around a bit as I recall there have been some nice write ups earlier this year by people using plain TCP to serve video streams. Might find some helpful details there.

Hope this helps,
John