How to host a high performing Netty App in a fluctuating network environment?

[George]

Thanks Lee,

I am using TCP NIO and the servers will be in different host environments
across regions.
I was looking into collectd to gather bandwidth throughput of the server and
compare with the Netty app throughput to decide on the routing logic.
But that didn't seem like a proper solution because of Netty app hitting OOM
in case Channel.isWritable becomes false.

As you explained, if taking some kind of average across Channels writable
status, can give a close indication on congestion, then it will be a good
solution.
I will read more about this in api docs and try it out. Hope it gives a
reliable result.

One more question, Is there an api to get the global throughput statistics?

George

On Wed, Jul 27, 2011 at 12:35 PM, ljohnston <johnstlr at yahoo.co.uk> wrote:

> Hi George
>
> You can't tell directly if a network is congested or not. As Mike has noted
> TCP uses various algorithms to decide if a network is congested or not but
> they basically come down to whether a series of packets sent at a given
> time
> are acknowledged within a certain time window. If they are then there's no
> congestion, if they're not then there's congestion.
>
> You don't state whether you are streaming over TCP or UDP. It's important
> because with TCP it is possible to infer whether there is congestion. With
> UDP it depends on whether your protocol has some form of congestion
> avoidance. If it doesn't then you'll send UDP packets onto the network at
> the top speed the server can manage only for them to be dropped at the
> first
> congested router. Without some form of congestion avoidance you can't
> detect
> this unless the client tells you it's not receiving anything.
>
> Assuming you are using TCP, and that you are using NIO sockets, then there
> is a way you can potentially detect congestion on a single connection. NIO
> sockets have a writeBufferhighWaterMark and a writeBufferLowWaterMark.
> Normally Netty forwards your data to the network stack send buffer
> immediately. If Netty is unable to do so, because the send buffer is full,
> then these two parameters specify how much data Netty will queue before
> setting Channel.isWritable to false.
>
> Once the network stack send buffer drains, and Netty is able to reduce it's
> queue size below writeBufferLowWaterMark, Netty will raise an event stating
> that the channel is again writable.
>
> By checking if the channel is writable after is call to Channel.write, and
> capturing the channelInterestChanged event, you can probably infer whether
> a
> channel is managing to process your traffic in a timely manner. If it
> isn't,
> and you see the same behaviour on other channels, then you can probably
> infer network congestion.
>
> You can tweak the network stack send buffer and Netty
> writeBufferHighWaterMark sizes to find the levels which allow you to detect
> congestion in reasonable time.
>
> One final thought, I assume the server you will redirect the client to is
> on
> a completely different network which doesn't share common links with the
> congested network. Otherwise the redirect might not help.
>
> Cheers
> Lee
>
> --
> View this message in context:
> http://netty-forums-and-mailing-lists.685743.n2.nabble.com/How-to-host-a-high-performing-Netty-App-in-a-fluctuating-network-environment-tp6623821p6627045.html
> Sent from the Netty User Group mailing list archive at Nabble.com.
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/netty-users/attachments/20110727/178651d0/attachment.html