What might be causing increasing latency?
justkevin
filter.netty at wx3.com
Thu Jan 7 21:38:28 EST 2010
Thanks for the suggestions-- it turns out the problem was client side, not
Netty at all.
If anyone wants to learn from my mistake:
After posting I added a ChannelFuture listener object which timed how long
it took to write, and saw that it was zero. Combined with the fact I knew
the delay was after the creation of the Pong event almost ruled out anything
server side.
I reexamined the client code and found the error. The client processes a
message frame (length prepended group of messages) when a Flash socket fires
a SocketData event letting it know some data arrives. The server sends one
message frame every 200 ms. Each message frame creates a socket data event
when it arrives.
Except that pings are processed immediately, so it's possible for a pong
response to arrive bunched up with another message, creating a backlog. The
backlogged pong didn't get processed until some other message came in after
it. Since only pings create a backlog, the problem was caused, and
exacerbated, by my attempts to measure it.
Thingfish wrote:
>
> Hehe, that sounds like the logical value. My first search for 'netty
> tcpnodelay" returned this
> http://www.jboss.org/file-access/default/members/jbossmessaging/freezone/docs/usermanual-2.0.0.beta1/html/configuring-transports.html
> which states "jbm.remoting.netty.tcpnodelay. If this is true then Nagle's
> algorithm <http://en.wikipedia.org/wiki/Nagle's_algorithm> will be
> enabled.
> The default value for this property is true."
>
> Clearly not easy to get right :)
>
>
> - Jannick
>
>
> 2010/1/8 "Trustin Lee (이희승)" <trustin at gmail.com>
>
>> Jannick Bitsch wrote:
>> > Hi
>> >
>> > One possibility is that the effect is caused by
>> > http://en.wikipedia.org/wiki/Nagle's_algorithm. Could be that the tcp
>> stack
>> > holds on to the data with an increasing timeout algorithm, although
>> that
>> > doesn't really account for the variance. You can try to turn it off.
>> >
>> > From
>> >
>> http://www.jboss.org/file-access/default/members/netty/freezone/api/3.2/index.html
>> > :
>> > ServerBootstrap b = ...;
>> > b.setOption("child.tcpNoDelay", true);
>> >
>> > I actually think you need to set if to false to turn nagle's algorithm
>> off,
>> > although that doesn't seem very intuitive :)
>>
>> Actually, setting tcpNoDelay to true disables Nagle's algorithm.
>> Confusing! :)
>>
>> > If that doesn't work, try to profile your gc and see if theres some
>> > excessive object allocations or something, that could cause it to
>> introduce
>> > long pauses.
>> >
>> >
>> > Regards
>> > Jannick
>> >
>> >
>> > 2010/1/7 justkevin <filter.netty at wx3.com>
>> >
>> >> After noticing my game app was occasionally running sluggish, I spent
>> quite
>> >> a
>> >> bit of time trying to figure out why without success. If anyone has
>> >> suggestions on what to try I'd appreciate it. The details are long,
>> but
>> I
>> >> don't know what might be important:
>> >>
>> >> Details:
>> >>
>> >> The client is written in AS3.
>> >>
>> >> The server is written in Java using Netty 3.2 (I've updated to the
>> most
>> >> recent alpha).
>> >>
>> >> I'm testing locally, with client and server on the same box.
>> >>
>> >> When the first client connects, latency as measured with a round-trip
>> ping
>> >> pong is low, < 10 ms. It can remain low for a while, but eventually
>> begins
>> >> increasing in spurts at varying rates. For example, it might be < 10
>> ms
>> >> for
>> >> a while, then 50-100 ms for a while, then 100-200 ms, and so forth.
>> There's
>> >> no upper limit, I've seen delays in multiple seconds. The larger the
>> >> latency, the larger the variance. The increases may be correlated to
>> large
>> >> bursts of data.
>> >>
>> >> If I restart the client, the latency will usually drop, but not
>> always.
>> >> Sometimes it restarts high.
>> >>
>> >> If I restart the server, the next client to connect will get a low
>> latency.
>> >>
>> >> Using system times, I've established that the latency is entirely on
>> the
>> >> downstream-- the ping takes 0ms to reach the server, and all the
>> increase
>> >> is
>> >> between the time the pong is created and when its received by the
>> client.
>> >> >From this I'm guessing it's something with how I'm sending messages.
>> >>
>> >> The only ChannelPipeline handler on outbound is a
>> LengthFieldPrepender.
>> >>
>> >> Here is how I'm sending the message:
>> >>
>> >> // A Netty channel expects a message in the format of a
>> >> ChannelBuffer:
>> >> ChannelBuffer buffer = ChannelBuffers.dynamicBuffer();
>> >> // Wrap the buffer with a ChannelBufferOutputStream,
>> which
>> >> implements
>> >> DataOutput
>> >> // (our messages expect to write to a DataOutput)
>> >> ChannelBufferOutputStream output = new
>> >> ChannelBufferOutputStream(buffer);
>> >> // First write the message id:
>> >> output.writeByte(message.getId());
>> >> // Then have the message write itself, checking to make
>> sure
>> >> the message
>> >> // obeys its length contract:
>> >> int beforeLength = output.writtenBytes();
>> >> int written = message.write(output);
>> >> if(beforeLength + written != output.writtenBytes()){
>> >> logger.severe("Outbound message violated length
>> >> contract:" + message);
>> >> throw new RuntimeException("Message violated
>> length
>> >> contract:" +
>> >> message);
>> >> }
>> >> // Finally, put into the channel. The channel pipeline
>> >> should already have
>> >> a length prepender:
>> >> this.channel.write(buffer);
>> >>
>> >> Thanks in advance for any suggestions,
>> >> --
>> >> View this message in context:
>> >>
>> http://n2.nabble.com/What-might-be-causing-increasing-latency-tp4269271p4269271.html
>> >> Sent from the Netty User Group mailing list archive at Nabble.com.
>> >> _______________________________________________
>> >> netty-users mailing list
>> >> netty-users at lists.jboss.org
>> >> https://lists.jboss.org/mailman/listinfo/netty-users
>> >>
>> >
>> >
>> >
>> ------------------------------------------------------------------------
>> >
>> > _______________________________________________
>> > netty-users mailing list
>> > netty-users at lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/netty-users
>>
>> --
>> what we call human nature in actuality is human habit
>> http://gleamynode.net/
>>
>>
>>
>> _______________________________________________
>> netty-users mailing list
>> netty-users at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/netty-users
>>
>>
>
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users
>
>
--
View this message in context: http://n2.nabble.com/What-might-be-causing-increasing-latency-tp4269271p4270142.html
Sent from the Netty User Group mailing list archive at Nabble.com.
More information about the netty-users
mailing list