Ultra low latency in cluster application

[dralves]

Hi Trustin

I tryed the latest trunk and it still happens.
It's a bit difficult to try and separate the netty code from the rest in
order to provide a test, and I can't reproduce the problem locally. But if
you'r still interested I'll do it, no problem.
Meanwhile here's where I got:
The problem lies when the client is producing more than the server can
handle. If I run the program locally the throughputs are great (40.000
evts/sec) and the latency minimal (under 2 ms on average). Over the wire
with equivalent nodes works fine too (a big drop in throughput (6-7.000
evts/sec in avg) and an increase in latency, but stable and under 1 sec).
Over the wire with different nodes all hell breaks loose :) throughput
increases at first until it reaches about 15.000 (by this time latency is
already in 100+ sec:) an then it starts dropping (until it reaches as low as
2000 evts/sec).
Maybe I'm doing something wrong. My way of flushing the buffer is doing an
awayt() periodically on the ChannelFuture from channel.write(), is this
correct?
The behaviour changes between Nio and Oio are that Oio performs slightly
better in single thread but tanks on multi thread and Nio performs about the
same (latency increases much more rapidly in multithread).

Do you have any suggestions? I getting quite desperate...
Thanks for your quick reply btw.
Regards
David Alves

On Wed, Feb 25, 2009 at 2:43 AM, Trustin Lee-2 (via Nabble) <
ml-user+63047-62405660 at n2.nabble.com<ml-user%2B63047-62405660 at n2.nabble.com>
> wrote:

> Hi David,
>
> I'm not sure, but a resource leak due to incorrect shared resource
> usage is suspected.  Could you provide a simple application that
> reproduces your problem?  Also, please try again with the latest trunk
> snapshot to see if any fixes made since the last release helps it.
>
> HTH,
>
> — Trustin Lee, http://gleamynode.net/
>
>
>
> On Wed, Feb 25, 2009 at 10:31 AM, dralves <davidralves at ...<http://n2.nabble.com/user/SendEmail.jtp?type=node&node=2381665&i=0>>
> wrote:
> >
> > Hi
> >  I'm currently  writing a cluster application that handles streams of
> data,
> > and I'm using netty as the fw for async delivery. The application's data
> is
> > very time sensitive so even though I need both latency and throughput if
> I
> > have to choose I'll choose latency.
> >  The application resides in a cluster of servers where each node accepts
> > connections from multiples clients (not that many at this point, max 10
> > clients) each message arrives is processed and then there is an
> independent
> > (as in it might be to another machine), async reply. At first I started
> with
> > the NIO sockets, witch held good thoughput but as latencies increased
> over
> > time (reaching some whopping 60secs until i stopped the tests). Then I
> tried
> > the Oio wich held also good thoughput but had lower latencies. Still the
> > latency is increasing over time, slower but still increasing (27 secs at
> the
> > moment of this post)(server machine under 100% cpu) and I have ran out of
>
> > ideas.
> > I flush the buffers on the client side every 10 messages (around
> 1500bytes)
> > and have set tcp no delay to true.
> > Can anyone help me?
> > Best Regards
> > David Alves
> > --
> > View this message in context:
> http://n2.nabble.com/Ultra-low-latency-in-cluster-application-tp2381448p2381448.html
> > Sent from the Netty User Group mailing list archive at Nabble.com.
> >
> > _______________________________________________
> > netty-users mailing list
> > netty-users at ...<http://n2.nabble.com/user/SendEmail.jtp?type=node&node=2381665&i=1>
> > https://lists.jboss.org/mailman/listinfo/netty-users
> >
>
> _______________________________________________
> netty-users mailing list
> netty-users at ...<http://n2.nabble.com/user/SendEmail.jtp?type=node&node=2381665&i=2>
> https://lists.jboss.org/mailman/listinfo/netty-users
>
>
> ------------------------------
>  This email is a reply to your post @
> http://n2.nabble.com/Ultra-low-latency-in-cluster-application-tp2381448p2381665.html
> You can reply by email or by visting the link above.
>
>

-- 
View this message in context: http://n2.nabble.com/Ultra-low-latency-in-cluster-application-tp2381448p2382399.html
Sent from the Netty User Group mailing list archive at Nabble.com.