Ultra low latency in cluster application

[David Alves]

Hi Trustin

	I tryed the latest trunk and it still happens.
	It's a bit difficult to try and separate the netty code from the rest  
but I'll try.
	Meanwhile here's where I got:
	The problem lies when the client is producing more than the server  
can handle. If I run the program locally the throughputs are great  
(40.000 evts/sec) and the latency minimal (under 2 ms on average).  
Over the wire with equivalent nodes works fine too (a big drop in  
throughput (6-7.000 evts/sec in avg) and an increase in latency, but  
stable and under 1 sec). Over the wire with different nodes all hell  
breaks loose :) throughput increases at first until it reaches about  
15.000 (by this time latency is already in 100+ sec:) an then it  
starts dropping (until it reaches as low as 2000 evts/sec).
	Maybe I'm doing something wrong. My way of flushing the buffer is  
doing an awayt() periodically on the ChannelFuture from  
channel.write(), is this correct?
	The behaviour changes between Nio and Oio are that Oio performs  
slightly better in single thread but tanks on multi thread and Nio  
performs about the same (latency increases much more rapidly in  
multithread).

Do you have any suggestions? I getting quite desperate...
Thanks for your quick reply btw.
Regards
David Alves
	
	
On Feb 25, 2009, at 2:42 AM, Trustin Lee wrote:

> Hi David,
>
> I'm not sure, but a resource leak due to incorrect shared resource
> usage is suspected.  Could you provide a simple application that
> reproduces your problem?  Also, please try again with the latest trunk
> snapshot to see if any fixes made since the last release helps it.
>
> HTH,
>
> — Trustin Lee, http://gleamynode.net/
>
>
>
> On Wed, Feb 25, 2009 at 10:31 AM, dralves <davidralves at gmail.com>  
> wrote:
>>
>> Hi
>> I'm currently  writing a cluster application that handles streams  
>> of data,
>> and I'm using netty as the fw for async delivery. The application's  
>> data is
>> very time sensitive so even though I need both latency and  
>> throughput if I
>> have to choose I'll choose latency.
>> The application resides in a cluster of servers where each node  
>> accepts
>> connections from multiples clients (not that many at this point,  
>> max 10
>> clients) each message arrives is processed and then there is an  
>> independent
>> (as in it might be to another machine), async reply. At first I  
>> started with
>> the NIO sockets, witch held good thoughput but as latencies  
>> increased over
>> time (reaching some whopping 60secs until i stopped the tests).  
>> Then I tried
>> the Oio wich held also good thoughput but had lower latencies.  
>> Still the
>> latency is increasing over time, slower but still increasing (27  
>> secs at the
>> moment of this post)(server machine under 100% cpu) and I have ran  
>> out of
>> ideas.
>> I flush the buffers on the client side every 10 messages (around  
>> 1500bytes)
>> and have set tcp no delay to true.
>> Can anyone help me?
>> Best Regards
>> David Alves
>> --
>> View this message in context: http://n2.nabble.com/Ultra-low-latency-in-cluster-application-tp2381448p2381448.html
>> Sent from the Netty User Group mailing list archive at Nabble.com.
>>
>> _______________________________________________
>> netty-users mailing list
>> netty-users at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/netty-users
>>
>
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users