Unfair writers/readers ration (revisited)

[David Alves]

Hi Trustin

> If I understood correctly, the problem is that the client takes 3 times
> more CPU than the server for encoding, right?

	Actually more than that, one server is stable at 65% and I need three clients (that maintain a stable load at 75%) to flood the ethernet I/O. (Get to the point that if I add more clients the throughput for one server doesn't increase).

> 
>> - Should I make sure that "streams" of the same client to the same
>>  server use the same pipeline?
> 
> I don't think so.

	My question was actually If I should use the same channel (i.e. async socket connection) or keep the +- 12 connections (channels) from a single client to a single server (server usually has at most three clients, i.e. 36 connections, and clients usually send to three different servers, ie. 36 connections).

> 
>> - Can I make serialization any faster (I'm using
>>  Channels.dynamicBuffer())?
> 
> I guess you already tuned the dynamic buffer creation by specifying
> estimated message size, right?  If not, try to specify better default
> when you create a dynamic buffer to reduce the number of expansion or
> unnecessary allocation.

	You are right, I already specified the size, and I actually update the estimated length based on the size of the previous tuple, as tuples are mostly the same size.
	I.e. by (where bout is the buffer created by Channels.dynamicBuffer():
	
	ChannelBuffer encoded = bout.buffer();
   encoded.setInt(0, encoded.writerIndex() - 4);
   if (this.estimatedLength != bout.buffer().writerIndex()) {
	this.estimatedLength = bout.buffer().writerIndex();
   }

> 
>> - the serialization is performed on a OneToOneEncoder this is ran by
>>  my own threads right? not by Netty's client I/O workers?
> 
> The serialization is performed in the I/O worker thread if you called
> write() in your handler and you did not insert an ExecutionHandler to
> the pipeline.  If the number of connections is small, I'd recommend you
> to increase the number of I/O worker threads by specifying an integer to
> NioClient/ServerSocketChannelFactory constructor calls.  As of Netty
> 3.2.0.BETA1, the default has been doubled for better out-of-the-box
> performance.
> 

	I'm already using 3.2.0 BETA1 and I call channel.write() within my own code, so this means serialization is actually performed by my own threads right?
	Would I still gain from adding more workers? Would it be better to delegate the serialization to netty's I/O workers?


Any more suggestions?

Thanks

Best Regards
David



	

> HTH,
> Trustin
> 
> dralves wrote:
>> Hi 
>> 
>>        I've built a large scale application using netty and the unfair
>> writer/reader ratio is getting problematic. 
>>        I actually need 3 client machines to flood one server machine, which
>> means that to test scaling to 100 nodes I actually need 300 more (which is
>> very expensive in EC2 :). 
>>        If I can get this ratio even to 1.5/1 (from 3/1)  that would already
>> be an enormous progress and would allow me to continue my work (my ultimate
>> goal is to run a 1000 node cluster). 
>> 
>>        My setup: Each client runs several threads each thread has its own
>> set of connections to the servers (to avoid unecessary contention on some
>> bottlenecks) 
>>        Each thread I has its own set of netty pipelines (one for each
>> different server and for each "stream" within that server up to about 32
>> different "streams" split across 3-4 different servers). I did this for ease
>> of abstraction (clients simply request connections to streams, disregarding
>> where they are). 
>>        For this particular test (more of an I/O test) clients do mostly
>> nothing except serialization and socket writes (objects are kept in a pool
>> so no object creation overheads and serialization is very simple where each
>> object knows how to write and read itself from a DataOutput/DataInput) 
>> 
>>        Servers handle great (even when flooded i.e. no more net I/O in)
>> they maintain stable load (about 65%) 
>>        Clients require more cpu (about 75% each) and I actually need three
>> whole client machines to flood one server machine. 
>> 
>>        I've tested several configurations tuning the buffersize on the
>> client and server side as well as other parameters. Found the optimal
>> configuration but my problem didn't go away. 
>> 
>>        I must be doing something wrong. Any pointers? 
>> 
>>        Some specific doubts: 
>>        - Should I make sure that "streams" of the same client to the same
>> server use the same pipeline? 
>>        - Can I make serialization any faster (I'm using
>> Channels.dynamicBuffer())? 
>>        - the serialization is performed on a OneToOneEncoder this is ran by
>> my own threads right? not by Netty's client I/O workers? 
>> 
>>        On a more positive note, when configured for latency (small batch
>> flushes and tcpnodelay on) Netty handles great (total cluster throughput of
>> about 2.5 GB/sec and latencies of <0.5 sec :) 
>> 
>> Any help would be greatly appreciated 
>> Best Regards 
>> David Alves 
> 
> -- 
> what we call human nature in actuality is human habit
> http://gleamynode.net/
> 
> 
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/netty-users/attachments/20100324/6381d4f9/attachment-0001.html