<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi Trustin<div><br></div><div><div><blockquote type="cite"><div>If I understood correctly, the problem is that the client takes 3 times<br>more CPU than the server for encoding, right?<br></div></blockquote><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Actually more than that, one server is stable at 65% and I need three clients (that maintain a stable load at 75%) to flood the ethernet I/O. (Get to the point that if I add more clients the throughput for one server doesn't increase).</div><br><blockquote type="cite"><div><br><blockquote type="cite">- Should I make sure that "streams" of the same client to the same<br></blockquote><blockquote type="cite"> server use the same pipeline?<br></blockquote><br>I don't think so.<br></div></blockquote><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>My question was actually If I should use the same channel (i.e. async socket connection) or keep the +- 12 connections (channels) from a single client to a single server (server usually has at most three clients, i.e. 36 connections, and clients usually send to three different servers, ie. 36 connections).</div><br><blockquote type="cite"><div><br><blockquote type="cite">- Can I make serialization any faster (I'm using<br></blockquote><blockquote type="cite"> Channels.dynamicBuffer())?<br></blockquote><br>I guess you already tuned the dynamic buffer creation by specifying<br>estimated message size, right? If not, try to specify better default<br>when you create a dynamic buffer to reduce the number of expansion or<br>unnecessary allocation.<br></div></blockquote><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>You are right, I already specified the size, and I actually update the estimated length based on the size of the previous tuple, as tuples are mostly the same size.</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>I.e. by (where bout is the buffer created by Channels.dynamicBuffer():</div><div><span class="Apple-tab-span" style="white-space:pre"> </span></div><div><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-style-span" style="font-family: Monaco; font-size: 11px; ">ChannelBuffer encoded = bout.buffer();</span></div><div><span class="Apple-style-span" style="font-family: Monaco; font-size: 11px; "> encoded.setInt(0, encoded.writerIndex() - 4);</span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "> <span style="color: #931a68">if</span> (<span style="color: #931a68">this</span>.<span style="color: #0326cc">estimatedLength</span> != bout.buffer().writerIndex()) {</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "><span style="color: #931a68"><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" color="#000000"> </font></span>this</span>.<span style="color: #0326cc">estimatedLength</span> = bout.buffer().writerIndex();</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "> }</div><br><blockquote type="cite"><div><br><blockquote type="cite">- the serialization is performed on a OneToOneEncoder this is ran by<br></blockquote><blockquote type="cite"> my own threads right? not by Netty's client I/O workers?<br></blockquote><br>The serialization is performed in the I/O worker thread if you called<br>write() in your handler and you did not insert an ExecutionHandler to<br>the pipeline. If the number of connections is small, I'd recommend you<br>to increase the number of I/O worker threads by specifying an integer to<br>NioClient/ServerSocketChannelFactory constructor calls. As of Netty<br>3.2.0.BETA1, the default has been doubled for better out-of-the-box<br>performance.<br><br></div></blockquote><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>I'm already using 3.2.0 BETA1 and I call channel.write() within my own code, so this means serialization is actually performed by my own threads right?</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Would I still gain from adding more workers? Would it be better to delegate the serialization to netty's I/O workers?</div><div><br></div><div><br></div><div>Any more suggestions?</div><div><br></div><div>Thanks</div><div><br></div><div>Best Regards</div><div>David</div><div><br></div><div><br></div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span></div><br><blockquote type="cite"><div>HTH,<br>Trustin<br><br>dralves wrote:<br><blockquote type="cite">Hi <br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> I've built a large scale application using netty and the unfair<br></blockquote><blockquote type="cite">writer/reader ratio is getting problematic. <br></blockquote><blockquote type="cite"> I actually need 3 client machines to flood one server machine, which<br></blockquote><blockquote type="cite">means that to test scaling to 100 nodes I actually need 300 more (which is<br></blockquote><blockquote type="cite">very expensive in EC2 :). <br></blockquote><blockquote type="cite"> If I can get this ratio even to 1.5/1 (from 3/1) that would already<br></blockquote><blockquote type="cite">be an enormous progress and would allow me to continue my work (my ultimate<br></blockquote><blockquote type="cite">goal is to run a 1000 node cluster). <br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> My setup: Each client runs several threads each thread has its own<br></blockquote><blockquote type="cite">set of connections to the servers (to avoid unecessary contention on some<br></blockquote><blockquote type="cite">bottlenecks) <br></blockquote><blockquote type="cite"> Each thread I has its own set of netty pipelines (one for each<br></blockquote><blockquote type="cite">different server and for each "stream" within that server up to about 32<br></blockquote><blockquote type="cite">different "streams" split across 3-4 different servers). I did this for ease<br></blockquote><blockquote type="cite">of abstraction (clients simply request connections to streams, disregarding<br></blockquote><blockquote type="cite">where they are). <br></blockquote><blockquote type="cite"> For this particular test (more of an I/O test) clients do mostly<br></blockquote><blockquote type="cite">nothing except serialization and socket writes (objects are kept in a pool<br></blockquote><blockquote type="cite">so no object creation overheads and serialization is very simple where each<br></blockquote><blockquote type="cite">object knows how to write and read itself from a DataOutput/DataInput) <br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> Servers handle great (even when flooded i.e. no more net I/O in)<br></blockquote><blockquote type="cite">they maintain stable load (about 65%) <br></blockquote><blockquote type="cite"> Clients require more cpu (about 75% each) and I actually need three<br></blockquote><blockquote type="cite">whole client machines to flood one server machine. <br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> I've tested several configurations tuning the buffersize on the<br></blockquote><blockquote type="cite">client and server side as well as other parameters. Found the optimal<br></blockquote><blockquote type="cite">configuration but my problem didn't go away. <br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> I must be doing something wrong. Any pointers? <br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> Some specific doubts: <br></blockquote><blockquote type="cite"> - Should I make sure that "streams" of the same client to the same<br></blockquote><blockquote type="cite">server use the same pipeline? <br></blockquote><blockquote type="cite"> - Can I make serialization any faster (I'm using<br></blockquote><blockquote type="cite">Channels.dynamicBuffer())? <br></blockquote><blockquote type="cite"> - the serialization is performed on a OneToOneEncoder this is ran by<br></blockquote><blockquote type="cite">my own threads right? not by Netty's client I/O workers? <br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> On a more positive note, when configured for latency (small batch<br></blockquote><blockquote type="cite">flushes and tcpnodelay on) Netty handles great (total cluster throughput of<br></blockquote><blockquote type="cite">about 2.5 GB/sec and latencies of <0.5 sec :) <br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Any help would be greatly appreciated <br></blockquote><blockquote type="cite">Best Regards <br></blockquote><blockquote type="cite">David Alves <br></blockquote><br>-- <br>what we call human nature in actuality is human habit<br><a href="http://gleamynode.net/">http://gleamynode.net/</a><br><br><br>_______________________________________________<br>netty-users mailing list<br>netty-users@lists.jboss.org<br>https://lists.jboss.org/mailman/listinfo/netty-users</div></blockquote></div><br></div></body></html>