Maximum Performance: Netty with Protocol Buffers

[Marc-André Laverdière]

Hello,

I'm replying inline...
Marc-André LAVERDIÈRE
"Perseverance must finish its work so that you may be mature and complete,
not lacking anything." -James 1:4
http://asimplediscipleslife.blogspot.com/
mlaverd.theunixplace.com





On Thu, Sep 1, 2011 at 7:58 PM, Replicator <webakaunt at gmail.com> wrote:
> We are going through evaluation of Netty and ZeroMQ to see which one makes
> more sense for our task:
>
>    1. Mostly a one way communication => 99.999% of messages are coming from
> A (C++) to B (Java) through the socket.
>
>    2. A single message is roughly 120 bytes
>
>    3. Need to consume 200,000 messages a second ( given a Gigabit network )
>
>    4. Will use google protobufs to set a transparent protocol between a C++
> producer and a Java consumer
>
> Questions for the Netty community:
>
>    1. A consumer side, don't really need to send messages to the producer
> => it just needs to consume. What would be the best way to simulate this use
> case? An Echo server (but then a consumer constantly sending messages)? A
> streaming server, some other setup?
I think you can just have a simple server code from any of the
examples, and add one handler to record and output your performance
data.
>
>    2. When using protocol buffers [ we did play with netty's LocalTime
> example ], does it make sense to set FixedReceiveBufferSizePredictor /
> AdaptiveReceiveBufferSizePredictor to change the buffer size, or since a
> protocol would dictate a message length, the above parameters would not make
> sense?

I guess it depends if you are sending the data and then disconnecting
the socket. In that case, you don't need to prepend the length.
However, if you are keeping it open, then having a length parameter
really helps.

>
>    3. We tried an example posted here on a mailing list: http://d.pr/SLYr
> and we only see a steady 35,000 messages per second rate, and that is
> without protocol buffers, or any major message parsing [ just getting first
> byte of the message on arrival ]. It probably relates to the question #1,
> but is there a way to improve this example?
I didn'thave the time to look at your code sample... but I'd say that
you can use NIO + ExecutionHandler + MATPE to scale out of many many
such problems.

>
>    4. Since we are after the maximum throughput, would it make sense to use
> a chunking functionality of Netty ( e.g. ChunkedWriteHandler, ChunkedInput,
> etc.. ). If yes, are there any examples on how it may be applicable to
> transferring large chunks, and then parsing messages out? Is there a way to
> fetch uneven sized chunks that e.g. would stretch not to chop the last
> message?
I don't know about that, I never used it.
>
> Thank you!
>
> --
> View this message in context: http://netty-forums-and-mailing-lists.685743.n2.nabble.com/Maximum-Performance-Netty-with-Protocol-Buffers-tp6750165p6750165.html
> Sent from the Netty User Group mailing list archive at Nabble.com.
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users
>