Google Protocol Buffers integration is ready.

Thu Jul 23 02:52:57 EDT 2009

Hi Tomasz,

see my comment inline.

On Thu, Jul 23, 2009 at 12:21 AM, Tomasz
Blachowicz<tblachowicz at gmail.com> wrote:
>
> Hi Christian,
>
> I agree with you when we are speaking about HTTP alike network protocols
> that we can see almost everywhere. However bear in mind that Google Protocol
> Buffers are not replacement of HTTP os similar protocols. To me protobuf is
> a tool to quickly build binary protocols, which I highly optimized for
> speed, are compact and interoperable, hence python and C support. Excuse me
^^^^^^^^^
exactly. I've written why using this variable-length datatypes for the
message length should be quite sure slower than using a fixed length
type in my last mail.
Although I am not too convinced it is really a performance gain to use
sub-bytes and sub-integers for storing numbers (you know, the CPU
registers are 32 or 64 bit anyways, there is a reason why an int is of
that size), it may be the case if the information needs to be kept,
like the coordinates in your example. And even if its not performaning
better, it saves resources, which is good.
However, this is irrelevant to the message length because (a) it needs
to be converted into a java-readable data type anyway so Netty can
read that many bytes from the socket and (b) the information is
temporary, as soon as you've got the message you really don't care
abut the msg length anymore.

What Trustin did was IMO the most obvious and efficient way, using
this varint would be IMO a "golden hammer" approach.

regards,
christian!

> that pitch, I'm not related to Google anyway ;)
>
> The one particular feature of protobuf is that the messages are as small as
> it is possible, each byte can be saved is saved. This is why they utilize
> the concept of so called varints. If we know how to read varints from the
> wire why not to follow the principle and encode the frame size as varint and
> save some bytes?
>
> Here is the real life example, where it might matter. Imagine a massive
> online multiplayer game. The client written in C is displaying the space
> where we might have, say a 10 space ships controlled by other players. Each
> player sends constant stream of updates to the server on the location of her
> spacecraft. At the same time, each player's client application reads or is
> being notified about location of other space vehicles. In this trivialized
> example we can see how many tiny messages might be exchanged between
> multiple concurrent clients and a server. Say, each message contains x, y
> and z coordinates expressed as integer numbers ranging from 0 - 2047. That
> means that the message encoded as protobuf binary would contain exactly 3 x
> 2 = 6 bytes. If we'd use fixed integer represented as 4 bytes the entire
> frame become 6 + 4 = 10 bytes. But if we would use varint32 the size can be
> encoded as a single byte, so the entire frame is only 6 + 1 = 7 bytes. The
> ratio is 0.42, which means a lot to me if only we'd notice that there might
> be hundreds of millions of such messages being send and received.
>
> The Google Protocol Buffers were also designed to use varints to express the
> length a message. Take a look at the AbstractMessage.delimitedTo method
> (http://code.google.com/p/protobuf/source/browse/trunk/java/src/main/java/com/google/protobuf/AbstractMessage.java):
>
> <code>
>  public void writeDelimitedTo(OutputStream output) throws IOException {
>    CodedOutputStream codedOutput = CodedOutputStream.newInstance(output);
>    codedOutput.writeRawVarint32(getSerializedSize());
>    writeTo(codedOutput);
>    codedOutput.flush();
>  }
> </code>
>
> I'm aware that my explanation is a bit lengthy, but I simply wanted to make
> my point as clear as I can. I hope I have made already.
>
> Cheers,
> Tom
>
>
> christian wrote:
>>
>> I, personally, am not convinced that this would be an improvement.
>> Reading four bytes and then being able to read the message should
>> perform much better then reading a byte, do some (albeit fast binary)
>> checks, potentially read some more bytes and then calculate the
>> message length from it.
>> I know it is apples and oranges, but with the nowadays much abused
>> HTTP protocol you haven't even read the "protocol id header" with 4
>> bytes...
>>
>>
>> christian!
>>
>>
>> On Wed, Jul 22, 2009 at 7:09 PM, Tomasz Blachowicz<tblachowicz at gmail.com>
>> wrote:
>>>
>>>
>>> Trustin Lee wrote:
>>>>
>>>> I have finished the implementation of Google Protocol Buffers codec for
>>>> Netty.
>>>>
>>>> Google Protocol Buffers (protobuf) are a way of encoding structured data
>>>> in
>>>> an efficient yet extensible format. Google uses Protocol Buffers for
>>>> almost
>>>> all of its internal RPC protocols and file formats.  With protobuf, you
>>>> can
>>>> define a binary protocol very quickly and efficiently.  For more
>>>> information, please see here: http://code.google.com/p/protobuf/
>>>>
>>>> With the ProtobufEncoder and ProtobufDecoder that Netty provides, you
>>>> can
>>>> now write a network application with a very efficient binary encoding
>>>> even
>>>> more quickly than ever.  It just takes little time to implement a codec
>>>> with
>>>> protobuf.
>>>>
>>>> To help your understanding, I have written a time client / server
>>>> example.
>>>> The time client sends a list of time zones, and then the time server
>>>> responds with the local time for each time zone.  Please brwose the
>>>> following directory:
>>>>
>>>>    -
>>>>
>>>> http://fisheye.jboss.org/browse/Netty/trunk/src/main/java/org/jboss/netty/example/localtime
>>>>
>>>> Please note that LocalTimeProtocol.java has been generated from
>>>> LocalTimeProtocol.proto by protobuf compiler.
>>>>
>>>> Please feel free to send me a feed back once you review the source code.
>>>> I'd like to know what could be improved to support protobuf better and
>>>> to
>>>> help you implement an efficient binary protocol more quickly than ever.
>>>>
>>>
>>> Here is an idea for improvement of Google Protocol Buffers (protobuf)
>>> support in Netty. Currently each frame is prepended with fixed size four
>>> byte integer value that is a length of the message. Why not to use cool
>>> protobuf varint
>>> (http://code.google.com/apis/protocolbuffers/docs/encoding.html#varints)
>>> for
>>> that purpose? By doing this for small messages one could save a few bits
>>> of
>>> bandwidth. For instance, messages smaller that 128 bytes can be "framed"
>>> only by one byte representing the size.
>>>
>>> I've already implmented FrameDecoded and FieldPrepender that I could
>>> share
>>> with you through JIRA. Please, let me know what you think and if you'd
>>> considered such feature for Netty.
>>>
>>> Cheers,
>>> Tom
>>>
>>> --
>>> View this message in context:
>>> http://n2.nabble.com/Google-Protocol-Buffers-integration-is-ready.-tp2122078p3304575.html
>>> Sent from the Netty Developer Group mailing list archive at Nabble.com.
>>> _______________________________________________
>>> netty-dev mailing list
>>> netty-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/netty-dev
>>>
>>
>> _______________________________________________
>> netty-dev mailing list
>> netty-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/netty-dev
>>
>>
>
> --
> View this message in context: http://n2.nabble.com/Google-Protocol-Buffers-integration-is-ready.-tp2122078p3306268.html
> Sent from the Netty Developer Group mailing list archive at Nabble.com.
>
> _______________________________________________
> netty-dev mailing list
> netty-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-dev
>