Google Protocol Buffers integration is ready.

[Tomasz Blachowicz]

Hi Christian,

I agree with you when we are speaking about HTTP alike network protocols
that we can see almost everywhere. However bear in mind that Google Protocol
Buffers are not replacement of HTTP os similar protocols. To me protobuf is
a tool to quickly build binary protocols, which I highly optimized for
speed, are compact and interoperable, hence python and C support. Excuse me
that pitch, I'm not related to Google anyway ;)

The one particular feature of protobuf is that the messages are as small as
it is possible, each byte can be saved is saved. This is why they utilize
the concept of so called varints. If we know how to read varints from the
wire why not to follow the principle and encode the frame size as varint and
save some bytes?

Here is the real life example, where it might matter. Imagine a massive
online multiplayer game. The client written in C is displaying the space
where we might have, say a 10 space ships controlled by other players. Each
player sends constant stream of updates to the server on the location of her
spacecraft. At the same time, each player's client application reads or is
being notified about location of other space vehicles. In this trivialized
example we can see how many tiny messages might be exchanged between
multiple concurrent clients and a server. Say, each message contains x, y
and z coordinates expressed as integer numbers ranging from 0 - 2047. That
means that the message encoded as protobuf binary would contain exactly 3 x
2 = 6 bytes. If we'd use fixed integer represented as 4 bytes the entire
frame become 6 + 4 = 10 bytes. But if we would use varint32 the size can be
encoded as a single byte, so the entire frame is only 6 + 1 = 7 bytes. The
ratio is 0.42, which means a lot to me if only we'd notice that there might
be hundreds of millions of such messages being send and received.

The Google Protocol Buffers were also designed to use varints to express the
length a message. Take a look at the AbstractMessage.delimitedTo method
(http://code.google.com/p/protobuf/source/browse/trunk/java/src/main/java/com/google/protobuf/AbstractMessage.java):

<code>
  public void writeDelimitedTo(OutputStream output) throws IOException {
    CodedOutputStream codedOutput = CodedOutputStream.newInstance(output);
    codedOutput.writeRawVarint32(getSerializedSize());
    writeTo(codedOutput);
    codedOutput.flush();
  }
</code>

I'm aware that my explanation is a bit lengthy, but I simply wanted to make
my point as clear as I can. I hope I have made already. 

Cheers,
Tom


christian wrote:
> 
> I, personally, am not convinced that this would be an improvement.
> Reading four bytes and then being able to read the message should
> perform much better then reading a byte, do some (albeit fast binary)
> checks, potentially read some more bytes and then calculate the
> message length from it.
> I know it is apples and oranges, but with the nowadays much abused
> HTTP protocol you haven't even read the "protocol id header" with 4
> bytes...
> 
> 
> christian!
> 
> 
> On Wed, Jul 22, 2009 at 7:09 PM, Tomasz Blachowicz<tblachowicz at gmail.com>
> wrote:
>>
>>
>> Trustin Lee wrote:
>>>
>>> I have finished the implementation of Google Protocol Buffers codec for
>>> Netty.
>>>
>>> Google Protocol Buffers (protobuf) are a way of encoding structured data
>>> in
>>> an efficient yet extensible format. Google uses Protocol Buffers for
>>> almost
>>> all of its internal RPC protocols and file formats.  With protobuf, you
>>> can
>>> define a binary protocol very quickly and efficiently.  For more
>>> information, please see here: http://code.google.com/p/protobuf/
>>>
>>> With the ProtobufEncoder and ProtobufDecoder that Netty provides, you
>>> can
>>> now write a network application with a very efficient binary encoding
>>> even
>>> more quickly than ever.  It just takes little time to implement a codec
>>> with
>>> protobuf.
>>>
>>> To help your understanding, I have written a time client / server
>>> example.
>>> The time client sends a list of time zones, and then the time server
>>> responds with the local time for each time zone.  Please brwose the
>>> following directory:
>>>
>>>    -
>>>
>>> http://fisheye.jboss.org/browse/Netty/trunk/src/main/java/org/jboss/netty/example/localtime
>>>
>>> Please note that LocalTimeProtocol.java has been generated from
>>> LocalTimeProtocol.proto by protobuf compiler.
>>>
>>> Please feel free to send me a feed back once you review the source code.
>>> I'd like to know what could be improved to support protobuf better and
>>> to
>>> help you implement an efficient binary protocol more quickly than ever.
>>>
>>
>> Here is an idea for improvement of Google Protocol Buffers (protobuf)
>> support in Netty. Currently each frame is prepended with fixed size four
>> byte integer value that is a length of the message. Why not to use cool
>> protobuf varint
>> (http://code.google.com/apis/protocolbuffers/docs/encoding.html#varints)
>> for
>> that purpose? By doing this for small messages one could save a few bits
>> of
>> bandwidth. For instance, messages smaller that 128 bytes can be "framed"
>> only by one byte representing the size.
>>
>> I've already implmented FrameDecoded and FieldPrepender that I could
>> share
>> with you through JIRA. Please, let me know what you think and if you'd
>> considered such feature for Netty.
>>
>> Cheers,
>> Tom
>>
>> --
>> View this message in context:
>> http://n2.nabble.com/Google-Protocol-Buffers-integration-is-ready.-tp2122078p3304575.html
>> Sent from the Netty Developer Group mailing list archive at Nabble.com.
>> _______________________________________________
>> netty-dev mailing list
>> netty-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/netty-dev
>>
> 
> _______________________________________________
> netty-dev mailing list
> netty-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-dev
> 
> 

-- 
View this message in context: http://n2.nabble.com/Google-Protocol-Buffers-integration-is-ready.-tp2122078p3306268.html
Sent from the Netty Developer Group mailing list archive at Nabble.com.