Google Protocol Buffers integration is ready.

Tomasz Blachowicz tblachowicz at gmail.com
Fri Jul 24 10:55:12 EDT 2009 

Hi Trustin,

I've just created https://jira.jboss.org/jira/browse/NETTY-203 with the
patch implementing the discussed feature. If you need anything more drop me
a line.

Cheers,
Tom


Trustin Lee wrote:
> 
> Hi Tomasz,
> 
> Thanks for the contribution first of all!
> 
> As you know ProtobufEncoder/Decoder does not enforce any header format.
>  It's entirely up to a user which length format to use.  Also, I am
> aware of the fact that people in the protobuf community prefers variant
> length header you mentioned.  That being said, I think it's a good idea
> to add the length prepender and relevant frame decoder in the
> org.jboss.netty.handler.codec.protobuf package.  Would you contribute
> some code?  I will include it in 3.2.0 - 3.1.0 is feature frozen already.
> 
> Cheers,
> Trustin
> 
> On 07/23/2009 06:26 PM, Tomasz Blachowicz wrote:
>> Hi Christian,
>> 
>> I can see your point about the speed and I agree. What I'm proposing here
>> is
>> option rather than replacement. Having two addition simple classes in
>> library I could decide whether to optimize for speed of processing or for
>> size of frames.
>> 
>> Cheers,
>> Tomasz
>> 
>> 
>> christian wrote:
>>> Hi Tomasz,
>>>
>>> see my comment inline.
>>>
>>> On Thu, Jul 23, 2009 at 12:21 AM, Tomasz
>>> Blachowicz<tblachowicz at gmail.com> wrote:
>>>> Hi Christian,
>>>>
>>>> I agree with you when we are speaking about HTTP alike network
>>>> protocols
>>>> that we can see almost everywhere. However bear in mind that Google
>>>> Protocol
>>>> Buffers are not replacement of HTTP os similar protocols. To me
>>>> protobuf
>>>> is
>>>> a tool to quickly build binary protocols, which I highly optimized for
>>>> speed, are compact and interoperable, hence python and C support.
>>>> Excuse
>>>> me
>>> ^^^^^^^^^
>>> exactly. I've written why using this variable-length datatypes for the
>>> message length should be quite sure slower than using a fixed length
>>> type in my last mail.
>>> Although I am not too convinced it is really a performance gain to use
>>> sub-bytes and sub-integers for storing numbers (you know, the CPU
>>> registers are 32 or 64 bit anyways, there is a reason why an int is of
>>> that size), it may be the case if the information needs to be kept,
>>> like the coordinates in your example. And even if its not performaning
>>> better, it saves resources, which is good.
>>> However, this is irrelevant to the message length because (a) it needs
>>> to be converted into a java-readable data type anyway so Netty can
>>> read that many bytes from the socket and (b) the information is
>>> temporary, as soon as you've got the message you really don't care
>>> abut the msg length anymore.
>>>
>>> What Trustin did was IMO the most obvious and efficient way, using
>>> this varint would be IMO a "golden hammer" approach.
>>>
>>> regards,
>>> christian!
>>>
>>>
>>>> that pitch, I'm not related to Google anyway ;)
>>>>
>>>> The one particular feature of protobuf is that the messages are as
>>>> small
>>>> as
>>>> it is possible, each byte can be saved is saved. This is why they
>>>> utilize
>>>> the concept of so called varints. If we know how to read varints from
>>>> the
>>>> wire why not to follow the principle and encode the frame size as
>>>> varint
>>>> and
>>>> save some bytes?
>>>>
>>>> Here is the real life example, where it might matter. Imagine a massive
>>>> online multiplayer game. The client written in C is displaying the
>>>> space
>>>> where we might have, say a 10 space ships controlled by other players.
>>>> Each
>>>> player sends constant stream of updates to the server on the location
>>>> of
>>>> her
>>>> spacecraft. At the same time, each player's client application reads or
>>>> is
>>>> being notified about location of other space vehicles. In this
>>>> trivialized
>>>> example we can see how many tiny messages might be exchanged between
>>>> multiple concurrent clients and a server. Say, each message contains x,
>>>> y
>>>> and z coordinates expressed as integer numbers ranging from 0 - 2047.
>>>> That
>>>> means that the message encoded as protobuf binary would contain exactly
>>>> 3
>>>> x
>>>> 2 = 6 bytes. If we'd use fixed integer represented as 4 bytes the
>>>> entire
>>>> frame become 6 + 4 = 10 bytes. But if we would use varint32 the size
>>>> can
>>>> be
>>>> encoded as a single byte, so the entire frame is only 6 + 1 = 7 bytes.
>>>> The
>>>> ratio is 0.42, which means a lot to me if only we'd notice that there
>>>> might
>>>> be hundreds of millions of such messages being send and received.
>>>>
>>>> The Google Protocol Buffers were also designed to use varints to
>>>> express
>>>> the
>>>> length a message. Take a look at the AbstractMessage.delimitedTo method
>>>> (http://code.google.com/p/protobuf/source/browse/trunk/java/src/main/java/com/google/protobuf/AbstractMessage.java):
>>>>
>>>> <code>
>>>>  public void writeDelimitedTo(OutputStream output) throws IOException {
>>>>    CodedOutputStream codedOutput =
>>>> CodedOutputStream.newInstance(output);
>>>>    codedOutput.writeRawVarint32(getSerializedSize());
>>>>    writeTo(codedOutput);
>>>>    codedOutput.flush();
>>>>  }
>>>> </code>
>>>>
>>>> I'm aware that my explanation is a bit lengthy, but I simply wanted to
>>>> make
>>>> my point as clear as I can. I hope I have made already.
>>>>
>>>> Cheers,
>>>> Tom
>>>>
>>>>
>>>> christian wrote:
>>>>> I, personally, am not convinced that this would be an improvement.
>>>>> Reading four bytes and then being able to read the message should
>>>>> perform much better then reading a byte, do some (albeit fast binary)
>>>>> checks, potentially read some more bytes and then calculate the
>>>>> message length from it.
>>>>> I know it is apples and oranges, but with the nowadays much abused
>>>>> HTTP protocol you haven't even read the "protocol id header" with 4
>>>>> bytes...
>>>>>
>>>>>
>>>>> christian!
>>>>>
>>>>>
>>>>> On Wed, Jul 22, 2009 at 7:09 PM, Tomasz
>>>>> Blachowicz<tblachowicz at gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Trustin Lee wrote:
>>>>>>> I have finished the implementation of Google Protocol Buffers codec
>>>>>>> for
>>>>>>> Netty.
>>>>>>>
>>>>>>> Google Protocol Buffers (protobuf) are a way of encoding structured
>>>>>>> data
>>>>>>> in
>>>>>>> an efficient yet extensible format. Google uses Protocol Buffers for
>>>>>>> almost
>>>>>>> all of its internal RPC protocols and file formats.  With protobuf,
>>>>>>> you
>>>>>>> can
>>>>>>> define a binary protocol very quickly and efficiently.  For more
>>>>>>> information, please see here: http://code.google.com/p/protobuf/
>>>>>>>
>>>>>>> With the ProtobufEncoder and ProtobufDecoder that Netty provides,
>>>>>>> you
>>>>>>> can
>>>>>>> now write a network application with a very efficient binary
>>>>>>> encoding
>>>>>>> even
>>>>>>> more quickly than ever.  It just takes little time to implement a
>>>>>>> codec
>>>>>>> with
>>>>>>> protobuf.
>>>>>>>
>>>>>>> To help your understanding, I have written a time client / server
>>>>>>> example.
>>>>>>> The time client sends a list of time zones, and then the time server
>>>>>>> responds with the local time for each time zone.  Please brwose the
>>>>>>> following directory:
>>>>>>>
>>>>>>>    -
>>>>>>>
>>>>>>> http://fisheye.jboss.org/browse/Netty/trunk/src/main/java/org/jboss/netty/example/localtime
>>>>>>>
>>>>>>> Please note that LocalTimeProtocol.java has been generated from
>>>>>>> LocalTimeProtocol.proto by protobuf compiler.
>>>>>>>
>>>>>>> Please feel free to send me a feed back once you review the source
>>>>>>> code.
>>>>>>> I'd like to know what could be improved to support protobuf better
>>>>>>> and
>>>>>>> to
>>>>>>> help you implement an efficient binary protocol more quickly than
>>>>>>> ever.
>>>>>>>
>>>>>> Here is an idea for improvement of Google Protocol Buffers (protobuf)
>>>>>> support in Netty. Currently each frame is prepended with fixed size
>>>>>> four
>>>>>> byte integer value that is a length of the message. Why not to use
>>>>>> cool
>>>>>> protobuf varint
>>>>>> (http://code.google.com/apis/protocolbuffers/docs/encoding.html#varints)
>>>>>> for
>>>>>> that purpose? By doing this for small messages one could save a few
>>>>>> bits
>>>>>> of
>>>>>> bandwidth. For instance, messages smaller that 128 bytes can be
>>>>>> "framed"
>>>>>> only by one byte representing the size.
>>>>>>
>>>>>> I've already implmented FrameDecoded and FieldPrepender that I could
>>>>>> share
>>>>>> with you through JIRA. Please, let me know what you think and if
>>>>>> you'd
>>>>>> considered such feature for Netty.
>>>>>>
>>>>>> Cheers,
>>>>>> Tom
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://n2.nabble.com/Google-Protocol-Buffers-integration-is-ready.-tp2122078p3304575.html
>>>>>> Sent from the Netty Developer Group mailing list archive at
>>>>>> Nabble.com.
>>>>>> _______________________________________________
>>>>>> netty-dev mailing list
>>>>>> netty-dev at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/netty-dev
>>>>>>
>>>>> _______________________________________________
>>>>> netty-dev mailing list
>>>>> netty-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/netty-dev
>>>>>
>>>>>
>>>> --
>>>> View this message in context:
>>>> http://n2.nabble.com/Google-Protocol-Buffers-integration-is-ready.-tp2122078p3306268.html
>>>> Sent from the Netty Developer Group mailing list archive at Nabble.com.
>>>>
>>>> _______________________________________________
>>>> netty-dev mailing list
>>>> netty-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/netty-dev
>>>>
>>> _______________________________________________
>>> netty-dev mailing list
>>> netty-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/netty-dev
>>>
>>>
>> 
> 
> _______________________________________________
> netty-dev mailing list
> netty-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-dev
> 
> 

-- 
View this message in context: http://n2.nabble.com/Google-Protocol-Buffers-integration-is-ready.-tp2122078p3316090.html
Sent from the Netty Developer Group mailing list archive at Nabble.com.

More information about the netty-dev mailing list