Google Protocol Buffers integration is ready.
Tomasz Blachowicz
tblachowicz at gmail.com
Thu Jul 23 05:26:20 EDT 2009
Hi Christian,
I can see your point about the speed and I agree. What I'm proposing here is
option rather than replacement. Having two addition simple classes in
library I could decide whether to optimize for speed of processing or for
size of frames.
Cheers,
Tomasz
christian wrote:
>
> Hi Tomasz,
>
> see my comment inline.
>
> On Thu, Jul 23, 2009 at 12:21 AM, Tomasz
> Blachowicz<tblachowicz at gmail.com> wrote:
>>
>> Hi Christian,
>>
>> I agree with you when we are speaking about HTTP alike network protocols
>> that we can see almost everywhere. However bear in mind that Google
>> Protocol
>> Buffers are not replacement of HTTP os similar protocols. To me protobuf
>> is
>> a tool to quickly build binary protocols, which I highly optimized for
>> speed, are compact and interoperable, hence python and C support. Excuse
>> me
> ^^^^^^^^^
> exactly. I've written why using this variable-length datatypes for the
> message length should be quite sure slower than using a fixed length
> type in my last mail.
> Although I am not too convinced it is really a performance gain to use
> sub-bytes and sub-integers for storing numbers (you know, the CPU
> registers are 32 or 64 bit anyways, there is a reason why an int is of
> that size), it may be the case if the information needs to be kept,
> like the coordinates in your example. And even if its not performaning
> better, it saves resources, which is good.
> However, this is irrelevant to the message length because (a) it needs
> to be converted into a java-readable data type anyway so Netty can
> read that many bytes from the socket and (b) the information is
> temporary, as soon as you've got the message you really don't care
> abut the msg length anymore.
>
> What Trustin did was IMO the most obvious and efficient way, using
> this varint would be IMO a "golden hammer" approach.
>
> regards,
> christian!
>
>
>> that pitch, I'm not related to Google anyway ;)
>>
>> The one particular feature of protobuf is that the messages are as small
>> as
>> it is possible, each byte can be saved is saved. This is why they utilize
>> the concept of so called varints. If we know how to read varints from the
>> wire why not to follow the principle and encode the frame size as varint
>> and
>> save some bytes?
>>
>> Here is the real life example, where it might matter. Imagine a massive
>> online multiplayer game. The client written in C is displaying the space
>> where we might have, say a 10 space ships controlled by other players.
>> Each
>> player sends constant stream of updates to the server on the location of
>> her
>> spacecraft. At the same time, each player's client application reads or
>> is
>> being notified about location of other space vehicles. In this
>> trivialized
>> example we can see how many tiny messages might be exchanged between
>> multiple concurrent clients and a server. Say, each message contains x, y
>> and z coordinates expressed as integer numbers ranging from 0 - 2047.
>> That
>> means that the message encoded as protobuf binary would contain exactly 3
>> x
>> 2 = 6 bytes. If we'd use fixed integer represented as 4 bytes the entire
>> frame become 6 + 4 = 10 bytes. But if we would use varint32 the size can
>> be
>> encoded as a single byte, so the entire frame is only 6 + 1 = 7 bytes.
>> The
>> ratio is 0.42, which means a lot to me if only we'd notice that there
>> might
>> be hundreds of millions of such messages being send and received.
>>
>> The Google Protocol Buffers were also designed to use varints to express
>> the
>> length a message. Take a look at the AbstractMessage.delimitedTo method
>> (http://code.google.com/p/protobuf/source/browse/trunk/java/src/main/java/com/google/protobuf/AbstractMessage.java):
>>
>> <code>
>> public void writeDelimitedTo(OutputStream output) throws IOException {
>> CodedOutputStream codedOutput = CodedOutputStream.newInstance(output);
>> codedOutput.writeRawVarint32(getSerializedSize());
>> writeTo(codedOutput);
>> codedOutput.flush();
>> }
>> </code>
>>
>> I'm aware that my explanation is a bit lengthy, but I simply wanted to
>> make
>> my point as clear as I can. I hope I have made already.
>>
>> Cheers,
>> Tom
>>
>>
>> christian wrote:
>>>
>>> I, personally, am not convinced that this would be an improvement.
>>> Reading four bytes and then being able to read the message should
>>> perform much better then reading a byte, do some (albeit fast binary)
>>> checks, potentially read some more bytes and then calculate the
>>> message length from it.
>>> I know it is apples and oranges, but with the nowadays much abused
>>> HTTP protocol you haven't even read the "protocol id header" with 4
>>> bytes...
>>>
>>>
>>> christian!
>>>
>>>
>>> On Wed, Jul 22, 2009 at 7:09 PM, Tomasz
>>> Blachowicz<tblachowicz at gmail.com>
>>> wrote:
>>>>
>>>>
>>>> Trustin Lee wrote:
>>>>>
>>>>> I have finished the implementation of Google Protocol Buffers codec
>>>>> for
>>>>> Netty.
>>>>>
>>>>> Google Protocol Buffers (protobuf) are a way of encoding structured
>>>>> data
>>>>> in
>>>>> an efficient yet extensible format. Google uses Protocol Buffers for
>>>>> almost
>>>>> all of its internal RPC protocols and file formats. With protobuf,
>>>>> you
>>>>> can
>>>>> define a binary protocol very quickly and efficiently. For more
>>>>> information, please see here: http://code.google.com/p/protobuf/
>>>>>
>>>>> With the ProtobufEncoder and ProtobufDecoder that Netty provides, you
>>>>> can
>>>>> now write a network application with a very efficient binary encoding
>>>>> even
>>>>> more quickly than ever. It just takes little time to implement a
>>>>> codec
>>>>> with
>>>>> protobuf.
>>>>>
>>>>> To help your understanding, I have written a time client / server
>>>>> example.
>>>>> The time client sends a list of time zones, and then the time server
>>>>> responds with the local time for each time zone. Please brwose the
>>>>> following directory:
>>>>>
>>>>> -
>>>>>
>>>>> http://fisheye.jboss.org/browse/Netty/trunk/src/main/java/org/jboss/netty/example/localtime
>>>>>
>>>>> Please note that LocalTimeProtocol.java has been generated from
>>>>> LocalTimeProtocol.proto by protobuf compiler.
>>>>>
>>>>> Please feel free to send me a feed back once you review the source
>>>>> code.
>>>>> I'd like to know what could be improved to support protobuf better and
>>>>> to
>>>>> help you implement an efficient binary protocol more quickly than
>>>>> ever.
>>>>>
>>>>
>>>> Here is an idea for improvement of Google Protocol Buffers (protobuf)
>>>> support in Netty. Currently each frame is prepended with fixed size
>>>> four
>>>> byte integer value that is a length of the message. Why not to use cool
>>>> protobuf varint
>>>> (http://code.google.com/apis/protocolbuffers/docs/encoding.html#varints)
>>>> for
>>>> that purpose? By doing this for small messages one could save a few
>>>> bits
>>>> of
>>>> bandwidth. For instance, messages smaller that 128 bytes can be
>>>> "framed"
>>>> only by one byte representing the size.
>>>>
>>>> I've already implmented FrameDecoded and FieldPrepender that I could
>>>> share
>>>> with you through JIRA. Please, let me know what you think and if you'd
>>>> considered such feature for Netty.
>>>>
>>>> Cheers,
>>>> Tom
>>>>
>>>> --
>>>> View this message in context:
>>>> http://n2.nabble.com/Google-Protocol-Buffers-integration-is-ready.-tp2122078p3304575.html
>>>> Sent from the Netty Developer Group mailing list archive at Nabble.com.
>>>> _______________________________________________
>>>> netty-dev mailing list
>>>> netty-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/netty-dev
>>>>
>>>
>>> _______________________________________________
>>> netty-dev mailing list
>>> netty-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/netty-dev
>>>
>>>
>>
>> --
>> View this message in context:
>> http://n2.nabble.com/Google-Protocol-Buffers-integration-is-ready.-tp2122078p3306268.html
>> Sent from the Netty Developer Group mailing list archive at Nabble.com.
>>
>> _______________________________________________
>> netty-dev mailing list
>> netty-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/netty-dev
>>
>
> _______________________________________________
> netty-dev mailing list
> netty-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-dev
>
>
--
View this message in context: http://n2.nabble.com/Google-Protocol-Buffers-integration-is-ready.-tp2122078p3308444.html
Sent from the Netty Developer Group mailing list archive at Nabble.com.
More information about the netty-dev
mailing list