Google Protocol Buffers integration is ready.
"이희승 (Trustin Lee)"
trustin at gmail.com
Thu Jul 23 22:31:09 EDT 2009
Hi Tomasz,
Thanks for the contribution first of all!
As you know ProtobufEncoder/Decoder does not enforce any header format.
It's entirely up to a user which length format to use. Also, I am
aware of the fact that people in the protobuf community prefers variant
length header you mentioned. That being said, I think it's a good idea
to add the length prepender and relevant frame decoder in the
org.jboss.netty.handler.codec.protobuf package. Would you contribute
some code? I will include it in 3.2.0 - 3.1.0 is feature frozen already.
Cheers,
Trustin
On 07/23/2009 06:26 PM, Tomasz Blachowicz wrote:
> Hi Christian,
>
> I can see your point about the speed and I agree. What I'm proposing here is
> option rather than replacement. Having two addition simple classes in
> library I could decide whether to optimize for speed of processing or for
> size of frames.
>
> Cheers,
> Tomasz
>
>
> christian wrote:
>> Hi Tomasz,
>>
>> see my comment inline.
>>
>> On Thu, Jul 23, 2009 at 12:21 AM, Tomasz
>> Blachowicz<tblachowicz at gmail.com> wrote:
>>> Hi Christian,
>>>
>>> I agree with you when we are speaking about HTTP alike network protocols
>>> that we can see almost everywhere. However bear in mind that Google
>>> Protocol
>>> Buffers are not replacement of HTTP os similar protocols. To me protobuf
>>> is
>>> a tool to quickly build binary protocols, which I highly optimized for
>>> speed, are compact and interoperable, hence python and C support. Excuse
>>> me
>> ^^^^^^^^^
>> exactly. I've written why using this variable-length datatypes for the
>> message length should be quite sure slower than using a fixed length
>> type in my last mail.
>> Although I am not too convinced it is really a performance gain to use
>> sub-bytes and sub-integers for storing numbers (you know, the CPU
>> registers are 32 or 64 bit anyways, there is a reason why an int is of
>> that size), it may be the case if the information needs to be kept,
>> like the coordinates in your example. And even if its not performaning
>> better, it saves resources, which is good.
>> However, this is irrelevant to the message length because (a) it needs
>> to be converted into a java-readable data type anyway so Netty can
>> read that many bytes from the socket and (b) the information is
>> temporary, as soon as you've got the message you really don't care
>> abut the msg length anymore.
>>
>> What Trustin did was IMO the most obvious and efficient way, using
>> this varint would be IMO a "golden hammer" approach.
>>
>> regards,
>> christian!
>>
>>
>>> that pitch, I'm not related to Google anyway ;)
>>>
>>> The one particular feature of protobuf is that the messages are as small
>>> as
>>> it is possible, each byte can be saved is saved. This is why they utilize
>>> the concept of so called varints. If we know how to read varints from the
>>> wire why not to follow the principle and encode the frame size as varint
>>> and
>>> save some bytes?
>>>
>>> Here is the real life example, where it might matter. Imagine a massive
>>> online multiplayer game. The client written in C is displaying the space
>>> where we might have, say a 10 space ships controlled by other players.
>>> Each
>>> player sends constant stream of updates to the server on the location of
>>> her
>>> spacecraft. At the same time, each player's client application reads or
>>> is
>>> being notified about location of other space vehicles. In this
>>> trivialized
>>> example we can see how many tiny messages might be exchanged between
>>> multiple concurrent clients and a server. Say, each message contains x, y
>>> and z coordinates expressed as integer numbers ranging from 0 - 2047.
>>> That
>>> means that the message encoded as protobuf binary would contain exactly 3
>>> x
>>> 2 = 6 bytes. If we'd use fixed integer represented as 4 bytes the entire
>>> frame become 6 + 4 = 10 bytes. But if we would use varint32 the size can
>>> be
>>> encoded as a single byte, so the entire frame is only 6 + 1 = 7 bytes.
>>> The
>>> ratio is 0.42, which means a lot to me if only we'd notice that there
>>> might
>>> be hundreds of millions of such messages being send and received.
>>>
>>> The Google Protocol Buffers were also designed to use varints to express
>>> the
>>> length a message. Take a look at the AbstractMessage.delimitedTo method
>>> (http://code.google.com/p/protobuf/source/browse/trunk/java/src/main/java/com/google/protobuf/AbstractMessage.java):
>>>
>>> <code>
>>> public void writeDelimitedTo(OutputStream output) throws IOException {
>>> CodedOutputStream codedOutput = CodedOutputStream.newInstance(output);
>>> codedOutput.writeRawVarint32(getSerializedSize());
>>> writeTo(codedOutput);
>>> codedOutput.flush();
>>> }
>>> </code>
>>>
>>> I'm aware that my explanation is a bit lengthy, but I simply wanted to
>>> make
>>> my point as clear as I can. I hope I have made already.
>>>
>>> Cheers,
>>> Tom
>>>
>>>
>>> christian wrote:
>>>> I, personally, am not convinced that this would be an improvement.
>>>> Reading four bytes and then being able to read the message should
>>>> perform much better then reading a byte, do some (albeit fast binary)
>>>> checks, potentially read some more bytes and then calculate the
>>>> message length from it.
>>>> I know it is apples and oranges, but with the nowadays much abused
>>>> HTTP protocol you haven't even read the "protocol id header" with 4
>>>> bytes...
>>>>
>>>>
>>>> christian!
>>>>
>>>>
>>>> On Wed, Jul 22, 2009 at 7:09 PM, Tomasz
>>>> Blachowicz<tblachowicz at gmail.com>
>>>> wrote:
>>>>>
>>>>> Trustin Lee wrote:
>>>>>> I have finished the implementation of Google Protocol Buffers codec
>>>>>> for
>>>>>> Netty.
>>>>>>
>>>>>> Google Protocol Buffers (protobuf) are a way of encoding structured
>>>>>> data
>>>>>> in
>>>>>> an efficient yet extensible format. Google uses Protocol Buffers for
>>>>>> almost
>>>>>> all of its internal RPC protocols and file formats. With protobuf,
>>>>>> you
>>>>>> can
>>>>>> define a binary protocol very quickly and efficiently. For more
>>>>>> information, please see here: http://code.google.com/p/protobuf/
>>>>>>
>>>>>> With the ProtobufEncoder and ProtobufDecoder that Netty provides, you
>>>>>> can
>>>>>> now write a network application with a very efficient binary encoding
>>>>>> even
>>>>>> more quickly than ever. It just takes little time to implement a
>>>>>> codec
>>>>>> with
>>>>>> protobuf.
>>>>>>
>>>>>> To help your understanding, I have written a time client / server
>>>>>> example.
>>>>>> The time client sends a list of time zones, and then the time server
>>>>>> responds with the local time for each time zone. Please brwose the
>>>>>> following directory:
>>>>>>
>>>>>> -
>>>>>>
>>>>>> http://fisheye.jboss.org/browse/Netty/trunk/src/main/java/org/jboss/netty/example/localtime
>>>>>>
>>>>>> Please note that LocalTimeProtocol.java has been generated from
>>>>>> LocalTimeProtocol.proto by protobuf compiler.
>>>>>>
>>>>>> Please feel free to send me a feed back once you review the source
>>>>>> code.
>>>>>> I'd like to know what could be improved to support protobuf better and
>>>>>> to
>>>>>> help you implement an efficient binary protocol more quickly than
>>>>>> ever.
>>>>>>
>>>>> Here is an idea for improvement of Google Protocol Buffers (protobuf)
>>>>> support in Netty. Currently each frame is prepended with fixed size
>>>>> four
>>>>> byte integer value that is a length of the message. Why not to use cool
>>>>> protobuf varint
>>>>> (http://code.google.com/apis/protocolbuffers/docs/encoding.html#varints)
>>>>> for
>>>>> that purpose? By doing this for small messages one could save a few
>>>>> bits
>>>>> of
>>>>> bandwidth. For instance, messages smaller that 128 bytes can be
>>>>> "framed"
>>>>> only by one byte representing the size.
>>>>>
>>>>> I've already implmented FrameDecoded and FieldPrepender that I could
>>>>> share
>>>>> with you through JIRA. Please, let me know what you think and if you'd
>>>>> considered such feature for Netty.
>>>>>
>>>>> Cheers,
>>>>> Tom
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://n2.nabble.com/Google-Protocol-Buffers-integration-is-ready.-tp2122078p3304575.html
>>>>> Sent from the Netty Developer Group mailing list archive at Nabble.com.
>>>>> _______________________________________________
>>>>> netty-dev mailing list
>>>>> netty-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/netty-dev
>>>>>
>>>> _______________________________________________
>>>> netty-dev mailing list
>>>> netty-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/netty-dev
>>>>
>>>>
>>> --
>>> View this message in context:
>>> http://n2.nabble.com/Google-Protocol-Buffers-integration-is-ready.-tp2122078p3306268.html
>>> Sent from the Netty Developer Group mailing list archive at Nabble.com.
>>>
>>> _______________________________________________
>>> netty-dev mailing list
>>> netty-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/netty-dev
>>>
>> _______________________________________________
>> netty-dev mailing list
>> netty-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/netty-dev
>>
>>
>
More information about the netty-dev
mailing list