Recommended way of decoding non-framed messages?

[Bruno de Carvalho]

Benoit,


Take a look at ReplayingDecoder (http://docs.jboss.org/netty/3.2/api/org/jboss/netty/handler/codec/replay/ReplayingDecoder.html). It's a mix of your solutions 1 and 3.

In every successful read you advance state and every time you try to read past the capacity an exception is thrown (maintaining current state). Difference is this exception is already created and the buffer itself is a special variant of ChannelBuffer. At first it may sound like it's not very performance friendly but trust me, it is ;)

Most of the binary protocols I implement are non-framed and this decoder is perfect for that.

Lemme know if you need any extra hints.


Cheers,
  Bruno

On 31 Jul 2010, at 10:44, tsuna <tsunanet at gmail.com> wrote:

> Hello,
> I'm implementing a legacy binary protocol in which the responses
> coming back from the server aren't framed (i.e. they don't contain
> some kind of an integer value near the beginning of the response to
> tell me ahead of time how many bytes I should expect).
> 
> For instance, a typical example of message I'm dealing with is:  If
> the first byte of the response is 0x01, then read the next 4 bytes
> into an int (let's call it N), then read N bytes, then read the next 4
> bytes into another int (let's call it M), then read M bytes.
> 
> This isn't a problem for the legacy implementation of this protocol
> since it's blocking on read() when it needs more data while
> de-serializing a response.  But it's a problem for me as I'm getting a
> IndexOutOfBoundsException when attempt to read past the end of a
> ChannelBuffer that contains a partial response.  I was wondering how
> people typically deal with such situations.
> 
> I can see multiple strategies but none of them seem really appealing to me:
>  1. When getting an IndexOutOfBoundsException, just throw away all
> the work done to de-serialize the response so far, reset the
> readerIndex of the buffer, and wait for more data.
>  2. Do the decoding in 2 passes.  The first pass will attempt to
> compute the size of the response by inspecting the ChannelBuffer
> without de-serializing it, and will ask for more data if there isn't
> enough in the buffer or if the size cannot be computed from the
> current contents of the buffer.  The second pass would then do the
> de-serialization normally.
>  3. Have a state machine in the decoder to "remember" where decoding
> stopped due to not having enough data in the buffer, so the decoder
> can pick up where it left when more data arrives.
> 
> I'm sure there are more strategies.  The strategy 1) is by far the
> easiest to implement, but it's also the least efficient, since it can
> throw away and re-do potentially expensive work multiple times
> (de-serializing, allocating objects, etc.).  The strategy 2) is kind
> of cumbersome to implement, but I think that barring any better
> proposal, that's the one I'm leaning towards.  The strategy 3) is
> really complicated to implement, particularly when messages are
> de-serialized into nested objects.
> 
> Thoughts?
> 
> -- 
> Benoit "tsuna" Sigoure
> Software Engineer @ www.StumbleUpon.com
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users