Recommended way of decoding non-framed messages?

[tsuna]

Hello,
I'm implementing a legacy binary protocol in which the responses
coming back from the server aren't framed (i.e. they don't contain
some kind of an integer value near the beginning of the response to
tell me ahead of time how many bytes I should expect).

For instance, a typical example of message I'm dealing with is:  If
the first byte of the response is 0x01, then read the next 4 bytes
into an int (let's call it N), then read N bytes, then read the next 4
bytes into another int (let's call it M), then read M bytes.

This isn't a problem for the legacy implementation of this protocol
since it's blocking on read() when it needs more data while
de-serializing a response.  But it's a problem for me as I'm getting a
IndexOutOfBoundsException when attempt to read past the end of a
ChannelBuffer that contains a partial response.  I was wondering how
people typically deal with such situations.

I can see multiple strategies but none of them seem really appealing to me:
  1. When getting an IndexOutOfBoundsException, just throw away all
the work done to de-serialize the response so far, reset the
readerIndex of the buffer, and wait for more data.
  2. Do the decoding in 2 passes.  The first pass will attempt to
compute the size of the response by inspecting the ChannelBuffer
without de-serializing it, and will ask for more data if there isn't
enough in the buffer or if the size cannot be computed from the
current contents of the buffer.  The second pass would then do the
de-serialization normally.
  3. Have a state machine in the decoder to "remember" where decoding
stopped due to not having enough data in the buffer, so the decoder
can pick up where it left when more data arrives.

I'm sure there are more strategies.  The strategy 1) is by far the
easiest to implement, but it's also the least efficient, since it can
throw away and re-do potentially expensive work multiple times
(de-serializing, allocating objects, etc.).  The strategy 2) is kind
of cumbersome to implement, but I think that barring any better
proposal, that's the one I'm leaning towards.  The strategy 3) is
really complicated to implement, particularly when messages are
de-serialized into nested objects.

Thoughts?

-- 
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com