DelimiterBasedFrameDecoder Help

Tue Sep 29 04:21:59 EDT 2009

Hi David,

Because HttpMessageDecoder already transformed a ChannelBuffer into
HttpMessage, you don't need to add another decoder in the pipeline.
All decoders, by definition, decodes only ChannelBuffers, and
therefore HttpMessage is never handled by decoders
(FixedLengthFrameDecoder in your case.)  Instead, you can get the
content of the HTTP message by calling HttpMessage.getContent().  If
you need to decode the content of the HTTP message, you can start from
there.

HTH,
Trustin

— Trustin Lee, http://gleamynode.net/

On Sat, Sep 26, 2009 at 11:16 AM, Hoyt, David <hoyt6 at llnl.gov> wrote:
> Thanks for your reply - I'm working on it right now. Our stream is actually contained inside an http stream. So we connect via http, I have to verify that there's a correct content type from the server and then the http content is where our data actually lives. So the frame decoder needs to be inside an http decoder, right? I thought I did this, but I'm getting DefaultHttpResponse messages and not frames as I'd expect. Here's my code:
>
> final ChannelPipeline pipeline = new DefaultChannelPipeline();
> pipeline.addLast("encoder", new HttpRequestEncoder());
> pipeline.addLast("decoder", new HttpResponseEncoder());
> pipeline.addLast("aggregator", new HttpChunkAggregator(4096));
> pipeline.addLast("frameDecoder", new FixedLengthFrameDecoder(<PACKET LENGTH>));
> pipeline.addLast("myHandler", new MyHandler());
>
> ...
>
> public class MyHandler extends SimpleChannelHandler {
>    public void messageReceived(ChannelHandlerContext ctx, MessageEvent e) throws Exception {
>        Object o = e.getMessage();
>        System.out.println(o);
>    }
> }
>
> "o" here is DefaultHttpResponse. I thought it would be a channel buffer from FixedLengthFrameDecoder. How do I get that right?
>
> Thanks!
>
> -----Original Message-----
> From: netty-users-bounces at lists.jboss.org [mailto:netty-users-bounces at lists.jboss.org] On Behalf Of Trustin Lee (???)
> Sent: Friday, September 25, 2009 1:27 AM
> To: Netty Users
> Subject: Re: DelimiterBasedFrameDecoder Help
>
> Hi David,
>
> If there is an extra bad byte in the frame, I usually raise an
> exception in the decoder.  Netty will trigger an exceptionCaught
> event, then you can handle it to close the connection immediately.
> Depending on your preference, you can just close the connection
> immediately instead of raising an exception.
>
> FixedLengthFrameDecoder simply generates a fixed length frame, and
> therefore it does not know anything about the semantics of the
> received data.  Therefore bad bytes are not skipped at all.  If you
> want to implement such functionality, you need to implement a decoder
> by yourself.  In my opinion, generally, you should not implement such
> functionality to detect bad data because 1) TCP/IP already has a
> mechanism for error detection and 2) misbehaving (or malicious) client
> should be disconnected immediately to avoid possible misinterpretation
> of packets.  Bad data could accidently change the semantic of the
> message, bypassing the validation layer of your server and leading to
> a security issue.
>
> HTH
>
> — Trustin Lee, http://*gleamynode.net/
>
> On Fri, Sep 25, 2009 at 5:09 PM, Hoyt, David <hoyt6 at llnl.gov> wrote:
>> Thanks for your help - it's very much appreciated! If we connected to a malicious server and it injected an extra byte somewhere (even though all subsequent packets would be good - say packet 1 is fine, packet 2 contains 1 extra byte, but packets 3 through 1 million are all fine), is FixedLengthFrameDecoder resilient enough to catch it and skip a frame until we get the next valid packet? Or would packets 3 through 1 million all be off? Is it easy enough to add this functionality? I think I understand how I could do it - by returning null until I see the first byte of my header and then discarding everything in the meantime.
>>
>> How do you discard bytes in the frame decoder? I don't want incoming bytes to accumulate if I don't care about them. If I got 100 bad packets in a row and then 1 good one, I don't want to buffer 100 bad packets - I'd want to discard all of them as they come in and keep only the good one.
>>
>>
>>
>> On another note, btw -- excellent work w/ netty! We're using it in other places, and it's outperforming every other implementation we've ever used! Very impressive...
>>
>> -----Original Message-----
>> From: netty-users-bounces at lists.jboss.org [mailto:netty-users-bounces at lists.jboss.org] On Behalf Of Trustin Lee (???)
>> Sent: Thursday, September 24, 2009 5:14 PM
>> To: Netty Users
>> Subject: Re: DelimiterBasedFrameDecoder Help
>>
>> Hi David,
>>
>> Actually, your message got through the mailing list.  It just took a
>> while for me to answer your question.  Thanks for your patience.
>>
>> On Fri, Sep 25, 2009 at 2:33 AM, Hoyt, David <hoyt6 at llnl.gov> wrote:
>>> I'm not sure if my original post made it or not because I had a few failed attempts - but I'm hoping this one makes it through...
>>>
>>> I’m still learning netty and I need to find the best classes to do what I need or find out if I need to code it myself – we have our own protocol that’s very, very simple – a header used to synchronize where we’re at in the stream and then some int’s, long’s, etc. in a known order. The entire length of the packet is known ahead of time (it's a fixed-length packet). We intend to maintain this connection for days, weeks, months, and years. So an example of a stream is (just the bytes -- "Packet #", "Header", and "Payload" don't actually appear in the data):
>>>
>>> Packet 1
>>> Header: 0xdeadbeef
>>> Payload: 0x00000000, 0x00000001, 0x00000002
>>>
>>> Packet 2
>>> Header: 0xdeadbeef
>>> Payload: 0x00000000, 0x00000001, 0x00000002
>>
>> If the length of the packet is fixed, it is dead simple - use
>> FixedLengthFrameDecoder.  You never need DelimiterBasedFrameDecoder.
>>
>>> We use the header (delimiter) in case (for whatever reason – a particle from space flips a bit on the server producing these messages) we lose where we’re at and need to synchronize with the next packet. So say we got the following sequence:
>>>
>>> Packet 1: 0xdeadbeef 0x0x00000000 0x00000000 0x00000000
>>> Packet 2: 0xabcdabcd 0x0x00000000 0x00000000 0x00000000
>>> Packet 3: 0xdeadbeef 0x0x00000000 0x00000000 0x00000000
>>>
>>> We would want to skip packet 2 and continue on with packet 3. We would like to know if we found a problem with packet 2, but it's not a requirement.
>>
>> Once the FixedLengthFrameDecoder splits the incoming data as you want,
>> you can inspect the header of the message to determine if the message
>> is what you were expecting.
>>
>>> We also need to know if there's a significant delay between messages (e.g. a read timeout of, say, 30 seconds or more). We only need to ever read and never write to this stream.
>>
>> You could use ReadTimeoutHandler or IdleStateHandler (in
>> org.jboss.netty.handler.timeout).
>>
>>> Does DelimiterBasedFrameDecoder allow me to recover from a bad packet? Can I recover without having to catch an exception? I'm not sure where to begin. I know how I would do it using old IO, but I'd prefer to take advantage of netty's power and expressiveness and I'm sure netty can handle this. I just don't want to reinvent the wheel.
>>
>> DelimiterBasedFrameDecoder is used only when there is an explicit
>> delimiter defined in the protocol.  For example, a pipe (|), NUL
>> (0x00), or newline character is the case - mostly text protocols.
>> Your protocol does not have a delimiter, so DelimiterBasedFrameDecoder
>> cannot be used.
>>
>> HTH
>>
>> — Trustin Lee, http://**gleamynode.net/
>>
>> _______________________________________________
>> netty-users mailing list
>> netty-users at lists.jboss.org
>> https://**lists.jboss.org/mailman/listinfo/netty-users
>>
>>
>> _______________________________________________
>> netty-users mailing list
>> netty-users at lists.jboss.org
>> https://*lists.jboss.org/mailman/listinfo/netty-users
>>
>
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://*lists.jboss.org/mailman/listinfo/netty-users
>
>
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users
>