Announcing LittleProxy 0.1 based on Netty

Trustin Lee (이희승) trustin at gmail.com
Mon Nov 16 10:42:07 EST 2009 

Actually, I've just realized that trailing headers are allowed only at
the last chunk.  Therefore we don't need to add the header accessors
to any chunk messages but the last one.  We can create a new subtype
of HttpPartialContent to represent the last chunk.  Problem solved ..
or my knowledge base fixed! :)

Cheers,

-- Trustin Lee, http://gleamynode.net/

2009/11/17 Trustin Lee (이희승) <trustin at gmail.com>:
> Sure.  No problem.  Until the best solution is found for trailing
> headers, we can apply the solution we've reached to so far.  Here's
> the JIRA issue, just in case you want to keep track of the progress or
> you want to contribute the fix:
>
>    https://jira.jboss.org/jira/browse/NETTY-250
>
> -- Trustin Lee, http://gleamynode.net/
>
>
> On Mon, Nov 16, 2009 at 7:02 AM, Adam Fisk <a at littleshoot.org> wrote:
>> I'm not super familiar with the trailing headers, Trustin - I need to
>> look at the spec a bit. I'm assuming you don't feel comfortable
>> trusting clients to send reasonably-sized chunks though right? I guess
>> that would be one easy way to bring down a server if an attacker
>> wanted to.
>>
>> I didn't realize you were re-chunking bodies that are already chunked.
>> I'd need to look at the code, but I'm juggling a few things right now
>> and don't know when I'll get to it.
>>
>> -Adam
>>
>>
>> On Fri, Nov 13, 2009 at 12:14 AM, Trustin Lee (이희승) <trustin at gmail.com> wrote:
>>> Glad that you like the proposal! :)
>>>
>>> I realized that there is another issue with automatic conversion
>>> though.  What should we do if a client sends the content in chunked
>>> encoding and the chunks contain trailing headers such as
>>> 'Content-MD5'?  The current implementation splits a large chunk into
>>> multiple chunks, and it can be a problem when the decoder splits the
>>> chunk with trailing headers.
>>>
>>> -- Trustin Lee, http://gleamynode.net/
>>>
>>>
>>>
>>> On Thu, Nov 12, 2009 at 2:27 PM, Adam Fisk <a at littleshoot.org> wrote:
>>>> I think that sounds like the perfect solution, Trustin. It allows you
>>>> to fulfill the desired goal of not holding a huge message body in
>>>> memory while getting around the issue having somewhat complex logic
>>>> for when to add Transfer-Encoding. So it not only solves the problem,
>>>> but it overall simplifies the code. Big thumbs up from me!
>>>>
>>>> -Adam
>>>>
>>>>
>>>> On Wed, Nov 11, 2009 at 8:14 PM, Trustin Lee (이희승) <trustin at gmail.com> wrote:
>>>>> Hi Adam,
>>>>>
>>>>> Strictly speaking, it is wrong to convert a HTTP/1.0 message to
>>>>> chunked encoding.  To conform to the standard, the decoder could be
>>>>> modified to disable the conversion for HTTP/1.0 messages, but I don't
>>>>> think this is what people expects from Netty.  No conversion means
>>>>> greater latency and memory footprint.
>>>>>
>>>>> So, I'd like to suggest to:
>>>>>
>>>>> 1) Redefine HttpChunk - say it can be used for either chunked encoding
>>>>> or just large content streaming,
>>>>> 2) modify the decoder not to update the 'Transfer-Encoding' while
>>>>> letting HttpMessage.isChunked() return true, and
>>>>> 3) modify the encoder not to generate chunked response if the message
>>>>> version is old.
>>>>>
>>>>> By doing this, we do not violate standard but simply redefine how
>>>>> large content is delivered.
>>>>>
>>>>> However, the name HttpChunk becomes confusing, so I'd like to
>>>>> introduce a new type - HttpPartialContent:
>>>>>
>>>>> public interface HttpPartialContent {
>>>>>    // get/setContent() and isLast()
>>>>> }
>>>>> public interface HttpChunk extends HttpPartialContent {
>>>>>    // access methods for trailing headers
>>>>> }
>>>>>
>>>>> also HttpMessage.isChunked() should be replaced with
>>>>> 'hasFollowingPartialContent()' or something similar (better name
>>>>> suggestion?)
>>>>>
>>>>> WDYT?
>>>>>
>>>>> -- Trustin Lee, http://gleamynode.net/
>>>>>
>>>>> On Thu, Nov 12, 2009 at 11:36 AM, Adam Fisk <a at littleshoot.org> wrote:
>>>>>> Hi Trustin- Thanks for the detailed response. I already fixed this
>>>>>> independently in LittleProxy running with Netty 3.1.5 through creating
>>>>>> a new message, so it's not an issue there. I *think* there's still a
>>>>>> problem in Netty in the following case, however.
>>>>>>
>>>>>> Leaving the proxy case aside, the straight HttpMessageDecoder will
>>>>>> switch to chunked encoding any time the response body length is
>>>>>> greater than maxChunkSize. It does this regardless of protocol
>>>>>> version, so it will produce HTTP 1.0 request and responses that use
>>>>>> Transfer-Encoding: chunked.
>>>>>>
>>>>>> It seems like it's always wrong whenever that happens, no? It could
>>>>>> potentially only convert to chunked encoding if the version is HTTP
>>>>>> 1.1, but I'm pretty sure the current implementation will break things
>>>>>> other than LittleProxy.
>>>>>>
>>>>>> Thanks again.
>>>>>>
>>>>>> -Adam
>>>>>>
>>>>>>
>>>>>> On Wed, Nov 11, 2009 at 1:41 AM, Trustin Lee (이희승) <trustin at gmail.com> wrote:
>>>>>>> I guess I found a problem with upgrading the protocol version automatically.
>>>>>>>
>>>>>>> If a client sends a request in HTTP/1.0 (i.e. no chunked encoding
>>>>>>> support on the client side), the server (Netty HTTP) should not
>>>>>>> respond with chunked content.  If the decoder upgrades the protocol
>>>>>>> version, there's no way for an HTTP request handler to determine if
>>>>>>> the client actually supports HTTP/1.1 or not.
>>>>>>>
>>>>>>> Therefore, the original protocol version needs to be retained instead
>>>>>>> of being simply overridden by the decoder.  Actually, I don't see a
>>>>>>> problem with retaining the original version because a broken client
>>>>>>> implementation can send such a request and the request handler should
>>>>>>> be able to handle it anyway.
>>>>>>>
>>>>>>> Because HttpMessageDecoder should not modify the original protocol
>>>>>>> version, LittleProxy will have to handle protocol version upgrade and
>>>>>>> downgrade by itself like the following:
>>>>>>>
>>>>>>> For the HTTP proxy to send the chunked content converted by
>>>>>>> HttpRequestDecoder to the actual HTTP server, the proxy can either:
>>>>>>>
>>>>>>> 1) modify the outbound HTTP request by calling
>>>>>>> HttpMessage.setProtocolVersion() and store the actual client protocol
>>>>>>> version somewhere (assuming NETTY-247[1] has been resolved), or
>>>>>>> 2) create a new outbound HTTP request object with upgraded protocol version.
>>>>>>>
>>>>>>> If the proxy receives the chunked content converted by
>>>>>>> HttpResponseDecoder from the actual server, the same modification
>>>>>>> needs to be performed against the received HttpResponse.  If the
>>>>>>> client sent HTTP/1.0 request, the received chunks have to be merged
>>>>>>> back because the client will probably not understand chunked encoding.
>>>>>>>
>>>>>>> Another tricky case would be the case where the actual server does not
>>>>>>> understand chunked encoding.  In such a case, the actual server will
>>>>>>> respond with "HTTP/1.0 406 Not Acceptable" and the proxy server should
>>>>>>> retry after merging the chunks and downgrading the protocol version.
>>>>>>>
>>>>>>> Please let me know if I am missing something.  Otherwise, I will
>>>>>>> retain the current behavior while adding setter methods to
>>>>>>> HttpMessage.
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> -- Trustin Lee, http://gleamynode.net/
>>>>>>>
>>>>>>> [1] NETTY-247: https://jira.jboss.org/jira/browse/NETTY-247
>>>>>>>
>>>>>>> On Wed, Nov 4, 2009 at 3:34 PM, Adam Fisk <a at littleshoot.org> wrote:
>>>>>>>> OK - finally knocked this off, and it's working beautifully in FireFox and
>>>>>>>> Safari. All the credit really goes to Stephen - I'm just implementing his
>>>>>>>> fix here.
>>>>>>>> I agree with some of the earlier comments that since HttpMessage and its
>>>>>>>> subclasses are already mutable, it's not a big loss to add another mutable
>>>>>>>> field. That's the approach I took with the patch. I also think it makes
>>>>>>>> sense to keep HttpChunk immutable unless there's a concrete need to make it
>>>>>>>> mutable. I also don't think this is indicative of a design flaw.
>>>>>>>> Feel free to change the style, approach, or comments as you see fit,
>>>>>>>> Trustin. Thanks again Stephen. I like that handy netcat trick, by the way!
>>>>>>>> -Adam
>>>>>>>>
>>>>>>>> On Tue, Nov 3, 2009 at 8:47 PM, Stephen Haberman <stephen at exigencecorp.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> > I'm also happy to give you commit permissions if you like. This in
>>>>>>>>> > particular is clearly a Netty and not a LittleProxy issue, but I can
>>>>>>>>> > give you access to the repo if you need to make any changes going
>>>>>>>>> > forward.
>>>>>>>>>
>>>>>>>>> That's fine--I enjoyed tracking the bug down and also reading through
>>>>>>>>> the LittleProxy code.
>>>>>>>>>
>>>>>>>>> My proxy is more of a weekend (or two or three) hobby project--I
>>>>>>>>> wouldn't be surprised if its missing several boundary cases that
>>>>>>>>> LittleProxy handles more robustly.
>>>>>>>>>
>>>>>>>>> But I did finally get it into github:
>>>>>>>>>
>>>>>>>>> http://github.com/stephenh/fakehttp
>>>>>>>>>
>>>>>>>>> Being a hobby project, I also wrote it in Scala and, purposefully
>>>>>>>>> trying to keep it small, have kept it to ~450 LOC so far.
>>>>>>>>>
>>>>>>>>> - Stephen
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> netty-users mailing list
>>>>>>>>> netty-users at lists.jboss.org
>>>>>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Adam Fisk
>>>>>>>> http://www.littleshoot.org | http://adamfisk.wordpress.com |
>>>>>>>> http://twitter.com/adamfisk
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> netty-users mailing list
>>>>>>>> netty-users at lists.jboss.org
>>>>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> netty-users mailing list
>>>>>>> netty-users at lists.jboss.org
>>>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Adam Fisk
>>>>>> http://www.littleshoot.org | http://adamfisk.wordpress.com |
>>>>>> http://twitter.com/adamfisk
>>>>>>
>>>>>> _______________________________________________
>>>>>> netty-users mailing list
>>>>>> netty-users at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> netty-users mailing list
>>>>> netty-users at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Adam Fisk
>>>> http://www.littleshoot.org | http://adamfisk.wordpress.com |
>>>> http://twitter.com/adamfisk
>>>>
>>>> _______________________________________________
>>>> netty-users mailing list
>>>> netty-users at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>
>>>
>>> _______________________________________________
>>> netty-users mailing list
>>> netty-users at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>
>>
>>
>>
>> --
>> Adam Fisk
>> http://www.littleshoot.org | http://adamfisk.wordpress.com |
>> http://twitter.com/adamfisk
>>
>> _______________________________________________
>> netty-users mailing list
>> netty-users at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/netty-users
>>
>

More information about the netty-users mailing list