Announcing LittleProxy 0.1 based on Netty

Sun Nov 15 17:02:12 EST 2009

I'm not super familiar with the trailing headers, Trustin - I need to
look at the spec a bit. I'm assuming you don't feel comfortable
trusting clients to send reasonably-sized chunks though right? I guess
that would be one easy way to bring down a server if an attacker
wanted to.

I didn't realize you were re-chunking bodies that are already chunked.
I'd need to look at the code, but I'm juggling a few things right now
and don't know when I'll get to it.

-Adam

On Fri, Nov 13, 2009 at 12:14 AM, Trustin Lee (이희승) <trustin at gmail.com> wrote:
> Glad that you like the proposal! :)
>
> I realized that there is another issue with automatic conversion
> though.  What should we do if a client sends the content in chunked
> encoding and the chunks contain trailing headers such as
> 'Content-MD5'?  The current implementation splits a large chunk into
> multiple chunks, and it can be a problem when the decoder splits the
> chunk with trailing headers.
>
> — Trustin Lee, http://gleamynode.net/
>
>
>
> On Thu, Nov 12, 2009 at 2:27 PM, Adam Fisk <a at littleshoot.org> wrote:
>> I think that sounds like the perfect solution, Trustin. It allows you
>> to fulfill the desired goal of not holding a huge message body in
>> memory while getting around the issue having somewhat complex logic
>> for when to add Transfer-Encoding. So it not only solves the problem,
>> but it overall simplifies the code. Big thumbs up from me!
>>
>> -Adam
>>
>>
>> On Wed, Nov 11, 2009 at 8:14 PM, Trustin Lee (이희승) <trustin at gmail.com> wrote:
>>> Hi Adam,
>>>
>>> Strictly speaking, it is wrong to convert a HTTP/1.0 message to
>>> chunked encoding.  To conform to the standard, the decoder could be
>>> modified to disable the conversion for HTTP/1.0 messages, but I don't
>>> think this is what people expects from Netty.  No conversion means
>>> greater latency and memory footprint.
>>>
>>> So, I'd like to suggest to:
>>>
>>> 1) Redefine HttpChunk - say it can be used for either chunked encoding
>>> or just large content streaming,
>>> 2) modify the decoder not to update the 'Transfer-Encoding' while
>>> letting HttpMessage.isChunked() return true, and
>>> 3) modify the encoder not to generate chunked response if the message
>>> version is old.
>>>
>>> By doing this, we do not violate standard but simply redefine how
>>> large content is delivered.
>>>
>>> However, the name HttpChunk becomes confusing, so I'd like to
>>> introduce a new type - HttpPartialContent:
>>>
>>> public interface HttpPartialContent {
>>>    // get/setContent() and isLast()
>>> }
>>> public interface HttpChunk extends HttpPartialContent {
>>>    // access methods for trailing headers
>>> }
>>>
>>> also HttpMessage.isChunked() should be replaced with
>>> 'hasFollowingPartialContent()' or something similar (better name
>>> suggestion?)
>>>
>>> WDYT?
>>>
>>> — Trustin Lee, http://gleamynode.net/
>>>
>>> On Thu, Nov 12, 2009 at 11:36 AM, Adam Fisk <a at littleshoot.org> wrote:
>>>> Hi Trustin- Thanks for the detailed response. I already fixed this
>>>> independently in LittleProxy running with Netty 3.1.5 through creating
>>>> a new message, so it's not an issue there. I *think* there's still a
>>>> problem in Netty in the following case, however.
>>>>
>>>> Leaving the proxy case aside, the straight HttpMessageDecoder will
>>>> switch to chunked encoding any time the response body length is
>>>> greater than maxChunkSize. It does this regardless of protocol
>>>> version, so it will produce HTTP 1.0 request and responses that use
>>>> Transfer-Encoding: chunked.
>>>>
>>>> It seems like it's always wrong whenever that happens, no? It could
>>>> potentially only convert to chunked encoding if the version is HTTP
>>>> 1.1, but I'm pretty sure the current implementation will break things
>>>> other than LittleProxy.
>>>>
>>>> Thanks again.
>>>>
>>>> -Adam
>>>>
>>>>
>>>> On Wed, Nov 11, 2009 at 1:41 AM, Trustin Lee (이희승) <trustin at gmail.com> wrote:
>>>>> I guess I found a problem with upgrading the protocol version automatically.
>>>>>
>>>>> If a client sends a request in HTTP/1.0 (i.e. no chunked encoding
>>>>> support on the client side), the server (Netty HTTP) should not
>>>>> respond with chunked content.  If the decoder upgrades the protocol
>>>>> version, there's no way for an HTTP request handler to determine if
>>>>> the client actually supports HTTP/1.1 or not.
>>>>>
>>>>> Therefore, the original protocol version needs to be retained instead
>>>>> of being simply overridden by the decoder.  Actually, I don't see a
>>>>> problem with retaining the original version because a broken client
>>>>> implementation can send such a request and the request handler should
>>>>> be able to handle it anyway.
>>>>>
>>>>> Because HttpMessageDecoder should not modify the original protocol
>>>>> version, LittleProxy will have to handle protocol version upgrade and
>>>>> downgrade by itself like the following:
>>>>>
>>>>> For the HTTP proxy to send the chunked content converted by
>>>>> HttpRequestDecoder to the actual HTTP server, the proxy can either:
>>>>>
>>>>> 1) modify the outbound HTTP request by calling
>>>>> HttpMessage.setProtocolVersion() and store the actual client protocol
>>>>> version somewhere (assuming NETTY-247[1] has been resolved), or
>>>>> 2) create a new outbound HTTP request object with upgraded protocol version.
>>>>>
>>>>> If the proxy receives the chunked content converted by
>>>>> HttpResponseDecoder from the actual server, the same modification
>>>>> needs to be performed against the received HttpResponse.  If the
>>>>> client sent HTTP/1.0 request, the received chunks have to be merged
>>>>> back because the client will probably not understand chunked encoding.
>>>>>
>>>>> Another tricky case would be the case where the actual server does not
>>>>> understand chunked encoding.  In such a case, the actual server will
>>>>> respond with "HTTP/1.0 406 Not Acceptable" and the proxy server should
>>>>> retry after merging the chunks and downgrading the protocol version.
>>>>>
>>>>> Please let me know if I am missing something.  Otherwise, I will
>>>>> retain the current behavior while adding setter methods to
>>>>> HttpMessage.
>>>>>
>>>>> Thanks
>>>>>
>>>>> — Trustin Lee, http://gleamynode.net/
>>>>>
>>>>> [1] NETTY-247: https://jira.jboss.org/jira/browse/NETTY-247
>>>>>
>>>>> On Wed, Nov 4, 2009 at 3:34 PM, Adam Fisk <a at littleshoot.org> wrote:
>>>>>> OK - finally knocked this off, and it's working beautifully in FireFox and
>>>>>> Safari. All the credit really goes to Stephen - I'm just implementing his
>>>>>> fix here.
>>>>>> I agree with some of the earlier comments that since HttpMessage and its
>>>>>> subclasses are already mutable, it's not a big loss to add another mutable
>>>>>> field. That's the approach I took with the patch. I also think it makes
>>>>>> sense to keep HttpChunk immutable unless there's a concrete need to make it
>>>>>> mutable. I also don't think this is indicative of a design flaw.
>>>>>> Feel free to change the style, approach, or comments as you see fit,
>>>>>> Trustin. Thanks again Stephen. I like that handy netcat trick, by the way!
>>>>>> -Adam
>>>>>>
>>>>>> On Tue, Nov 3, 2009 at 8:47 PM, Stephen Haberman <stephen at exigencecorp.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> > I'm also happy to give you commit permissions if you like. This in
>>>>>>> > particular is clearly a Netty and not a LittleProxy issue, but I can
>>>>>>> > give you access to the repo if you need to make any changes going
>>>>>>> > forward.
>>>>>>>
>>>>>>> That's fine--I enjoyed tracking the bug down and also reading through
>>>>>>> the LittleProxy code.
>>>>>>>
>>>>>>> My proxy is more of a weekend (or two or three) hobby project--I
>>>>>>> wouldn't be surprised if its missing several boundary cases that
>>>>>>> LittleProxy handles more robustly.
>>>>>>>
>>>>>>> But I did finally get it into github:
>>>>>>>
>>>>>>> http://github.com/stephenh/fakehttp
>>>>>>>
>>>>>>> Being a hobby project, I also wrote it in Scala and, purposefully
>>>>>>> trying to keep it small, have kept it to ~450 LOC so far.
>>>>>>>
>>>>>>> - Stephen
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> netty-users mailing list
>>>>>>> netty-users at lists.jboss.org
>>>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Adam Fisk
>>>>>> http://www.littleshoot.org | http://adamfisk.wordpress.com |
>>>>>> http://twitter.com/adamfisk
>>>>>>
>>>>>> _______________________________________________
>>>>>> netty-users mailing list
>>>>>> netty-users at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> netty-users mailing list
>>>>> netty-users at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Adam Fisk
>>>> http://www.littleshoot.org | http://adamfisk.wordpress.com |
>>>> http://twitter.com/adamfisk
>>>>
>>>> _______________________________________________
>>>> netty-users mailing list
>>>> netty-users at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>
>>>
>>> _______________________________________________
>>> netty-users mailing list
>>> netty-users at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>
>>
>>
>>
>> --
>> Adam Fisk
>> http://www.littleshoot.org | http://adamfisk.wordpress.com |
>> http://twitter.com/adamfisk
>>
>> _______________________________________________
>> netty-users mailing list
>> netty-users at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/netty-users
>>
>
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users
>

-- 
Adam Fisk
http://www.littleshoot.org | http://adamfisk.wordpress.com |
http://twitter.com/adamfisk