Announcing LittleProxy 0.1 based on Netty

Fri Nov 13 03:14:41 EST 2009

Glad that you like the proposal! :)

I realized that there is another issue with automatic conversion
though.  What should we do if a client sends the content in chunked
encoding and the chunks contain trailing headers such as
'Content-MD5'?  The current implementation splits a large chunk into
multiple chunks, and it can be a problem when the decoder splits the
chunk with trailing headers.

— Trustin Lee, http://gleamynode.net/

On Thu, Nov 12, 2009 at 2:27 PM, Adam Fisk <a at littleshoot.org> wrote:
> I think that sounds like the perfect solution, Trustin. It allows you
> to fulfill the desired goal of not holding a huge message body in
> memory while getting around the issue having somewhat complex logic
> for when to add Transfer-Encoding. So it not only solves the problem,
> but it overall simplifies the code. Big thumbs up from me!
>
> -Adam
>
>
> On Wed, Nov 11, 2009 at 8:14 PM, Trustin Lee (이희승) <trustin at gmail.com> wrote:
>> Hi Adam,
>>
>> Strictly speaking, it is wrong to convert a HTTP/1.0 message to
>> chunked encoding.  To conform to the standard, the decoder could be
>> modified to disable the conversion for HTTP/1.0 messages, but I don't
>> think this is what people expects from Netty.  No conversion means
>> greater latency and memory footprint.
>>
>> So, I'd like to suggest to:
>>
>> 1) Redefine HttpChunk - say it can be used for either chunked encoding
>> or just large content streaming,
>> 2) modify the decoder not to update the 'Transfer-Encoding' while
>> letting HttpMessage.isChunked() return true, and
>> 3) modify the encoder not to generate chunked response if the message
>> version is old.
>>
>> By doing this, we do not violate standard but simply redefine how
>> large content is delivered.
>>
>> However, the name HttpChunk becomes confusing, so I'd like to
>> introduce a new type - HttpPartialContent:
>>
>> public interface HttpPartialContent {
>>    // get/setContent() and isLast()
>> }
>> public interface HttpChunk extends HttpPartialContent {
>>    // access methods for trailing headers
>> }
>>
>> also HttpMessage.isChunked() should be replaced with
>> 'hasFollowingPartialContent()' or something similar (better name
>> suggestion?)
>>
>> WDYT?
>>
>> — Trustin Lee, http://gleamynode.net/
>>
>> On Thu, Nov 12, 2009 at 11:36 AM, Adam Fisk <a at littleshoot.org> wrote:
>>> Hi Trustin- Thanks for the detailed response. I already fixed this
>>> independently in LittleProxy running with Netty 3.1.5 through creating
>>> a new message, so it's not an issue there. I *think* there's still a
>>> problem in Netty in the following case, however.
>>>
>>> Leaving the proxy case aside, the straight HttpMessageDecoder will
>>> switch to chunked encoding any time the response body length is
>>> greater than maxChunkSize. It does this regardless of protocol
>>> version, so it will produce HTTP 1.0 request and responses that use
>>> Transfer-Encoding: chunked.
>>>
>>> It seems like it's always wrong whenever that happens, no? It could
>>> potentially only convert to chunked encoding if the version is HTTP
>>> 1.1, but I'm pretty sure the current implementation will break things
>>> other than LittleProxy.
>>>
>>> Thanks again.
>>>
>>> -Adam
>>>
>>>
>>> On Wed, Nov 11, 2009 at 1:41 AM, Trustin Lee (이희승) <trustin at gmail.com> wrote:
>>>> I guess I found a problem with upgrading the protocol version automatically.
>>>>
>>>> If a client sends a request in HTTP/1.0 (i.e. no chunked encoding
>>>> support on the client side), the server (Netty HTTP) should not
>>>> respond with chunked content.  If the decoder upgrades the protocol
>>>> version, there's no way for an HTTP request handler to determine if
>>>> the client actually supports HTTP/1.1 or not.
>>>>
>>>> Therefore, the original protocol version needs to be retained instead
>>>> of being simply overridden by the decoder.  Actually, I don't see a
>>>> problem with retaining the original version because a broken client
>>>> implementation can send such a request and the request handler should
>>>> be able to handle it anyway.
>>>>
>>>> Because HttpMessageDecoder should not modify the original protocol
>>>> version, LittleProxy will have to handle protocol version upgrade and
>>>> downgrade by itself like the following:
>>>>
>>>> For the HTTP proxy to send the chunked content converted by
>>>> HttpRequestDecoder to the actual HTTP server, the proxy can either:
>>>>
>>>> 1) modify the outbound HTTP request by calling
>>>> HttpMessage.setProtocolVersion() and store the actual client protocol
>>>> version somewhere (assuming NETTY-247[1] has been resolved), or
>>>> 2) create a new outbound HTTP request object with upgraded protocol version.
>>>>
>>>> If the proxy receives the chunked content converted by
>>>> HttpResponseDecoder from the actual server, the same modification
>>>> needs to be performed against the received HttpResponse.  If the
>>>> client sent HTTP/1.0 request, the received chunks have to be merged
>>>> back because the client will probably not understand chunked encoding.
>>>>
>>>> Another tricky case would be the case where the actual server does not
>>>> understand chunked encoding.  In such a case, the actual server will
>>>> respond with "HTTP/1.0 406 Not Acceptable" and the proxy server should
>>>> retry after merging the chunks and downgrading the protocol version.
>>>>
>>>> Please let me know if I am missing something.  Otherwise, I will
>>>> retain the current behavior while adding setter methods to
>>>> HttpMessage.
>>>>
>>>> Thanks
>>>>
>>>> — Trustin Lee, http://gleamynode.net/
>>>>
>>>> [1] NETTY-247: https://jira.jboss.org/jira/browse/NETTY-247
>>>>
>>>> On Wed, Nov 4, 2009 at 3:34 PM, Adam Fisk <a at littleshoot.org> wrote:
>>>>> OK - finally knocked this off, and it's working beautifully in FireFox and
>>>>> Safari. All the credit really goes to Stephen - I'm just implementing his
>>>>> fix here.
>>>>> I agree with some of the earlier comments that since HttpMessage and its
>>>>> subclasses are already mutable, it's not a big loss to add another mutable
>>>>> field. That's the approach I took with the patch. I also think it makes
>>>>> sense to keep HttpChunk immutable unless there's a concrete need to make it
>>>>> mutable. I also don't think this is indicative of a design flaw.
>>>>> Feel free to change the style, approach, or comments as you see fit,
>>>>> Trustin. Thanks again Stephen. I like that handy netcat trick, by the way!
>>>>> -Adam
>>>>>
>>>>> On Tue, Nov 3, 2009 at 8:47 PM, Stephen Haberman <stephen at exigencecorp.com>
>>>>> wrote:
>>>>>>
>>>>>> > I'm also happy to give you commit permissions if you like. This in
>>>>>> > particular is clearly a Netty and not a LittleProxy issue, but I can
>>>>>> > give you access to the repo if you need to make any changes going
>>>>>> > forward.
>>>>>>
>>>>>> That's fine--I enjoyed tracking the bug down and also reading through
>>>>>> the LittleProxy code.
>>>>>>
>>>>>> My proxy is more of a weekend (or two or three) hobby project--I
>>>>>> wouldn't be surprised if its missing several boundary cases that
>>>>>> LittleProxy handles more robustly.
>>>>>>
>>>>>> But I did finally get it into github:
>>>>>>
>>>>>> http://github.com/stephenh/fakehttp
>>>>>>
>>>>>> Being a hobby project, I also wrote it in Scala and, purposefully
>>>>>> trying to keep it small, have kept it to ~450 LOC so far.
>>>>>>
>>>>>> - Stephen
>>>>>>
>>>>>> _______________________________________________
>>>>>> netty-users mailing list
>>>>>> netty-users at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Adam Fisk
>>>>> http://www.littleshoot.org | http://adamfisk.wordpress.com |
>>>>> http://twitter.com/adamfisk
>>>>>
>>>>> _______________________________________________
>>>>> netty-users mailing list
>>>>> netty-users at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> netty-users mailing list
>>>> netty-users at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>
>>>
>>>
>>>
>>> --
>>> Adam Fisk
>>> http://www.littleshoot.org | http://adamfisk.wordpress.com |
>>> http://twitter.com/adamfisk
>>>
>>> _______________________________________________
>>> netty-users mailing list
>>> netty-users at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>
>>
>> _______________________________________________
>> netty-users mailing list
>> netty-users at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/netty-users
>>
>
>
>
> --
> Adam Fisk
> http://www.littleshoot.org | http://adamfisk.wordpress.com |
> http://twitter.com/adamfisk
>
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users
>