Announcing LittleProxy 0.1 based on Netty

Thu Nov 12 00:27:55 EST 2009

I think that sounds like the perfect solution, Trustin. It allows you
to fulfill the desired goal of not holding a huge message body in
memory while getting around the issue having somewhat complex logic
for when to add Transfer-Encoding. So it not only solves the problem,
but it overall simplifies the code. Big thumbs up from me!

-Adam

On Wed, Nov 11, 2009 at 8:14 PM, Trustin Lee (이희승) <trustin at gmail.com> wrote:
> Hi Adam,
>
> Strictly speaking, it is wrong to convert a HTTP/1.0 message to
> chunked encoding.  To conform to the standard, the decoder could be
> modified to disable the conversion for HTTP/1.0 messages, but I don't
> think this is what people expects from Netty.  No conversion means
> greater latency and memory footprint.
>
> So, I'd like to suggest to:
>
> 1) Redefine HttpChunk - say it can be used for either chunked encoding
> or just large content streaming,
> 2) modify the decoder not to update the 'Transfer-Encoding' while
> letting HttpMessage.isChunked() return true, and
> 3) modify the encoder not to generate chunked response if the message
> version is old.
>
> By doing this, we do not violate standard but simply redefine how
> large content is delivered.
>
> However, the name HttpChunk becomes confusing, so I'd like to
> introduce a new type - HttpPartialContent:
>
> public interface HttpPartialContent {
>    // get/setContent() and isLast()
> }
> public interface HttpChunk extends HttpPartialContent {
>    // access methods for trailing headers
> }
>
> also HttpMessage.isChunked() should be replaced with
> 'hasFollowingPartialContent()' or something similar (better name
> suggestion?)
>
> WDYT?
>
> — Trustin Lee, http://gleamynode.net/
>
> On Thu, Nov 12, 2009 at 11:36 AM, Adam Fisk <a at littleshoot.org> wrote:
>> Hi Trustin- Thanks for the detailed response. I already fixed this
>> independently in LittleProxy running with Netty 3.1.5 through creating
>> a new message, so it's not an issue there. I *think* there's still a
>> problem in Netty in the following case, however.
>>
>> Leaving the proxy case aside, the straight HttpMessageDecoder will
>> switch to chunked encoding any time the response body length is
>> greater than maxChunkSize. It does this regardless of protocol
>> version, so it will produce HTTP 1.0 request and responses that use
>> Transfer-Encoding: chunked.
>>
>> It seems like it's always wrong whenever that happens, no? It could
>> potentially only convert to chunked encoding if the version is HTTP
>> 1.1, but I'm pretty sure the current implementation will break things
>> other than LittleProxy.
>>
>> Thanks again.
>>
>> -Adam
>>
>>
>> On Wed, Nov 11, 2009 at 1:41 AM, Trustin Lee (이희승) <trustin at gmail.com> wrote:
>>> I guess I found a problem with upgrading the protocol version automatically.
>>>
>>> If a client sends a request in HTTP/1.0 (i.e. no chunked encoding
>>> support on the client side), the server (Netty HTTP) should not
>>> respond with chunked content.  If the decoder upgrades the protocol
>>> version, there's no way for an HTTP request handler to determine if
>>> the client actually supports HTTP/1.1 or not.
>>>
>>> Therefore, the original protocol version needs to be retained instead
>>> of being simply overridden by the decoder.  Actually, I don't see a
>>> problem with retaining the original version because a broken client
>>> implementation can send such a request and the request handler should
>>> be able to handle it anyway.
>>>
>>> Because HttpMessageDecoder should not modify the original protocol
>>> version, LittleProxy will have to handle protocol version upgrade and
>>> downgrade by itself like the following:
>>>
>>> For the HTTP proxy to send the chunked content converted by
>>> HttpRequestDecoder to the actual HTTP server, the proxy can either:
>>>
>>> 1) modify the outbound HTTP request by calling
>>> HttpMessage.setProtocolVersion() and store the actual client protocol
>>> version somewhere (assuming NETTY-247[1] has been resolved), or
>>> 2) create a new outbound HTTP request object with upgraded protocol version.
>>>
>>> If the proxy receives the chunked content converted by
>>> HttpResponseDecoder from the actual server, the same modification
>>> needs to be performed against the received HttpResponse.  If the
>>> client sent HTTP/1.0 request, the received chunks have to be merged
>>> back because the client will probably not understand chunked encoding.
>>>
>>> Another tricky case would be the case where the actual server does not
>>> understand chunked encoding.  In such a case, the actual server will
>>> respond with "HTTP/1.0 406 Not Acceptable" and the proxy server should
>>> retry after merging the chunks and downgrading the protocol version.
>>>
>>> Please let me know if I am missing something.  Otherwise, I will
>>> retain the current behavior while adding setter methods to
>>> HttpMessage.
>>>
>>> Thanks
>>>
>>> — Trustin Lee, http://gleamynode.net/
>>>
>>> [1] NETTY-247: https://jira.jboss.org/jira/browse/NETTY-247
>>>
>>> On Wed, Nov 4, 2009 at 3:34 PM, Adam Fisk <a at littleshoot.org> wrote:
>>>> OK - finally knocked this off, and it's working beautifully in FireFox and
>>>> Safari. All the credit really goes to Stephen - I'm just implementing his
>>>> fix here.
>>>> I agree with some of the earlier comments that since HttpMessage and its
>>>> subclasses are already mutable, it's not a big loss to add another mutable
>>>> field. That's the approach I took with the patch. I also think it makes
>>>> sense to keep HttpChunk immutable unless there's a concrete need to make it
>>>> mutable. I also don't think this is indicative of a design flaw.
>>>> Feel free to change the style, approach, or comments as you see fit,
>>>> Trustin. Thanks again Stephen. I like that handy netcat trick, by the way!
>>>> -Adam
>>>>
>>>> On Tue, Nov 3, 2009 at 8:47 PM, Stephen Haberman <stephen at exigencecorp.com>
>>>> wrote:
>>>>>
>>>>> > I'm also happy to give you commit permissions if you like. This in
>>>>> > particular is clearly a Netty and not a LittleProxy issue, but I can
>>>>> > give you access to the repo if you need to make any changes going
>>>>> > forward.
>>>>>
>>>>> That's fine--I enjoyed tracking the bug down and also reading through
>>>>> the LittleProxy code.
>>>>>
>>>>> My proxy is more of a weekend (or two or three) hobby project--I
>>>>> wouldn't be surprised if its missing several boundary cases that
>>>>> LittleProxy handles more robustly.
>>>>>
>>>>> But I did finally get it into github:
>>>>>
>>>>> http://github.com/stephenh/fakehttp
>>>>>
>>>>> Being a hobby project, I also wrote it in Scala and, purposefully
>>>>> trying to keep it small, have kept it to ~450 LOC so far.
>>>>>
>>>>> - Stephen
>>>>>
>>>>> _______________________________________________
>>>>> netty-users mailing list
>>>>> netty-users at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>
>>>>
>>>>
>>>> --
>>>> Adam Fisk
>>>> http://www.littleshoot.org | http://adamfisk.wordpress.com |
>>>> http://twitter.com/adamfisk
>>>>
>>>> _______________________________________________
>>>> netty-users mailing list
>>>> netty-users at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>
>>>>
>>>
>>> _______________________________________________
>>> netty-users mailing list
>>> netty-users at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>
>>
>>
>>
>> --
>> Adam Fisk
>> http://www.littleshoot.org | http://adamfisk.wordpress.com |
>> http://twitter.com/adamfisk
>>
>> _______________________________________________
>> netty-users mailing list
>> netty-users at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/netty-users
>>
>
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users
>

-- 
Adam Fisk
http://www.littleshoot.org | http://adamfisk.wordpress.com |
http://twitter.com/adamfisk