Announcing LittleProxy 0.1 based on Netty

Wed Nov 11 23:14:00 EST 2009

Hi Adam,

Strictly speaking, it is wrong to convert a HTTP/1.0 message to
chunked encoding.  To conform to the standard, the decoder could be
modified to disable the conversion for HTTP/1.0 messages, but I don't
think this is what people expects from Netty.  No conversion means
greater latency and memory footprint.

So, I'd like to suggest to:

1) Redefine HttpChunk - say it can be used for either chunked encoding
or just large content streaming,
2) modify the decoder not to update the 'Transfer-Encoding' while
letting HttpMessage.isChunked() return true, and
3) modify the encoder not to generate chunked response if the message
version is old.

By doing this, we do not violate standard but simply redefine how
large content is delivered.

However, the name HttpChunk becomes confusing, so I'd like to
introduce a new type - HttpPartialContent:

public interface HttpPartialContent {
    // get/setContent() and isLast()
}
public interface HttpChunk extends HttpPartialContent {
    // access methods for trailing headers
}

also HttpMessage.isChunked() should be replaced with
'hasFollowingPartialContent()' or something similar (better name
suggestion?)

WDYT?

— Trustin Lee, http://gleamynode.net/

On Thu, Nov 12, 2009 at 11:36 AM, Adam Fisk <a at littleshoot.org> wrote:
> Hi Trustin- Thanks for the detailed response. I already fixed this
> independently in LittleProxy running with Netty 3.1.5 through creating
> a new message, so it's not an issue there. I *think* there's still a
> problem in Netty in the following case, however.
>
> Leaving the proxy case aside, the straight HttpMessageDecoder will
> switch to chunked encoding any time the response body length is
> greater than maxChunkSize. It does this regardless of protocol
> version, so it will produce HTTP 1.0 request and responses that use
> Transfer-Encoding: chunked.
>
> It seems like it's always wrong whenever that happens, no? It could
> potentially only convert to chunked encoding if the version is HTTP
> 1.1, but I'm pretty sure the current implementation will break things
> other than LittleProxy.
>
> Thanks again.
>
> -Adam
>
>
> On Wed, Nov 11, 2009 at 1:41 AM, Trustin Lee (이희승) <trustin at gmail.com> wrote:
>> I guess I found a problem with upgrading the protocol version automatically.
>>
>> If a client sends a request in HTTP/1.0 (i.e. no chunked encoding
>> support on the client side), the server (Netty HTTP) should not
>> respond with chunked content.  If the decoder upgrades the protocol
>> version, there's no way for an HTTP request handler to determine if
>> the client actually supports HTTP/1.1 or not.
>>
>> Therefore, the original protocol version needs to be retained instead
>> of being simply overridden by the decoder.  Actually, I don't see a
>> problem with retaining the original version because a broken client
>> implementation can send such a request and the request handler should
>> be able to handle it anyway.
>>
>> Because HttpMessageDecoder should not modify the original protocol
>> version, LittleProxy will have to handle protocol version upgrade and
>> downgrade by itself like the following:
>>
>> For the HTTP proxy to send the chunked content converted by
>> HttpRequestDecoder to the actual HTTP server, the proxy can either:
>>
>> 1) modify the outbound HTTP request by calling
>> HttpMessage.setProtocolVersion() and store the actual client protocol
>> version somewhere (assuming NETTY-247[1] has been resolved), or
>> 2) create a new outbound HTTP request object with upgraded protocol version.
>>
>> If the proxy receives the chunked content converted by
>> HttpResponseDecoder from the actual server, the same modification
>> needs to be performed against the received HttpResponse.  If the
>> client sent HTTP/1.0 request, the received chunks have to be merged
>> back because the client will probably not understand chunked encoding.
>>
>> Another tricky case would be the case where the actual server does not
>> understand chunked encoding.  In such a case, the actual server will
>> respond with "HTTP/1.0 406 Not Acceptable" and the proxy server should
>> retry after merging the chunks and downgrading the protocol version.
>>
>> Please let me know if I am missing something.  Otherwise, I will
>> retain the current behavior while adding setter methods to
>> HttpMessage.
>>
>> Thanks
>>
>> — Trustin Lee, http://gleamynode.net/
>>
>> [1] NETTY-247: https://jira.jboss.org/jira/browse/NETTY-247
>>
>> On Wed, Nov 4, 2009 at 3:34 PM, Adam Fisk <a at littleshoot.org> wrote:
>>> OK - finally knocked this off, and it's working beautifully in FireFox and
>>> Safari. All the credit really goes to Stephen - I'm just implementing his
>>> fix here.
>>> I agree with some of the earlier comments that since HttpMessage and its
>>> subclasses are already mutable, it's not a big loss to add another mutable
>>> field. That's the approach I took with the patch. I also think it makes
>>> sense to keep HttpChunk immutable unless there's a concrete need to make it
>>> mutable. I also don't think this is indicative of a design flaw.
>>> Feel free to change the style, approach, or comments as you see fit,
>>> Trustin. Thanks again Stephen. I like that handy netcat trick, by the way!
>>> -Adam
>>>
>>> On Tue, Nov 3, 2009 at 8:47 PM, Stephen Haberman <stephen at exigencecorp.com>
>>> wrote:
>>>>
>>>> > I'm also happy to give you commit permissions if you like. This in
>>>> > particular is clearly a Netty and not a LittleProxy issue, but I can
>>>> > give you access to the repo if you need to make any changes going
>>>> > forward.
>>>>
>>>> That's fine--I enjoyed tracking the bug down and also reading through
>>>> the LittleProxy code.
>>>>
>>>> My proxy is more of a weekend (or two or three) hobby project--I
>>>> wouldn't be surprised if its missing several boundary cases that
>>>> LittleProxy handles more robustly.
>>>>
>>>> But I did finally get it into github:
>>>>
>>>> http://github.com/stephenh/fakehttp
>>>>
>>>> Being a hobby project, I also wrote it in Scala and, purposefully
>>>> trying to keep it small, have kept it to ~450 LOC so far.
>>>>
>>>> - Stephen
>>>>
>>>> _______________________________________________
>>>> netty-users mailing list
>>>> netty-users at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>
>>>
>>>
>>> --
>>> Adam Fisk
>>> http://www.littleshoot.org | http://adamfisk.wordpress.com |
>>> http://twitter.com/adamfisk
>>>
>>> _______________________________________________
>>> netty-users mailing list
>>> netty-users at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>
>>>
>>
>> _______________________________________________
>> netty-users mailing list
>> netty-users at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/netty-users
>>
>
>
>
> --
> Adam Fisk
> http://www.littleshoot.org | http://adamfisk.wordpress.com |
> http://twitter.com/adamfisk
>
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users
>