Announcing LittleProxy 0.1 based on Netty

Wed Nov 18 03:23:26 EST 2009

Hi Trustin- Great!!! Which docs need reviewing exactly? The ones in
trunk don't look changed -- on the 3.2 branch? Will definitely give
them a look if I'm not too late. -Adam

2009/11/17 Trustin Lee (이희승) <trustin at gmail.com>:
> All done.  HttpMessageDecoder does not change the Transfer-Encoding
> header anymore.  HttpMessage maintains an additional boolean flag to
> handle the case where the content split occurs.  Consequently, the
> semantic of HttpMessage.chunked property has been changed.
>
> However, I did not rename HttpChunk and HttpMessage.isChunked() yet
> because I think there should be better names and the current behavior
> is backward-compatible.
>
> Could you review the updated Javadoc of HttpMessage.is/setChunked()
> and HttpMessageDecoder?  Also, any suggestion for the new names would
> be appreciated.
>
> Cheers
>
> -- Trustin Lee, http://gleamynode.net/
>
>
>
> 2009/11/17 Trustin Lee (이희승) <trustin at gmail.com>:
>> Actually, I've just realized that trailing headers are allowed only at
>> the last chunk.  Therefore we don't need to add the header accessors
>> to any chunk messages but the last one.  We can create a new subtype
>> of HttpPartialContent to represent the last chunk.  Problem solved ..
>> or my knowledge base fixed! :)
>>
>> Cheers,
>>
>> -- Trustin Lee, http://gleamynode.net/
>>
>> 2009/11/17 Trustin Lee (이희승) <trustin at gmail.com>:
>>> Sure.  No problem.  Until the best solution is found for trailing
>>> headers, we can apply the solution we've reached to so far.  Here's
>>> the JIRA issue, just in case you want to keep track of the progress or
>>> you want to contribute the fix:
>>>
>>>    https://jira.jboss.org/jira/browse/NETTY-250
>>>
>>> -- Trustin Lee, http://gleamynode.net/
>>>
>>>
>>> On Mon, Nov 16, 2009 at 7:02 AM, Adam Fisk <a at littleshoot.org> wrote:
>>>> I'm not super familiar with the trailing headers, Trustin - I need to
>>>> look at the spec a bit. I'm assuming you don't feel comfortable
>>>> trusting clients to send reasonably-sized chunks though right? I guess
>>>> that would be one easy way to bring down a server if an attacker
>>>> wanted to.
>>>>
>>>> I didn't realize you were re-chunking bodies that are already chunked.
>>>> I'd need to look at the code, but I'm juggling a few things right now
>>>> and don't know when I'll get to it.
>>>>
>>>> -Adam
>>>>
>>>>
>>>> On Fri, Nov 13, 2009 at 12:14 AM, Trustin Lee (이희승) <trustin at gmail.com> wrote:
>>>>> Glad that you like the proposal! :)
>>>>>
>>>>> I realized that there is another issue with automatic conversion
>>>>> though.  What should we do if a client sends the content in chunked
>>>>> encoding and the chunks contain trailing headers such as
>>>>> 'Content-MD5'?  The current implementation splits a large chunk into
>>>>> multiple chunks, and it can be a problem when the decoder splits the
>>>>> chunk with trailing headers.
>>>>>
>>>>> -- Trustin Lee, http://gleamynode.net/
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Nov 12, 2009 at 2:27 PM, Adam Fisk <a at littleshoot.org> wrote:
>>>>>> I think that sounds like the perfect solution, Trustin. It allows you
>>>>>> to fulfill the desired goal of not holding a huge message body in
>>>>>> memory while getting around the issue having somewhat complex logic
>>>>>> for when to add Transfer-Encoding. So it not only solves the problem,
>>>>>> but it overall simplifies the code. Big thumbs up from me!
>>>>>>
>>>>>> -Adam
>>>>>>
>>>>>>
>>>>>> On Wed, Nov 11, 2009 at 8:14 PM, Trustin Lee (이희승) <trustin at gmail.com> wrote:
>>>>>>> Hi Adam,
>>>>>>>
>>>>>>> Strictly speaking, it is wrong to convert a HTTP/1.0 message to
>>>>>>> chunked encoding.  To conform to the standard, the decoder could be
>>>>>>> modified to disable the conversion for HTTP/1.0 messages, but I don't
>>>>>>> think this is what people expects from Netty.  No conversion means
>>>>>>> greater latency and memory footprint.
>>>>>>>
>>>>>>> So, I'd like to suggest to:
>>>>>>>
>>>>>>> 1) Redefine HttpChunk - say it can be used for either chunked encoding
>>>>>>> or just large content streaming,
>>>>>>> 2) modify the decoder not to update the 'Transfer-Encoding' while
>>>>>>> letting HttpMessage.isChunked() return true, and
>>>>>>> 3) modify the encoder not to generate chunked response if the message
>>>>>>> version is old.
>>>>>>>
>>>>>>> By doing this, we do not violate standard but simply redefine how
>>>>>>> large content is delivered.
>>>>>>>
>>>>>>> However, the name HttpChunk becomes confusing, so I'd like to
>>>>>>> introduce a new type - HttpPartialContent:
>>>>>>>
>>>>>>> public interface HttpPartialContent {
>>>>>>>    // get/setContent() and isLast()
>>>>>>> }
>>>>>>> public interface HttpChunk extends HttpPartialContent {
>>>>>>>    // access methods for trailing headers
>>>>>>> }
>>>>>>>
>>>>>>> also HttpMessage.isChunked() should be replaced with
>>>>>>> 'hasFollowingPartialContent()' or something similar (better name
>>>>>>> suggestion?)
>>>>>>>
>>>>>>> WDYT?
>>>>>>>
>>>>>>> -- Trustin Lee, http://gleamynode.net/
>>>>>>>
>>>>>>> On Thu, Nov 12, 2009 at 11:36 AM, Adam Fisk <a at littleshoot.org> wrote:
>>>>>>>> Hi Trustin- Thanks for the detailed response. I already fixed this
>>>>>>>> independently in LittleProxy running with Netty 3.1.5 through creating
>>>>>>>> a new message, so it's not an issue there. I *think* there's still a
>>>>>>>> problem in Netty in the following case, however.
>>>>>>>>
>>>>>>>> Leaving the proxy case aside, the straight HttpMessageDecoder will
>>>>>>>> switch to chunked encoding any time the response body length is
>>>>>>>> greater than maxChunkSize. It does this regardless of protocol
>>>>>>>> version, so it will produce HTTP 1.0 request and responses that use
>>>>>>>> Transfer-Encoding: chunked.
>>>>>>>>
>>>>>>>> It seems like it's always wrong whenever that happens, no? It could
>>>>>>>> potentially only convert to chunked encoding if the version is HTTP
>>>>>>>> 1.1, but I'm pretty sure the current implementation will break things
>>>>>>>> other than LittleProxy.
>>>>>>>>
>>>>>>>> Thanks again.
>>>>>>>>
>>>>>>>> -Adam
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Nov 11, 2009 at 1:41 AM, Trustin Lee (이희승) <trustin at gmail.com> wrote:
>>>>>>>>> I guess I found a problem with upgrading the protocol version automatically.
>>>>>>>>>
>>>>>>>>> If a client sends a request in HTTP/1.0 (i.e. no chunked encoding
>>>>>>>>> support on the client side), the server (Netty HTTP) should not
>>>>>>>>> respond with chunked content.  If the decoder upgrades the protocol
>>>>>>>>> version, there's no way for an HTTP request handler to determine if
>>>>>>>>> the client actually supports HTTP/1.1 or not.
>>>>>>>>>
>>>>>>>>> Therefore, the original protocol version needs to be retained instead
>>>>>>>>> of being simply overridden by the decoder.  Actually, I don't see a
>>>>>>>>> problem with retaining the original version because a broken client
>>>>>>>>> implementation can send such a request and the request handler should
>>>>>>>>> be able to handle it anyway.
>>>>>>>>>
>>>>>>>>> Because HttpMessageDecoder should not modify the original protocol
>>>>>>>>> version, LittleProxy will have to handle protocol version upgrade and
>>>>>>>>> downgrade by itself like the following:
>>>>>>>>>
>>>>>>>>> For the HTTP proxy to send the chunked content converted by
>>>>>>>>> HttpRequestDecoder to the actual HTTP server, the proxy can either:
>>>>>>>>>
>>>>>>>>> 1) modify the outbound HTTP request by calling
>>>>>>>>> HttpMessage.setProtocolVersion() and store the actual client protocol
>>>>>>>>> version somewhere (assuming NETTY-247[1] has been resolved), or
>>>>>>>>> 2) create a new outbound HTTP request object with upgraded protocol version.
>>>>>>>>>
>>>>>>>>> If the proxy receives the chunked content converted by
>>>>>>>>> HttpResponseDecoder from the actual server, the same modification
>>>>>>>>> needs to be performed against the received HttpResponse.  If the
>>>>>>>>> client sent HTTP/1.0 request, the received chunks have to be merged
>>>>>>>>> back because the client will probably not understand chunked encoding.
>>>>>>>>>
>>>>>>>>> Another tricky case would be the case where the actual server does not
>>>>>>>>> understand chunked encoding.  In such a case, the actual server will
>>>>>>>>> respond with "HTTP/1.0 406 Not Acceptable" and the proxy server should
>>>>>>>>> retry after merging the chunks and downgrading the protocol version.
>>>>>>>>>
>>>>>>>>> Please let me know if I am missing something.  Otherwise, I will
>>>>>>>>> retain the current behavior while adding setter methods to
>>>>>>>>> HttpMessage.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> -- Trustin Lee, http://gleamynode.net/
>>>>>>>>>
>>>>>>>>> [1] NETTY-247: https://jira.jboss.org/jira/browse/NETTY-247
>>>>>>>>>
>>>>>>>>> On Wed, Nov 4, 2009 at 3:34 PM, Adam Fisk <a at littleshoot.org> wrote:
>>>>>>>>>> OK - finally knocked this off, and it's working beautifully in FireFox and
>>>>>>>>>> Safari. All the credit really goes to Stephen - I'm just implementing his
>>>>>>>>>> fix here.
>>>>>>>>>> I agree with some of the earlier comments that since HttpMessage and its
>>>>>>>>>> subclasses are already mutable, it's not a big loss to add another mutable
>>>>>>>>>> field. That's the approach I took with the patch. I also think it makes
>>>>>>>>>> sense to keep HttpChunk immutable unless there's a concrete need to make it
>>>>>>>>>> mutable. I also don't think this is indicative of a design flaw.
>>>>>>>>>> Feel free to change the style, approach, or comments as you see fit,
>>>>>>>>>> Trustin. Thanks again Stephen. I like that handy netcat trick, by the way!
>>>>>>>>>> -Adam
>>>>>>>>>>
>>>>>>>>>> On Tue, Nov 3, 2009 at 8:47 PM, Stephen Haberman <stephen at exigencecorp.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> > I'm also happy to give you commit permissions if you like. This in
>>>>>>>>>>> > particular is clearly a Netty and not a LittleProxy issue, but I can
>>>>>>>>>>> > give you access to the repo if you need to make any changes going
>>>>>>>>>>> > forward.
>>>>>>>>>>>
>>>>>>>>>>> That's fine--I enjoyed tracking the bug down and also reading through
>>>>>>>>>>> the LittleProxy code.
>>>>>>>>>>>
>>>>>>>>>>> My proxy is more of a weekend (or two or three) hobby project--I
>>>>>>>>>>> wouldn't be surprised if its missing several boundary cases that
>>>>>>>>>>> LittleProxy handles more robustly.
>>>>>>>>>>>
>>>>>>>>>>> But I did finally get it into github:
>>>>>>>>>>>
>>>>>>>>>>> http://github.com/stephenh/fakehttp
>>>>>>>>>>>
>>>>>>>>>>> Being a hobby project, I also wrote it in Scala and, purposefully
>>>>>>>>>>> trying to keep it small, have kept it to ~450 LOC so far.
>>>>>>>>>>>
>>>>>>>>>>> - Stephen
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> netty-users mailing list
>>>>>>>>>>> netty-users at lists.jboss.org
>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Adam Fisk
>>>>>>>>>> http://www.littleshoot.org | http://adamfisk.wordpress.com |
>>>>>>>>>> http://twitter.com/adamfisk
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> netty-users mailing list
>>>>>>>>>> netty-users at lists.jboss.org
>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> netty-users mailing list
>>>>>>>>> netty-users at lists.jboss.org
>>>>>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Adam Fisk
>>>>>>>> http://www.littleshoot.org | http://adamfisk.wordpress.com |
>>>>>>>> http://twitter.com/adamfisk
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> netty-users mailing list
>>>>>>>> netty-users at lists.jboss.org
>>>>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> netty-users mailing list
>>>>>>> netty-users at lists.jboss.org
>>>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Adam Fisk
>>>>>> http://www.littleshoot.org | http://adamfisk.wordpress.com |
>>>>>> http://twitter.com/adamfisk
>>>>>>
>>>>>> _______________________________________________
>>>>>> netty-users mailing list
>>>>>> netty-users at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> netty-users mailing list
>>>>> netty-users at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Adam Fisk
>>>> http://www.littleshoot.org | http://adamfisk.wordpress.com |
>>>> http://twitter.com/adamfisk
>>>>
>>>> _______________________________________________
>>>> netty-users mailing list
>>>> netty-users at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>
>>>
>>
>
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users
>

-- 
Adam Fisk
http://www.littleshoot.org | http://adamfisk.wordpress.com |
http://twitter.com/adamfisk