Announcing LittleProxy 0.1 based on Netty

Trustin Lee (이희승) trustin at gmail.com
Thu Nov 19 23:47:56 EST 2009 

I forgot to deploy the javadoc :)

http://jboss.org/file-access/default/members/netty/freezone/api/3.2/org/jboss/netty/handler/codec/http/HttpMessageDecoder.html

and

http://jboss.org/file-access/default/members/netty/freezone/api/3.2/org/jboss/netty/handler/codec/http/HttpMessage.html#isChunked()

Cheers

-- Trustin Lee, http://gleamynode.net/



2009/11/18 Adam Fisk <a at littleshoot.org>:
> Hi Trustin- Great!!! Which docs need reviewing exactly? The ones in
> trunk don't look changed -- on the 3.2 branch? Will definitely give
> them a look if I'm not too late. -Adam
>
> 2009/11/17 Trustin Lee (이희승) <trustin at gmail.com>:
>> All done.  HttpMessageDecoder does not change the Transfer-Encoding
>> header anymore.  HttpMessage maintains an additional boolean flag to
>> handle the case where the content split occurs.  Consequently, the
>> semantic of HttpMessage.chunked property has been changed.
>>
>> However, I did not rename HttpChunk and HttpMessage.isChunked() yet
>> because I think there should be better names and the current behavior
>> is backward-compatible.
>>
>> Could you review the updated Javadoc of HttpMessage.is/setChunked()
>> and HttpMessageDecoder?  Also, any suggestion for the new names would
>> be appreciated.
>>
>> Cheers
>>
>> -- Trustin Lee, http://gleamynode.net/
>>
>>
>>
>> 2009/11/17 Trustin Lee (이희승) <trustin at gmail.com>:
>>> Actually, I've just realized that trailing headers are allowed only at
>>> the last chunk.  Therefore we don't need to add the header accessors
>>> to any chunk messages but the last one.  We can create a new subtype
>>> of HttpPartialContent to represent the last chunk.  Problem solved ..
>>> or my knowledge base fixed! :)
>>>
>>> Cheers,
>>>
>>> -- Trustin Lee, http://gleamynode.net/
>>>
>>> 2009/11/17 Trustin Lee (이희승) <trustin at gmail.com>:
>>>> Sure.  No problem.  Until the best solution is found for trailing
>>>> headers, we can apply the solution we've reached to so far.  Here's
>>>> the JIRA issue, just in case you want to keep track of the progress or
>>>> you want to contribute the fix:
>>>>
>>>>    https://jira.jboss.org/jira/browse/NETTY-250
>>>>
>>>> -- Trustin Lee, http://gleamynode.net/
>>>>
>>>>
>>>> On Mon, Nov 16, 2009 at 7:02 AM, Adam Fisk <a at littleshoot.org> wrote:
>>>>> I'm not super familiar with the trailing headers, Trustin - I need to
>>>>> look at the spec a bit. I'm assuming you don't feel comfortable
>>>>> trusting clients to send reasonably-sized chunks though right? I guess
>>>>> that would be one easy way to bring down a server if an attacker
>>>>> wanted to.
>>>>>
>>>>> I didn't realize you were re-chunking bodies that are already chunked.
>>>>> I'd need to look at the code, but I'm juggling a few things right now
>>>>> and don't know when I'll get to it.
>>>>>
>>>>> -Adam
>>>>>
>>>>>
>>>>> On Fri, Nov 13, 2009 at 12:14 AM, Trustin Lee (이희승) <trustin at gmail.com> wrote:
>>>>>> Glad that you like the proposal! :)
>>>>>>
>>>>>> I realized that there is another issue with automatic conversion
>>>>>> though.  What should we do if a client sends the content in chunked
>>>>>> encoding and the chunks contain trailing headers such as
>>>>>> 'Content-MD5'?  The current implementation splits a large chunk into
>>>>>> multiple chunks, and it can be a problem when the decoder splits the
>>>>>> chunk with trailing headers.
>>>>>>
>>>>>> -- Trustin Lee, http://gleamynode.net/
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Nov 12, 2009 at 2:27 PM, Adam Fisk <a at littleshoot.org> wrote:
>>>>>>> I think that sounds like the perfect solution, Trustin. It allows you
>>>>>>> to fulfill the desired goal of not holding a huge message body in
>>>>>>> memory while getting around the issue having somewhat complex logic
>>>>>>> for when to add Transfer-Encoding. So it not only solves the problem,
>>>>>>> but it overall simplifies the code. Big thumbs up from me!
>>>>>>>
>>>>>>> -Adam
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Nov 11, 2009 at 8:14 PM, Trustin Lee (이희승) <trustin at gmail.com> wrote:
>>>>>>>> Hi Adam,
>>>>>>>>
>>>>>>>> Strictly speaking, it is wrong to convert a HTTP/1.0 message to
>>>>>>>> chunked encoding.  To conform to the standard, the decoder could be
>>>>>>>> modified to disable the conversion for HTTP/1.0 messages, but I don't
>>>>>>>> think this is what people expects from Netty.  No conversion means
>>>>>>>> greater latency and memory footprint.
>>>>>>>>
>>>>>>>> So, I'd like to suggest to:
>>>>>>>>
>>>>>>>> 1) Redefine HttpChunk - say it can be used for either chunked encoding
>>>>>>>> or just large content streaming,
>>>>>>>> 2) modify the decoder not to update the 'Transfer-Encoding' while
>>>>>>>> letting HttpMessage.isChunked() return true, and
>>>>>>>> 3) modify the encoder not to generate chunked response if the message
>>>>>>>> version is old.
>>>>>>>>
>>>>>>>> By doing this, we do not violate standard but simply redefine how
>>>>>>>> large content is delivered.
>>>>>>>>
>>>>>>>> However, the name HttpChunk becomes confusing, so I'd like to
>>>>>>>> introduce a new type - HttpPartialContent:
>>>>>>>>
>>>>>>>> public interface HttpPartialContent {
>>>>>>>>    // get/setContent() and isLast()
>>>>>>>> }
>>>>>>>> public interface HttpChunk extends HttpPartialContent {
>>>>>>>>    // access methods for trailing headers
>>>>>>>> }
>>>>>>>>
>>>>>>>> also HttpMessage.isChunked() should be replaced with
>>>>>>>> 'hasFollowingPartialContent()' or something similar (better name
>>>>>>>> suggestion?)
>>>>>>>>
>>>>>>>> WDYT?
>>>>>>>>
>>>>>>>> -- Trustin Lee, http://gleamynode.net/
>>>>>>>>
>>>>>>>> On Thu, Nov 12, 2009 at 11:36 AM, Adam Fisk <a at littleshoot.org> wrote:
>>>>>>>>> Hi Trustin- Thanks for the detailed response. I already fixed this
>>>>>>>>> independently in LittleProxy running with Netty 3.1.5 through creating
>>>>>>>>> a new message, so it's not an issue there. I *think* there's still a
>>>>>>>>> problem in Netty in the following case, however.
>>>>>>>>>
>>>>>>>>> Leaving the proxy case aside, the straight HttpMessageDecoder will
>>>>>>>>> switch to chunked encoding any time the response body length is
>>>>>>>>> greater than maxChunkSize. It does this regardless of protocol
>>>>>>>>> version, so it will produce HTTP 1.0 request and responses that use
>>>>>>>>> Transfer-Encoding: chunked.
>>>>>>>>>
>>>>>>>>> It seems like it's always wrong whenever that happens, no? It could
>>>>>>>>> potentially only convert to chunked encoding if the version is HTTP
>>>>>>>>> 1.1, but I'm pretty sure the current implementation will break things
>>>>>>>>> other than LittleProxy.
>>>>>>>>>
>>>>>>>>> Thanks again.
>>>>>>>>>
>>>>>>>>> -Adam
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Nov 11, 2009 at 1:41 AM, Trustin Lee (이희승) <trustin at gmail.com> wrote:
>>>>>>>>>> I guess I found a problem with upgrading the protocol version automatically.
>>>>>>>>>>
>>>>>>>>>> If a client sends a request in HTTP/1.0 (i.e. no chunked encoding
>>>>>>>>>> support on the client side), the server (Netty HTTP) should not
>>>>>>>>>> respond with chunked content.  If the decoder upgrades the protocol
>>>>>>>>>> version, there's no way for an HTTP request handler to determine if
>>>>>>>>>> the client actually supports HTTP/1.1 or not.
>>>>>>>>>>
>>>>>>>>>> Therefore, the original protocol version needs to be retained instead
>>>>>>>>>> of being simply overridden by the decoder.  Actually, I don't see a
>>>>>>>>>> problem with retaining the original version because a broken client
>>>>>>>>>> implementation can send such a request and the request handler should
>>>>>>>>>> be able to handle it anyway.
>>>>>>>>>>
>>>>>>>>>> Because HttpMessageDecoder should not modify the original protocol
>>>>>>>>>> version, LittleProxy will have to handle protocol version upgrade and
>>>>>>>>>> downgrade by itself like the following:
>>>>>>>>>>
>>>>>>>>>> For the HTTP proxy to send the chunked content converted by
>>>>>>>>>> HttpRequestDecoder to the actual HTTP server, the proxy can either:
>>>>>>>>>>
>>>>>>>>>> 1) modify the outbound HTTP request by calling
>>>>>>>>>> HttpMessage.setProtocolVersion() and store the actual client protocol
>>>>>>>>>> version somewhere (assuming NETTY-247[1] has been resolved), or
>>>>>>>>>> 2) create a new outbound HTTP request object with upgraded protocol version.
>>>>>>>>>>
>>>>>>>>>> If the proxy receives the chunked content converted by
>>>>>>>>>> HttpResponseDecoder from the actual server, the same modification
>>>>>>>>>> needs to be performed against the received HttpResponse.  If the
>>>>>>>>>> client sent HTTP/1.0 request, the received chunks have to be merged
>>>>>>>>>> back because the client will probably not understand chunked encoding.
>>>>>>>>>>
>>>>>>>>>> Another tricky case would be the case where the actual server does not
>>>>>>>>>> understand chunked encoding.  In such a case, the actual server will
>>>>>>>>>> respond with "HTTP/1.0 406 Not Acceptable" and the proxy server should
>>>>>>>>>> retry after merging the chunks and downgrading the protocol version.
>>>>>>>>>>
>>>>>>>>>> Please let me know if I am missing something.  Otherwise, I will
>>>>>>>>>> retain the current behavior while adding setter methods to
>>>>>>>>>> HttpMessage.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>> -- Trustin Lee, http://gleamynode.net/
>>>>>>>>>>
>>>>>>>>>> [1] NETTY-247: https://jira.jboss.org/jira/browse/NETTY-247
>>>>>>>>>>
>>>>>>>>>> On Wed, Nov 4, 2009 at 3:34 PM, Adam Fisk <a at littleshoot.org> wrote:
>>>>>>>>>>> OK - finally knocked this off, and it's working beautifully in FireFox and
>>>>>>>>>>> Safari. All the credit really goes to Stephen - I'm just implementing his
>>>>>>>>>>> fix here.
>>>>>>>>>>> I agree with some of the earlier comments that since HttpMessage and its
>>>>>>>>>>> subclasses are already mutable, it's not a big loss to add another mutable
>>>>>>>>>>> field. That's the approach I took with the patch. I also think it makes
>>>>>>>>>>> sense to keep HttpChunk immutable unless there's a concrete need to make it
>>>>>>>>>>> mutable. I also don't think this is indicative of a design flaw.
>>>>>>>>>>> Feel free to change the style, approach, or comments as you see fit,
>>>>>>>>>>> Trustin. Thanks again Stephen. I like that handy netcat trick, by the way!
>>>>>>>>>>> -Adam
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Nov 3, 2009 at 8:47 PM, Stephen Haberman <stephen at exigencecorp.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> > I'm also happy to give you commit permissions if you like. This in
>>>>>>>>>>>> > particular is clearly a Netty and not a LittleProxy issue, but I can
>>>>>>>>>>>> > give you access to the repo if you need to make any changes going
>>>>>>>>>>>> > forward.
>>>>>>>>>>>>
>>>>>>>>>>>> That's fine--I enjoyed tracking the bug down and also reading through
>>>>>>>>>>>> the LittleProxy code.
>>>>>>>>>>>>
>>>>>>>>>>>> My proxy is more of a weekend (or two or three) hobby project--I
>>>>>>>>>>>> wouldn't be surprised if its missing several boundary cases that
>>>>>>>>>>>> LittleProxy handles more robustly.
>>>>>>>>>>>>
>>>>>>>>>>>> But I did finally get it into github:
>>>>>>>>>>>>
>>>>>>>>>>>> http://github.com/stephenh/fakehttp
>>>>>>>>>>>>
>>>>>>>>>>>> Being a hobby project, I also wrote it in Scala and, purposefully
>>>>>>>>>>>> trying to keep it small, have kept it to ~450 LOC so far.
>>>>>>>>>>>>
>>>>>>>>>>>> - Stephen
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> netty-users mailing list
>>>>>>>>>>>> netty-users at lists.jboss.org
>>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Adam Fisk
>>>>>>>>>>> http://www.littleshoot.org | http://adamfisk.wordpress.com |
>>>>>>>>>>> http://twitter.com/adamfisk
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> netty-users mailing list
>>>>>>>>>>> netty-users at lists.jboss.org
>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> netty-users mailing list
>>>>>>>>>> netty-users at lists.jboss.org
>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Adam Fisk
>>>>>>>>> http://www.littleshoot.org | http://adamfisk.wordpress.com |
>>>>>>>>> http://twitter.com/adamfisk
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> netty-users mailing list
>>>>>>>>> netty-users at lists.jboss.org
>>>>>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> netty-users mailing list
>>>>>>>> netty-users at lists.jboss.org
>>>>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Adam Fisk
>>>>>>> http://www.littleshoot.org | http://adamfisk.wordpress.com |
>>>>>>> http://twitter.com/adamfisk
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> netty-users mailing list
>>>>>>> netty-users at lists.jboss.org
>>>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> netty-users mailing list
>>>>>> netty-users at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Adam Fisk
>>>>> http://www.littleshoot.org | http://adamfisk.wordpress.com |
>>>>> http://twitter.com/adamfisk
>>>>>
>>>>> _______________________________________________
>>>>> netty-users mailing list
>>>>> netty-users at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>
>>>>
>>>
>>
>> _______________________________________________
>> netty-users mailing list
>> netty-users at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/netty-users
>>
>
>
>
> --
> Adam Fisk
> http://www.littleshoot.org | http://adamfisk.wordpress.com |
> http://twitter.com/adamfisk
>
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users
>

More information about the netty-users mailing list