Announcing LittleProxy 0.1 based on Netty

Sun Nov 1 18:11:49 EST 2009

Hi Stephen- Thanks very much for the detailed response. First, for
anyone following along, we're talking about this bug in LittleProxy:

http://dev.littleshoot.org:8081/browse/LP-1

I would move this over to a LittleProxy-only discussion, but this
really has to do with FireFox not being able to read
HttpResponseDecoder's chunked transfer encoding.

OK, Stephen, first off, very cool you've implemented a proxy as well.
I too switched back and forth a couple of times between decoding the
responses and simply relaying the responses, and I ultimately included
the decoder because many folks out there using proxies need to modify
the responses in some way. So this will make it more flexible moving
forward. The proxy should also response Connection: close headers,
although I'm just letting the remote servers close the connections for
now.

I too found the bug with Transfer-Encoding does not show up with the
straight relaying technique. I think the reason for this is that the
HttpResponseDecoder adds Transfer-Encoding: chunked and actually
transfers the body in chunked format whenever the response body is
over the specified byte count (the limit it wants to hold in memory).
In LittleProxy's case, I've set that limit quite high to limit the
impact of the bug: 81920 bytes.

So, for some reason FireFox doesn't understand HttpResponseDecoder's
chunked encoding. The odd part is I've manually decoded the chunked
encoding to try to find where it's off, and it looks perfect. I
haven't gotten in to trying variations of the encoding because I can't
imagine how else you would decode it -- there's really only one way to
do it as far as I can tell. I think what you're seeing is the browser
not liking the chunked encoding when it tries to read it, and it just
closes the connection. I have tried doing things like removing the
Content-Length header in these cases, but that didn't help.

So, I'm not entirely convinced the bug isn't in FireFox, but to some
extent it doesn't matter since the proxy clearly needs to support
whatever FireFox is expecting.

Oh, I've also added my test for this. It's at:

src/test/java/org/littleshoot/proxy/HttpProxyTest.java

If you just run mvn test, it'll run. It basically grabs that same test
file directly and also through the proxy. In the proxy case, it
decodes the chunked Transfer-Encoding and makes sure the resulting
file is the same as the file it gets directly. Somewhat unfortunately
(in terms of solving this =), the two files are identical.

Thanks again Stephen!

-Adam

On Sun, Nov 1, 2009 at 3:37 AM, Stephen Haberman
<stephen at exigencecorp.com> wrote:
>
>> I'd like to announce the release of LittleProxy 0.1, based on Netty.
>
> Spiffy.
>
> I've recently written a netty-based proxy as well and have been meaning
> to post it somewhere. I was interested to see how you went about
> tackling the same problem, so read through some of the code.
>
>> http://dev.littleshoot.org:8081/browse/LP-1
>
> I tried your test image in my own netty proxy, with an interesting
> result.
>
> Initially, it loaded fine. However, I'm not using HttpResponseDecoder in
> the relaying pipeline, and instead just putting onto the browser channel
> whatever raw bytes show up on the relay channel.
>
> I had a HttpResponseDecoder in the relay pipeline originally, but took
> it out because I'm not introspecting/modifying the returning data.
>
> So, if I put HttpResponseDecoder back in my relay pipeline, your test
> image no longer loads and I get this exception:
>
> 2 - Incoming exception java.io.IOException: Connection reset by peer
> java.io.IOException: Connection reset by peer
>        at sun.nio.ch.FileDispatcher.read0(Native Method)
>        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:251)
>        at sun.nio.ch.IOUtil.read(IOUtil.java:224)
>        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:254)
>        at org.jboss.netty.buffer.HeapChannelBuffer.setBytes(HeapChannelBuffer.java:156)
>        at org.jboss.netty.buffer.AbstractChannelBuffer.writeBytes(AbstractChannelBuffer.java:425)
>        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:305)
>        at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:275)
>        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:196)
>        at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
>        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>        at java.lang.Thread.run(Thread.java:636)
>
> Okay, here's what I'm seeing so far:
>
> 1) Without HttpResponseDecoder it works all the time
>
> 2) With HttpResponseDecoder it never works, /but/ I only get the above
>   stack trace /most/ of the time. Several times I'll see no errors in
>   my console, but firefox still doesn't load the image.
>
>   This seems to be the behavior you're seeing--I have no idea why
>   sometimes I see a stack trace and sometimes not.
>
> 3) With HttpResonseDecoder+HttpChunkAggregator it does work. Isn't
>   that odd...
>
> 4) When I do see the exception, it's not on the outgoing/relay channel,
>   it's on the incoming/browser channel. E.g. the message object is read
>   in perfectly fine from the relay channel, it's only when I drop that
>   message into the browser channel that the exception occurs.
>
> So, it looks like there is a bug in HttpResponseDecoder where its not
> making a copy of some data, e.g. it reads it lazily from the source
> relay channel. This image/host seems to be large/quick enough to be
> disconnecting before the data can get out to the browser. However, this
> only sometimes causes exceptions to be reported (even though the image
> never loads in the browser), and can also be avoided by using
> HttpChunkAggregator which probably copies the data off the wire in a way
> that HttpResponseDecoder does not.
>
> So it looks like the raw byte messages can be shared cross channels,
> the aggregated response messages can be shared cross channels, but
> the non-aggregated response messages cannot be shared cross channels--
> not sure whether this is a known property that we're violating or a
> bug.
>
> - Stephen
>
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users
>

-- 
Adam Fisk
http://www.littleshoot.org | http://adamfisk.wordpress.com |
http://twitter.com/adamfisk