Netty has problem to deal with the "Connection: close" message ... We cannot read the content.

"이희승 (Trustin Lee)" trustin at gmail.com
Thu Jun 4 02:09:25 EDT 2009


That's weird.  I've just managed to reproduce the problem. :)

Creating a HttpMessageDecoder with (1024, 1024, 1) always reproduced the
problem for me.  It might be just a similar issue with different cause,
so please let us know if you still have the problem.  The fix has been
checked in at revision 1322.

Thanks,
Trustin

On 04-Jun-2009 15:01, huican ping wrote:
> Hello Trustin,
> 
> Good news.
> I rebuilt my whole source code base, and it worked fine now. It is
> strange that why it is not working by only replacing the the netty lib
> file.
> 
> Thanks a lot.
> Huican
> 
> 2009/6/4 huican ping <pinghuican at gmail.com>:
>> no, the 242 is the response size which I got from tomcat server.
>> The maxChunkSize should be 8192 which is default.
>>
>> On Thu, Jun 4, 2009 at 12:19 AM, "이희승 (Trustin Lee)" <trustin at gmail.com> wrote:
>>> Did you set maxChunkSize to 242?
>>>
>>> On 04-Jun-2009 14:17, huican ping wrote:
>>>> Hello Trustin,
>>>>
>>>> Using the HttpClient example, I did see that " } END OF CHUNKED
>>>> CONTENT " for that "connection: close" response.
>>>> While in my other code base I am working on, I didn't get the last
>>>> (3rd) messageReceived() call. It is strange.
>>>>
>>>> Do you have any idea about it?
>>>>
>>>>    # 99th
>>>>    inside messageReceived()
>>>>    First chunk. Setting readingChunks to true
>>>>    inside messageReceived()
>>>>    another chunk chunksize = 242
>>>>    inside messageReceived()
>>>>    another chunk chunksize = 0
>>>>    is last chunk... yes
>>>>
>>>>    # 100th
>>>>    inside messageReceived()
>>>>    First chunk. Setting readingChunks to true
>>>>    inside messageReceived()
>>>>    inside chunk chunksize = 242
>>>>
>>>>
>>>> On Wed, Jun 3, 2009 at 11:36 PM, "이희승 (Trustin Lee)" <trustin at gmail.com> wrote:
>>>>> In revision 1320, I updated the HTTP example to reproduce the problem.
>>>>> With the HttpServer and HttpClient, I was able to receive the last
>>>>> HttpChunk successfully for a response without Content-Length.  Could you
>>>>> try to run the HttpClient and let me know if you see any differences.
>>>>> The following is an example output:
>>>>>
>>>>> STATUS: 200 OK
>>>>> VERSION: HTTP/1.1
>>>>>
>>>>> HEADER: Content-Type = text/plain; charset=UTF-8
>>>>> HEADER: Set-Cookie = another-cookie=bar;my-cookie=foo
>>>>> HEADER: Transfer-Encoding = chunked
>>>>>
>>>>> CHUNKED CONTENT {
>>>>> WELCOME TO THE WILD WILD WEB SERVER
>>>>> ===================================
>>>>> VERSION: HTTP/1.1
>>>>> HOSTNAME: localhost
>>>>> REQUEST_URI: http://localhost:8080/hello
>>>>>
>>>>> HEADER: Connection = close
>>>>> HEADER: Cookie = another-cookie=bar;my-cookie=foo
>>>>> HEADER: Host = localhost
>>>>>
>>>>> } END OF CHUNKED CONTENT  <-- isLast() returned true
>>>>>
>>>>> Thanks in advance,
>>>>> Trustin
>>>>>
>>>>> On 04-Jun-2009 13:16, huican ping wrote:
>>>>>> FYI, I synced to rev 1320, and "mvn clean install" generated the new jar file.
>>>>>>
>>>>>> 2009/6/3 huican ping <pinghuican at gmail.com>:
>>>>>>> Hello Trustin,
>>>>>>>
>>>>>>> Surprisingly, I got the same things as my very last time which is:
>>>>>>>  I saw the first 2 messageReceived() calls, but I didn't see the last
>>>>>>> messageReceived() call which will tells me chunk.isLast().
>>>>>>>
>>>>>>> Is there anything still missing.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Huican
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jun 3, 2009 at 10:11 PM, "이희승 (Trustin Lee)" <trustin at gmail.com> wrote:
>>>>>>>> The same fix should have been applied to
>>>>>>>> ReplayingDecoderBuffer.readable() but it was mistakenly forgotten.
>>>>>>>> Revision 1317 should now work fine.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Trustin
>>>>>>>>
>>>>>>>> On 04-Jun-2009 04:20, huican ping wrote:
>>>>>>>>> Hello Trustin,
>>>>>>>>>
>>>>>>>>> Debugged more, I noticed the 1316 is better than before BETA3, but the
>>>>>>>>> problem now is:
>>>>>>>>>
>>>>>>>>> For a normal request which content is smaller than MAXCHUNKSIZE, there
>>>>>>>>> are 3 messageReceived() calls.
>>>>>>>>>     The first one is just for Headers, no content; The second one is
>>>>>>>>> for the content; The last is for chunk.isLast() which satisfies the
>>>>>>>>> check "if (chunk.isLast())". This "if(chunk.isLast())" is same as the
>>>>>>>>> http sample code.
>>>>>>>>>
>>>>>>>>> Now at rev 1316, I saw the first 2 messageReceived() calls (sorry, I
>>>>>>>>> said no 2nd time, it was not right), but I didn't see the last
>>>>>>>>> messageReceived() call which will tells me chunk.isLast(). I need that
>>>>>>>>> check to do some essential steps at that condition, but now I didn't
>>>>>>>>> get that condition.
>>>>>>>>>
>>>>>>>>> You wont see the problem at http example since the sample code does
>>>>>>>>> nothing really important at that condition except set readingChunks =
>>>>>>>>> false.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> huican
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Jun 3, 2009 at 10:36 AM, Huican Ping <pinghuican at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Sorry Trustin,
>>>>>>>>>>
>>>>>>>>>> I synced to Rev 1316, and got the same result: the last response's content
>>>>>>>>>> chunk was not received, e.g, no messageReceived() called second time.
>>>>>>>>>>
>>>>>>>>>> For the 100th response like below (FYI, the body is NOT wrapped with chunked
>>>>>>>>>> byte size, I don't know whether it will cause problem since netty THINKs
>>>>>>>>>> this response is chunked encoding message but as you see the body is not
>>>>>>>>>> wrapped the content size as normally chunked message does):
>>>>>>>>>>        HTTP/1.1 200 OK
>>>>>>>>>>        Server: Apache-Coyote/1.1
>>>>>>>>>>        Content-Type: text/xml;charset=UTF-8
>>>>>>>>>>        Date: Tue, 02 Jun 2009 15:50:12 GMT
>>>>>>>>>>        Connection: close
>>>>>>>>>>
>>>>>>>>>>        <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope
>>>>>>>>>> xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><ns1:echo
>>>>>>>>>> xmlns:ns1="http://example1.org/example1"><Text>echo:
>>>>>>>>>> hello</Text></ns1:echo></soapenv:Body></soapenv:Envelope>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> In my httpResponseHandler, the code goes into messageReceived() first time,
>>>>>>>>>> and the check "respStatus.getCode() == 200 && resp.isChunked()" is true,
>>>>>>>>>> then I just mark "readingChunk = true". Later on, there is no 2nd
>>>>>>>>>> messageReceived() call.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The messageReceived method is same as the HttpResponseHandler example
>>>>>>>>>> logically, such as:
>>>>>>>>>> if ( !readingChunks )
>>>>>>>>>> {
>>>>>>>>>>        HttpResponse resp = (HttpResponse) event.getMessage();
>>>>>>>>>>
>>>>>>>>>>        ...
>>>>>>>>>>
>>>>>>>>>>        if (respStatus.getCode() == 200 && resp.isChunked() )
>>>>>>>>>>        {
>>>>>>>>>>                // First one is always 0 length, just set readingCHunks to true
>>>>>>>>>>                readingChunks = true;
>>>>>>>>>>                ....
>>>>>>>>>>        }
>>>>>>>>>>        else
>>>>>>>>>>        {
>>>>>>>>>>                // Here is the case of - content-length < 'maxChunkSize'
>>>>>>>>>>                // we need get the content and send resp.
>>>>>>>>>>                final ChannelBuffer buffer = resp.getContent();
>>>>>>>>>>                ....
>>>>>>>>>>        }
>>>>>>>>>> }
>>>>>>>>>> // for the left chunks
>>>>>>>>>> else
>>>>>>>>>> {
>>>>>>>>>>        HttpChunk chunk = (HttpChunk) event.getMessage();
>>>>>>>>>>        ...
>>>>>>>>>>
>>>>>>>>>>        if (chunk.isLast())
>>>>>>>>>>        {
>>>>>>>>>>                readingChunks = false;
>>>>>>>>>>                // Send the response to the workflow
>>>>>>>>>>                ....
>>>>>>>>>>        }
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Huican Ping wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks Trustin,
>>>>>>>>>>>
>>>>>>>>>>> Thank you for the quick fix. I will have a try then.
>>>>>>>>>>> I will let you know in cast if I still see the problem.
>>>>>>>>>>>
>>>>>>>>>>> Huican
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Trustin Lee wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Uh, I've just succeeded to reproduce the problem you described by
>>>>>>>>>>>> myself.  There was a regression in ReplayingDecoder when I fixed
>>>>>>>>>>>> NETTY-142.  I've just checked in the fix - revision 1316.  Here's the
>>>>>>>>>>>> filed JIRA issue:
>>>>>>>>>>>>
>>>>>>>>>>>>    * https://jira.jboss.org/jira/browse/NETTY-164
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks!
>>>>>>>>>>>> Trustin
>>>>>>>>>>>>
>>>>>>>>>>>> On 03-Jun-2009 17:21, "이희승 (Trustin Lee) wrote:
>>>>>>>>>>>>> 1) Netty HTTP decoder never closes the connection.  If the connection is
>>>>>>>>>>>>> closed, it's the remote peer (server in your case) or your handler
>>>>>>>>>>>>> implementation. Please let me know if you still think Netty HTTP decoder
>>>>>>>>>>>>> closes the connection.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2) If there's no way to know the length of the content, Netty transforms
>>>>>>>>>>>>> a non-chunked HTTP message into a chunked HTTP message.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In case of last response, the length of content is only determined when
>>>>>>>>>>>>> the connection is closed. Because it is not safe to buffer the whole
>>>>>>>>>>>>> content until the connection is closed, Netty generates HttpMessage with
>>>>>>>>>>>>> no content first and then HttpChunks.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Your handler should be able to handle this case because any HTTP server
>>>>>>>>>>>>> can send you a chunked HTTP message. Please make sure that you handle
>>>>>>>>>>>>> HttpChunk message properly. The HttpClient example already handles such
>>>>>>>>>>>>> a case, so you could refer to the example.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If you do not like this behavior but just want to limit the content
>>>>>>>>>>>>> length, you can insert HttpChunkAggregator right after
>>>>>>>>>>>>> Http(Request|Response)Decoder in the pipeline. Then you will never
>>>>>>>>>>>>> receive HttpChunk, and an exception will be raised if the content length
>>>>>>>>>>>>> is too large.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please let me know if HttpChunk is not received at all for the last
>>>>>>>>>>>>> response - it might be a bug then.
>>>>>>>>>>>>>
>>>>>>>>>>>>> HTH,
>>>>>>>>>>>>> Trustin
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 03-Jun-2009 04:02, Huican Ping wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> More information:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The messageReceived() message is same logic as the ones at http example
>>>>>>>>>>>>>> where it does
>>>>>>>>>>>>>> check "respStatus.getCode() == 200&& resp.isChunked() "
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I noticed for the headers at 100th response:
>>>>>>>>>>>>>> HTTP/1.1 200 OK
>>>>>>>>>>>>>> Server: Apache-Coyote/1.1
>>>>>>>>>>>>>> Content-Type: text/xml;charset=UTF-8
>>>>>>>>>>>>>> Date: Tue, 02 Jun 2009 15:50:12 GMT
>>>>>>>>>>>>>> Connection: close
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That resp.isChunked() returns true !! While stepping into the code (at
>>>>>>>>>>>>>> DefaultHttpMessage.java isChunked() method), and I dumpped out
>>>>>>>>>>>>>> headers, it
>>>>>>>>>>>>>> has Transfer-Encoding (which I don't know who added it mysteriously).
>>>>>>>>>>>>>> {Connection=[close], Content-Type=[text/xml;charset=UTF-8], Date=[Tue,
>>>>>>>>>>>>>> 02
>>>>>>>>>>>>>> Jun 2009 18:29:59 GMT], Server=[Apache-Coyote/1.1],
>>>>>>>>>>>>>> Transfer-Encoding=[chunked]}
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Huican Ping wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I am using the 3.1.0.Beta3.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That 100th message has no "content-length" header, but has
>>>>>>>>>>>>>>> "connection:
>>>>>>>>>>>>>>> close" inside. It is still a valid message.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Any suggestion or workaround?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> netty-users mailing list
>>>>>>>>>>>>> netty-users at lists.jboss.org
>>>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> netty-users mailing list
>>>>>>>>>>>> netty-users at lists.jboss.org
>>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
> 

-- 
— Trustin Lee, http://gleamynode.net/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 258 bytes
Desc: OpenPGP digital signature
Url : http://lists.jboss.org/pipermail/netty-users/attachments/20090604/bd47c2f1/attachment.bin 


More information about the netty-users mailing list