HTTP method for POST query argument

Frederic Bregier fredbregier at free.fr
Sat Sep 12 04:48:44 EDT 2009


Hi Trustin,

I realize while answering to another thread bout Http that the encoder could
rely nicely on the ChunkedWriter by implementing the ChunkedInput in the
encoder.

So I test it and rewrite a bit the encoder (mainly change the API to fit
ChunkedInput) and the client example, which is simpler.

http://n2.nabble.com/file/n3631026/http-decode6.zip http-decode6.zip 

I hope this is better,

Cheers,
Frederic


Frederic Bregier wrote:
> 
> Hi Trustin,
> 
> I know that I wrote I will not send a new version, but as I made a lot
> of improvement and finally get the encoder to work, I decide to post
> this primary work.
> 
> Here are the changes:
> - Change HttpBodyRequestXXX to HttpPostRequestXXX since in fact it does
> only encode or decode Post Body requests.
> 
> - I made Attribute also compatible with a Memory or Disk or mixed mode
> as FileUpload.
>   The reason is that in POST form a TEXTAREA could be huge (like a paste
> of a file) so useful to be on disk.
> 
> - I keep the memory for both Attribute (obvious since generally the
> case) and for FileUpload.
>   The reason is that in some case, even writing on files could be a
> problem for a web server (not enough disk).
>   And as you wrote before, other implementations can occur (webdav,
> database, ...).
>   For one of my project, my implementation will be a passthrough object
> (not stored but passed to the next network protocol by chunk).
>   A cleanFiles method allows to clean temporary files (if any).
>  
> - I finally made a first implementation of the HttpPostBodyEncoder.
>   It follows globally the same idea from Decoder except it will change
> the request header accordingly to multipart or not and the final size of
> the body.
>   It tries to be optimized (more is still possible) in term of memory by
> reusing HttpData object.
>   It is possible to get back the list of HttpDatathat were set into one
> encoder for another one.
>   It was not so easy since it is quite more complicated than decoder...
> 
>   The global idea is:
>   - create an HttpRequest
>   - create an HttpPostRequestEncoder from the request and a boolean for
> multipart or not, setting optionally the factory
>   - fill the request with the header or cookie
>   - fill the encoder with the Attribute or FileUpload
>   - call encoder.finalizeRequest()
>   - writing the request to the channel (also returned by the
> finalizeRequest in case)
>   - testing if the request is a chunked one
>   - if so
>     - getting the nextChunk from encoder (chunk =
> encoder.encodeNextChunk();) and write it to the channel
>     - if this is the last chunk, stops, else get the nextChunkfrom
> encoder again until end
>   - once all done, the user can call encoder.cleanFiles() if the list of
> HttpData is no more useful (so the reason it is not automatic).
>  
> - I made a client example that simulate the natural browser example that
> were in the previous example.
>   So one can test the server example directly using his/her own browser
> or the client that simulates the post of the three forms (get, post not
> multipart, post with multipart).
>   Of course reading the example will help to know how to use it.
>  
> I hope it is still ok and correct from your point of view.
> 
> The next step should be first reviewing the AggregateChannelBuffer to
> enhance the Component version if OK.
> Then if this work is OK too, I will enhance the doc in API to include
> some extra information in the header of the API for the encoder and the
> decoder (to not rely only on example).
> And finally, if all is OK, I could publish the work as a replacement of
> http current codec (I would renaming http2 to http and replace Aggregate
> by Component).
> So there are still some work to do. ;-)
> 
> I try to join the zip here: 
> http://n2.nabble.com/file/n3613671/http-decode5.zip http-decode5.zip 
> 
> Cheers,
> Frederic 
> 
> Frederic Bregier wrote:
>> 
>> Hi Trustin,
>> 
>> No problem... I'm still currently improving it (same idea, same API but
>> more efficient on memory aspect).
>> 
>> Also I try to start a simple encoder, but it is no so simple since I need
>> to make the header too (due to multipart) and to take care of specific
>> cases (multiple files send from one form parameter which implies
>> multipart with mixed multipart).
>> 
>> However, I will not publish it until I've come to something clear enough
>> and also until we make some general ideas on how to do this, perhaps even
>> on decoder, in order to not make this thread full of iterative
>> versions... ;-)
>> In this case, less version is better (at least, what I feel ;-)
>> 
>> Cheers,
>> Frederic
>> 
>> 
>> Trustin Lee wrote:
>>> 
>>> Hi Frederic,
>>> 
>>> This sounds pretty neat although I didn't review it yet.  I think I
>>> need to review the new buffer type you proposed first and then get back
>>> here.
>>> 
>>> Thanks and cheers,
>>> Trustin
>>> 
>>> On Sat, 29 Aug 2009 22:28:48 +0900 (KST)
>>> Frederic Bregier <fredbregier at free.fr> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> First, I thank you all for your advices!
>>>> 
>>>> I try to follow your ideas. I hope I did well. I join the zip that
>>>> includes all files.
>>>> As I said, I create a fork of http codec into http2 while
>>>> implementing and testing, but of course, it should replaced once ok
>>>> the current codec. In the zip, I join too the AggregateChannelBuffer
>>>> since my implementation uses it to minimize memory usage with still
>>>> no copy.
>>>> 
>>>> I retain the following ideas:
>>>> 
>>>> - I rename HttpDataDecoder to HttpBodyRequestDecoder, which seems to
>>>> me more accurate.
>>>> 
>>>> - Each time a chunk is offered, the decoder starts to decode what it
>>>> can already decode. The interest is it moves from the current
>>>> ChannelBuffer to HttpData objects as soon as possible, so, using
>>>> AggregateCB, minimize the memory usage compares to if we retain all
>>>> ChannelBuffer until all data will be decoded (doubles data, one in CB
>>>> and one in HttpData). I take your proposition of decoder.offer(chunk).
>>>>   It throws ErrorDataDecoderException if an error occurs while
>>>> decoding (bad charset, bad IO, ...).
>>>> 
>>>>   Note that Attribute can be huge (TEXTAREA in a FORM has no real
>>>> limit and can be huge, for instance copying the content of a Text
>>>> file into it) but it will not be stored in a File since it is not a
>>>> FileUpload but a simple parameter. So I like the idea to try to
>>>> prevent OOME as much as possible. Perhaps more optimization can be
>>>> done, like implementing a virtual attribute that could be stored into
>>>> a temporary file (like DiskFileUpload but for Attribute) if the size
>>>> becomes too high. WDYT?
>>>> 
>>>> - Of course it is compatible with both chunked and chunked body. In
>>>> not chunked body, from the constructor, all HttpData are decoded.
>>>> 
>>>> - No finish() is necessary since it will find by itself that the last
>>>> chunk is the last one.
>>>> 
>>>> - Two kinds of access to HttpData (either Attribute or FileUpload) co
>>>> exist since I feel like one is simpler (yours) but the other one
>>>> allows some specific optimizations if wanted by the user.
>>>> 
>>>> 1) Access to HttpData once all chunks are received (throw
>>>> NotEnoughDataDecoderException if the last chunk is still not
>>>> received).
>>>>   - List<HttpData> getBodyHttpDatas(); // all HttpData
>>>>   - List<HttpData> getBodyHttpDatas(String name); // all HttpData
>>>> associated with name
>>>>   - HttpData getBodyHttpData(String name); // first HttpData
>>>> associated with name
>>>> 
>>>> 2) Access to HttpData along offer(chunk) calls:
>>>>   - boolean hasNext() throws EndOfDataDecoderException 
>>>>     //which says if there is at least one available HttpData in order
>>>> (no back but could be implemented)
>>>>   - HttpData next() throws EndOfDataDecoderException // gives next
>>>> available HttpData in order
>>>> 
>>>>     EndOfDataDecoderException is throwed if the decoder arrives at
>>>> the very end (last chunk was received and the iteration is over).
>>>> This is necessary to make the difference between no data are
>>>> available but could be after next call to offer(chunk), or no data
>>>> are available and they will never be others (last chunk is already
>>>> received).
>>>> 
>>>> In the snoop example, I show both usages.
>>>> 
>>>> - I remove all references to Cookie and Header since you're right they
>>>> already have their own decoder and its fine. However I add some
>>>> convenient methods to HttpMessage that I feel they were missing:
>>>>    - Map<String, List<String>> getHeaders();
>>>>    - Set<Cookie> getCookies();
>>>>    - void addCookie(Cookie cookie, boolean isServer);
>>>>    - void setCookies(Set<Cookie> cookies, boolean isServer);
>>>> 
>>>>    For Cookies, I admit this is not efficient since I create a
>>>> encoder or decoder at each call. I would prefer at least to store the
>>>> Set result of decoder. For the encoder, as once encode() method is
>>>> called, it is purged, except rewriting a bit the encoder, I saw no
>>>> efficient implementations. I'm not so happy with this, but I feel
>>>> like at least the getHeaders() and getCookies() should be kept.
>>>>    setCookies() is so simple that it could be kept too.
>>>>    But addCookie could perhaps be forgotten since it is really not
>>>> efficient (better to manage itself the encoder).
>>>>    WDYT?
>>>> 
>>>> - I keep this as a Decoder, not a handler, in order to enable mixed
>>>> usage by end users (implementing for instance something else than a
>>>> pure HTTP server).
>>>> 
>>>> - I change the DefaultFileUpload to MemoryFileUpload name.
>>>> 
>>>> - I did not changed yet the FileUpload name, but as I wrote
>>>> previously in this post, if Attribute could have also a Disk
>>>> Implementation (or other), perhaps I can try to make a common part
>>>> for Memory, Disk and Mixte implementation, and then extend them into
>>>> Attribute or FileUpload. WDYT?
>>>>    
>>>>    By the way, I believe Attribute and FileUpload still need to be
>>>> different types since they don't have the same properties (FileUpload
>>>> must have a Content-Type for instance, while Attribute does not have
>>>> any).
>>>> 
>>>> - I agree with Jean-Francois and you that having
>>>> HttpServletRequest/Response could be really interesting. But I don't
>>>> feel right now that I've got enough knowledge to begin that work.
>>>> Perhaps later on, once I learn a bit more about under the wood of
>>>> servlet and take the time to read carefuly the Grizzly
>>>> implementation...
>>>> 
>>>> - For the moment, I remove the encoder from the zip, since I have
>>>> different options and I don't know which one could be better.
>>>> The question is basically:
>>>> Should the encoder encode the full response or just passing the
>>>> necessary body part?
>>>> The problem is that in Multipart mode or not, the header contains a
>>>> different Content-Type (multipart/form-data and
>>>> application/x-www-form-urlencoded respectively).
>>>> So should the encoder take care also about the header part encoding
>>>> (calling other encoders itself)?
>>>> If so, then it should take care too about the other headers and
>>>> Cookies... WDYT?
>>>> 
>>>> Well, I think it is a good start to stop here... ;-)
>>>> 
>>>> Cheers,
>>>> Frederic
>>>> 
>>>> http://n2.nabble.com/file/n3542051/http-decode4.zip http-decode4.zip 
>>>> 
>>>> 
>>>> -----
>>>> Hardware/Software Architect
>>> 
>>> 
>>> 
>>> -- 
>>> Trustin Lee, http://gleamynode.net/
>>> _______________________________________________
>>> netty-dev mailing list
>>> netty-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/netty-dev
>>> 
>>> 
>> 
>> 
> 
> 


-----
Hardware/Software Architect
-- 
View this message in context: http://n2.nabble.com/HTTP-method-for-POST-query-argument-tp3457351p3631026.html
Sent from the Netty Developer Group mailing list archive at Nabble.com.


More information about the netty-dev mailing list