[Repost] Re: HTTP method for POST query argument
Frederic Bregier
fredbregier at free.fr
Wed Sep 9 16:15:31 EDT 2009
Hi Trustin,
I know that I wrote I will not send a new version, but as I made a lot
of improvement and finally get the encoder to work, I decide to post
this primary work.
Here are the changes:
- Change HttpBodyRequestXXX to HttpPostRequestXXX since in fact it does
only encode or decode Post Body requests.
- I made Attribute also compatible with a Memory or Disk or mixed mode
as FileUpload.
The reason is that in POST form a TEXTAREA could be huge (like a paste
of a file) so useful to be on disk.
- I keep the memory for both Attribute (obvious since generally the
case) and for FileUpload.
The reason is that in some case, even writing on files could be a
problem for a web server (not enough disk).
And as you wrote before, other implementations can occur (webdav,
database, ...).
For one of my project, my implementation will be a passthrough object
(not stored but passed to the next network protocol by chunk).
A cleanFiles method allows to clean temporary files (if any).
- I finally made a first implementation of the HttpPostBodyEncoder.
It follows globally the same idea from Decoder except it will change
the request header accordingly to multipart or not and the final size of
the body.
It tries to be optimized (more is still possible) in term of memory by
reusing HttpData object.
It is possible to get back the list of HttpDatathat were set into one
encoder for another one.
It was not so easy since it is quite more complicated than decoder...
The global idea is:
- create an HttpRequest
- create an HttpPostRequestEncoder from the request and a boolean for
multipart or not, setting optionally the factory
- fill the request with the header or cookie
- fill the encoder with the Attribute or FileUpload
- call encoder.finalizeRequest()
- writing the request to the channel (also returned by the
finalizeRequest in case)
- testing if the request is a chunked one
- if so
- getting the nextChunk from encoder (chunk =
encoder.encodeNextChunk();) and write it to the channel
- if this is the last chunk, stops, else get the nextChunkfrom
encoder again until end
- once all done, the user can call encoder.cleanFiles() if the list of
HttpData is no more useful (so the reason it is not automatic).
- I made a client example that simulate the natural browser example that
were in the previous example.
So one can test the server example directly using his/her own browser
or the client that simulates the post of the three forms (get, post not
multipart, post with multipart).
Of course reading the example will help to know how to use it.
I hope it is still ok and correct from your point of view.
The next step should be first reviewing the AggregateChannelBuffer to
enhance the Component version if OK.
Then if this work is OK too, I will enhance the doc in API to include
some extra information in the header of the API for the encoder and the
decoder (to not rely only on example).
And finally, if all is OK, I could publish the work as a replacement of
http current codec (I would renaming http2 to http and replace Aggregate
by Component).
So there are still some work to do. ;-)
I try to join the zip here:
http://n2.nabble.com/file/n3613671/http-decode5.zip http-decode5.zip
Cheers,
Frederic
Frederic Bregier wrote:
>
> Hi Trustin,
>
> No problem... I'm still currently improving it (same idea, same API but
> more efficient on memory aspect).
>
> Also I try to start a simple encoder, but it is no so simple since I need
> to make the header too (due to multipart) and to take care of specific
> cases (multiple files send from one form parameter which implies multipart
> with mixed multipart).
>
> However, I will not publish it until I've come to something clear enough
> and also until we make some general ideas on how to do this, perhaps even
> on decoder, in order to not make this thread full of iterative versions...
> ;-)
> In this case, less version is better (at least, what I feel ;-)
>
> Cheers,
> Frederic
>
>
> Trustin Lee wrote:
>>
>> Hi Frederic,
>>
>> This sounds pretty neat although I didn't review it yet. I think I
>> need to review the new buffer type you proposed first and then get back
>> here.
>>
>> Thanks and cheers,
>> Trustin
>>
>> On Sat, 29 Aug 2009 22:28:48 +0900 (KST)
>> Frederic Bregier <fredbregier at free.fr> wrote:
>>>
>>> Hi,
>>>
>>> First, I thank you all for your advices!
>>>
>>> I try to follow your ideas. I hope I did well. I join the zip that
>>> includes all files.
>>> As I said, I create a fork of http codec into http2 while
>>> implementing and testing, but of course, it should replaced once ok
>>> the current codec. In the zip, I join too the AggregateChannelBuffer
>>> since my implementation uses it to minimize memory usage with still
>>> no copy.
>>>
>>> I retain the following ideas:
>>>
>>> - I rename HttpDataDecoder to HttpBodyRequestDecoder, which seems to
>>> me more accurate.
>>>
>>> - Each time a chunk is offered, the decoder starts to decode what it
>>> can already decode. The interest is it moves from the current
>>> ChannelBuffer to HttpData objects as soon as possible, so, using
>>> AggregateCB, minimize the memory usage compares to if we retain all
>>> ChannelBuffer until all data will be decoded (doubles data, one in CB
>>> and one in HttpData). I take your proposition of decoder.offer(chunk).
>>> It throws ErrorDataDecoderException if an error occurs while
>>> decoding (bad charset, bad IO, ...).
>>>
>>> Note that Attribute can be huge (TEXTAREA in a FORM has no real
>>> limit and can be huge, for instance copying the content of a Text
>>> file into it) but it will not be stored in a File since it is not a
>>> FileUpload but a simple parameter. So I like the idea to try to
>>> prevent OOME as much as possible. Perhaps more optimization can be
>>> done, like implementing a virtual attribute that could be stored into
>>> a temporary file (like DiskFileUpload but for Attribute) if the size
>>> becomes too high. WDYT?
>>>
>>> - Of course it is compatible with both chunked and chunked body. In
>>> not chunked body, from the constructor, all HttpData are decoded.
>>>
>>> - No finish() is necessary since it will find by itself that the last
>>> chunk is the last one.
>>>
>>> - Two kinds of access to HttpData (either Attribute or FileUpload) co
>>> exist since I feel like one is simpler (yours) but the other one
>>> allows some specific optimizations if wanted by the user.
>>>
>>> 1) Access to HttpData once all chunks are received (throw
>>> NotEnoughDataDecoderException if the last chunk is still not
>>> received).
>>> - List<HttpData> getBodyHttpDatas(); // all HttpData
>>> - List<HttpData> getBodyHttpDatas(String name); // all HttpData
>>> associated with name
>>> - HttpData getBodyHttpData(String name); // first HttpData
>>> associated with name
>>>
>>> 2) Access to HttpData along offer(chunk) calls:
>>> - boolean hasNext() throws EndOfDataDecoderException
>>> //which says if there is at least one available HttpData in order
>>> (no back but could be implemented)
>>> - HttpData next() throws EndOfDataDecoderException // gives next
>>> available HttpData in order
>>>
>>> EndOfDataDecoderException is throwed if the decoder arrives at
>>> the very end (last chunk was received and the iteration is over).
>>> This is necessary to make the difference between no data are
>>> available but could be after next call to offer(chunk), or no data
>>> are available and they will never be others (last chunk is already
>>> received).
>>>
>>> In the snoop example, I show both usages.
>>>
>>> - I remove all references to Cookie and Header since you're right they
>>> already have their own decoder and its fine. However I add some
>>> convenient methods to HttpMessage that I feel they were missing:
>>> - Map<String, List<String>> getHeaders();
>>> - Set<Cookie> getCookies();
>>> - void addCookie(Cookie cookie, boolean isServer);
>>> - void setCookies(Set<Cookie> cookies, boolean isServer);
>>>
>>> For Cookies, I admit this is not efficient since I create a
>>> encoder or decoder at each call. I would prefer at least to store the
>>> Set result of decoder. For the encoder, as once encode() method is
>>> called, it is purged, except rewriting a bit the encoder, I saw no
>>> efficient implementations. I'm not so happy with this, but I feel
>>> like at least the getHeaders() and getCookies() should be kept.
>>> setCookies() is so simple that it could be kept too.
>>> But addCookie could perhaps be forgotten since it is really not
>>> efficient (better to manage itself the encoder).
>>> WDYT?
>>>
>>> - I keep this as a Decoder, not a handler, in order to enable mixed
>>> usage by end users (implementing for instance something else than a
>>> pure HTTP server).
>>>
>>> - I change the DefaultFileUpload to MemoryFileUpload name.
>>>
>>> - I did not changed yet the FileUpload name, but as I wrote
>>> previously in this post, if Attribute could have also a Disk
>>> Implementation (or other), perhaps I can try to make a common part
>>> for Memory, Disk and Mixte implementation, and then extend them into
>>> Attribute or FileUpload. WDYT?
>>>
>>> By the way, I believe Attribute and FileUpload still need to be
>>> different types since they don't have the same properties (FileUpload
>>> must have a Content-Type for instance, while Attribute does not have
>>> any).
>>>
>>> - I agree with Jean-Francois and you that having
>>> HttpServletRequest/Response could be really interesting. But I don't
>>> feel right now that I've got enough knowledge to begin that work.
>>> Perhaps later on, once I learn a bit more about under the wood of
>>> servlet and take the time to read carefuly the Grizzly
>>> implementation...
>>>
>>> - For the moment, I remove the encoder from the zip, since I have
>>> different options and I don't know which one could be better.
>>> The question is basically:
>>> Should the encoder encode the full response or just passing the
>>> necessary body part?
>>> The problem is that in Multipart mode or not, the header contains a
>>> different Content-Type (multipart/form-data and
>>> application/x-www-form-urlencoded respectively).
>>> So should the encoder take care also about the header part encoding
>>> (calling other encoders itself)?
>>> If so, then it should take care too about the other headers and
>>> Cookies... WDYT?
>>>
>>> Well, I think it is a good start to stop here... ;-)
>>>
>>> Cheers,
>>> Frederic
>>>
>>> http://n2.nabble.com/file/n3542051/http-decode4.zip http-decode4.zip
>>>
>>>
>>> -----
>>> Hardware/Software Architect
>>
>>
>>
>> --
>> Trustin Lee, http://gleamynode.net/
>> _______________________________________________
>> netty-dev mailing list
>> netty-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/netty-dev
>>
>>
>
>
-----
Hardware/Software Architect
--
View this message in context: http://n2.nabble.com/HTTP-method-for-POST-query-argument-tp3457351p3613671.html
Sent from the Netty Developer Group mailing list archive at Nabble.com.
More information about the netty-dev
mailing list