HTTP method for POST query argument
Frederic Bregier
fredbregier at free.fr
Sat Sep 12 04:48:44 EDT 2009
Hi Trustin,
I realize while answering to another thread bout Http that the encoder could
rely nicely on the ChunkedWriter by implementing the ChunkedInput in the
encoder.
So I test it and rewrite a bit the encoder (mainly change the API to fit
ChunkedInput) and the client example, which is simpler.
http://n2.nabble.com/file/n3631026/http-decode6.zip http-decode6.zip
I hope this is better,
Cheers,
Frederic
Frederic Bregier wrote:
>
> Hi Trustin,
>
> I know that I wrote I will not send a new version, but as I made a lot
> of improvement and finally get the encoder to work, I decide to post
> this primary work.
>
> Here are the changes:
> - Change HttpBodyRequestXXX to HttpPostRequestXXX since in fact it does
> only encode or decode Post Body requests.
>
> - I made Attribute also compatible with a Memory or Disk or mixed mode
> as FileUpload.
> The reason is that in POST form a TEXTAREA could be huge (like a paste
> of a file) so useful to be on disk.
>
> - I keep the memory for both Attribute (obvious since generally the
> case) and for FileUpload.
> The reason is that in some case, even writing on files could be a
> problem for a web server (not enough disk).
> And as you wrote before, other implementations can occur (webdav,
> database, ...).
> For one of my project, my implementation will be a passthrough object
> (not stored but passed to the next network protocol by chunk).
> A cleanFiles method allows to clean temporary files (if any).
>
> - I finally made a first implementation of the HttpPostBodyEncoder.
> It follows globally the same idea from Decoder except it will change
> the request header accordingly to multipart or not and the final size of
> the body.
> It tries to be optimized (more is still possible) in term of memory by
> reusing HttpData object.
> It is possible to get back the list of HttpDatathat were set into one
> encoder for another one.
> It was not so easy since it is quite more complicated than decoder...
>
> The global idea is:
> - create an HttpRequest
> - create an HttpPostRequestEncoder from the request and a boolean for
> multipart or not, setting optionally the factory
> - fill the request with the header or cookie
> - fill the encoder with the Attribute or FileUpload
> - call encoder.finalizeRequest()
> - writing the request to the channel (also returned by the
> finalizeRequest in case)
> - testing if the request is a chunked one
> - if so
> - getting the nextChunk from encoder (chunk =
> encoder.encodeNextChunk();) and write it to the channel
> - if this is the last chunk, stops, else get the nextChunkfrom
> encoder again until end
> - once all done, the user can call encoder.cleanFiles() if the list of
> HttpData is no more useful (so the reason it is not automatic).
>
> - I made a client example that simulate the natural browser example that
> were in the previous example.
> So one can test the server example directly using his/her own browser
> or the client that simulates the post of the three forms (get, post not
> multipart, post with multipart).
> Of course reading the example will help to know how to use it.
>
> I hope it is still ok and correct from your point of view.
>
> The next step should be first reviewing the AggregateChannelBuffer to
> enhance the Component version if OK.
> Then if this work is OK too, I will enhance the doc in API to include
> some extra information in the header of the API for the encoder and the
> decoder (to not rely only on example).
> And finally, if all is OK, I could publish the work as a replacement of
> http current codec (I would renaming http2 to http and replace Aggregate
> by Component).
> So there are still some work to do. ;-)
>
> I try to join the zip here:
> http://n2.nabble.com/file/n3613671/http-decode5.zip http-decode5.zip
>
> Cheers,
> Frederic
>
> Frederic Bregier wrote:
>>
>> Hi Trustin,
>>
>> No problem... I'm still currently improving it (same idea, same API but
>> more efficient on memory aspect).
>>
>> Also I try to start a simple encoder, but it is no so simple since I need
>> to make the header too (due to multipart) and to take care of specific
>> cases (multiple files send from one form parameter which implies
>> multipart with mixed multipart).
>>
>> However, I will not publish it until I've come to something clear enough
>> and also until we make some general ideas on how to do this, perhaps even
>> on decoder, in order to not make this thread full of iterative
>> versions... ;-)
>> In this case, less version is better (at least, what I feel ;-)
>>
>> Cheers,
>> Frederic
>>
>>
>> Trustin Lee wrote:
>>>
>>> Hi Frederic,
>>>
>>> This sounds pretty neat although I didn't review it yet. I think I
>>> need to review the new buffer type you proposed first and then get back
>>> here.
>>>
>>> Thanks and cheers,
>>> Trustin
>>>
>>> On Sat, 29 Aug 2009 22:28:48 +0900 (KST)
>>> Frederic Bregier <fredbregier at free.fr> wrote:
>>>>
>>>> Hi,
>>>>
>>>> First, I thank you all for your advices!
>>>>
>>>> I try to follow your ideas. I hope I did well. I join the zip that
>>>> includes all files.
>>>> As I said, I create a fork of http codec into http2 while
>>>> implementing and testing, but of course, it should replaced once ok
>>>> the current codec. In the zip, I join too the AggregateChannelBuffer
>>>> since my implementation uses it to minimize memory usage with still
>>>> no copy.
>>>>
>>>> I retain the following ideas:
>>>>
>>>> - I rename HttpDataDecoder to HttpBodyRequestDecoder, which seems to
>>>> me more accurate.
>>>>
>>>> - Each time a chunk is offered, the decoder starts to decode what it
>>>> can already decode. The interest is it moves from the current
>>>> ChannelBuffer to HttpData objects as soon as possible, so, using
>>>> AggregateCB, minimize the memory usage compares to if we retain all
>>>> ChannelBuffer until all data will be decoded (doubles data, one in CB
>>>> and one in HttpData). I take your proposition of decoder.offer(chunk).
>>>> It throws ErrorDataDecoderException if an error occurs while
>>>> decoding (bad charset, bad IO, ...).
>>>>
>>>> Note that Attribute can be huge (TEXTAREA in a FORM has no real
>>>> limit and can be huge, for instance copying the content of a Text
>>>> file into it) but it will not be stored in a File since it is not a
>>>> FileUpload but a simple parameter. So I like the idea to try to
>>>> prevent OOME as much as possible. Perhaps more optimization can be
>>>> done, like implementing a virtual attribute that could be stored into
>>>> a temporary file (like DiskFileUpload but for Attribute) if the size
>>>> becomes too high. WDYT?
>>>>
>>>> - Of course it is compatible with both chunked and chunked body. In
>>>> not chunked body, from the constructor, all HttpData are decoded.
>>>>
>>>> - No finish() is necessary since it will find by itself that the last
>>>> chunk is the last one.
>>>>
>>>> - Two kinds of access to HttpData (either Attribute or FileUpload) co
>>>> exist since I feel like one is simpler (yours) but the other one
>>>> allows some specific optimizations if wanted by the user.
>>>>
>>>> 1) Access to HttpData once all chunks are received (throw
>>>> NotEnoughDataDecoderException if the last chunk is still not
>>>> received).
>>>> - List<HttpData> getBodyHttpDatas(); // all HttpData
>>>> - List<HttpData> getBodyHttpDatas(String name); // all HttpData
>>>> associated with name
>>>> - HttpData getBodyHttpData(String name); // first HttpData
>>>> associated with name
>>>>
>>>> 2) Access to HttpData along offer(chunk) calls:
>>>> - boolean hasNext() throws EndOfDataDecoderException
>>>> //which says if there is at least one available HttpData in order
>>>> (no back but could be implemented)
>>>> - HttpData next() throws EndOfDataDecoderException // gives next
>>>> available HttpData in order
>>>>
>>>> EndOfDataDecoderException is throwed if the decoder arrives at
>>>> the very end (last chunk was received and the iteration is over).
>>>> This is necessary to make the difference between no data are
>>>> available but could be after next call to offer(chunk), or no data
>>>> are available and they will never be others (last chunk is already
>>>> received).
>>>>
>>>> In the snoop example, I show both usages.
>>>>
>>>> - I remove all references to Cookie and Header since you're right they
>>>> already have their own decoder and its fine. However I add some
>>>> convenient methods to HttpMessage that I feel they were missing:
>>>> - Map<String, List<String>> getHeaders();
>>>> - Set<Cookie> getCookies();
>>>> - void addCookie(Cookie cookie, boolean isServer);
>>>> - void setCookies(Set<Cookie> cookies, boolean isServer);
>>>>
>>>> For Cookies, I admit this is not efficient since I create a
>>>> encoder or decoder at each call. I would prefer at least to store the
>>>> Set result of decoder. For the encoder, as once encode() method is
>>>> called, it is purged, except rewriting a bit the encoder, I saw no
>>>> efficient implementations. I'm not so happy with this, but I feel
>>>> like at least the getHeaders() and getCookies() should be kept.
>>>> setCookies() is so simple that it could be kept too.
>>>> But addCookie could perhaps be forgotten since it is really not
>>>> efficient (better to manage itself the encoder).
>>>> WDYT?
>>>>
>>>> - I keep this as a Decoder, not a handler, in order to enable mixed
>>>> usage by end users (implementing for instance something else than a
>>>> pure HTTP server).
>>>>
>>>> - I change the DefaultFileUpload to MemoryFileUpload name.
>>>>
>>>> - I did not changed yet the FileUpload name, but as I wrote
>>>> previously in this post, if Attribute could have also a Disk
>>>> Implementation (or other), perhaps I can try to make a common part
>>>> for Memory, Disk and Mixte implementation, and then extend them into
>>>> Attribute or FileUpload. WDYT?
>>>>
>>>> By the way, I believe Attribute and FileUpload still need to be
>>>> different types since they don't have the same properties (FileUpload
>>>> must have a Content-Type for instance, while Attribute does not have
>>>> any).
>>>>
>>>> - I agree with Jean-Francois and you that having
>>>> HttpServletRequest/Response could be really interesting. But I don't
>>>> feel right now that I've got enough knowledge to begin that work.
>>>> Perhaps later on, once I learn a bit more about under the wood of
>>>> servlet and take the time to read carefuly the Grizzly
>>>> implementation...
>>>>
>>>> - For the moment, I remove the encoder from the zip, since I have
>>>> different options and I don't know which one could be better.
>>>> The question is basically:
>>>> Should the encoder encode the full response or just passing the
>>>> necessary body part?
>>>> The problem is that in Multipart mode or not, the header contains a
>>>> different Content-Type (multipart/form-data and
>>>> application/x-www-form-urlencoded respectively).
>>>> So should the encoder take care also about the header part encoding
>>>> (calling other encoders itself)?
>>>> If so, then it should take care too about the other headers and
>>>> Cookies... WDYT?
>>>>
>>>> Well, I think it is a good start to stop here... ;-)
>>>>
>>>> Cheers,
>>>> Frederic
>>>>
>>>> http://n2.nabble.com/file/n3542051/http-decode4.zip http-decode4.zip
>>>>
>>>>
>>>> -----
>>>> Hardware/Software Architect
>>>
>>>
>>>
>>> --
>>> Trustin Lee, http://gleamynode.net/
>>> _______________________________________________
>>> netty-dev mailing list
>>> netty-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/netty-dev
>>>
>>>
>>
>>
>
>
-----
Hardware/Software Architect
--
View this message in context: http://n2.nabble.com/HTTP-method-for-POST-query-argument-tp3457351p3631026.html
Sent from the Netty Developer Group mailing list archive at Nabble.com.
More information about the netty-dev
mailing list