Using Netty

[rajks]

Thanks for taking time to look into my questions.

In my case I need to talk to backend DB before I can allow upload or
download of the content.

I found that suspend/resume on read impacts the performance very heavily.
Here is what I was trying to do. Am I using Netty correctly or is there a
better way to do it. 

My process flow for a CREATE content logically is
    MessageReceived(...) 
         if start of request
              db.createmetadata
              save partial content
         else
              save partial content
    
As you can see if I use OrderedMemoryAwareExcecutor I am tying up the thread
pool to not be used for other requests which have already got the DB
operations cleared and continue with upload/download of other requests.

I introduced another set of a thread pool (call it DB Threadpool) after the
OMA and push the DB operations to that pool to process synchronously, then I
had to make sure the OMA does not get any content for this corresponding
request until the db operation is cleared. Hence I need to make the IO
worker SUSPEND READ on that channel. Once DB operation is cleared for this
context then I RESUME READ to allow upload.

Suspend/Resume read is manipulating on the interested ops on channel and
hence I see a performance issue.

Is there something as partial ( or call it lazy) suspend/resume of read that
does not change the interested ops on the channel but sets a boolean flag
and the IO worker skips that selected channel if partial suspend flag is
set. Will this partial mechanism help to boos the performance ??

Another issue the approach I am using is costing more threads i.e. DB thread
pool apart from IO Worker threads and OMA threads. Doesn't look good to
scale.

Appreciate if I can get any help to use Netty efficiently in my case.

Thanks
Raj


Trustin Lee-2 wrote:
> 
> On Wed, May 13, 2009 at 2:30 PM, Christian Migowski
> <chrismfwrd at gmail.com> wrote:
>> Hi Rajks,
>>
>> just some random pointers from another Netty user:
>>
>> On Wed, May 13, 2009 at 2:46 AM, rajks <rajks at hotmail.com> wrote:
>>>
>>> Should I implement the DB server also as Netty NIO HTTP based and use
>>> the
>>> Netty NIO HTTP client as means to communicate between front end servers
>>> and
>>> db servers. NOTE: each front end server will be communicating with many
>>> back-end DB servers. We would like to have PERSISTENT connections with
>>> some
>>> limit per host;
>>
>> If I don't have to, I wouldn't use http as communication protocol.
>> IIRC http had originally non-persistent connections only (I am not
>> familiar with Nettys http implementation), also it has probably more
>> features (which means more overhead) then you need for your servers
>> talking to each other.
>> IMHO designing a simple protocol (with a small header like
>> <msglength><actioncode><msgId> ) for your own use would be better
>> (less complex in case of problems, propably more efficient)
>>
>>> Which approach is scalable and have performance benefits with Netty or
>>> is
>>> there a better way to exploit Netty. Can I use the Netty client in teh
>>> front
>>> end server to distribute the requests to back-end DB server on a set of
>>> persistent connections.
>>
>> As far as I know, Netty has no proxy functionality available.
> 
> However, it is definitely possible to implement a HTTP proxy server
> with Netty, and I know a couple users who successfully did it.
> 
>>> Does Netty handle by default uploading of big files by N number of
>>> concurrent sessions or do we need to do something in the server side
>>> after
>>> each MessageReceived event to say suspend read so that IO does not read
>>> until the every chunk is written to the disk. after this resume read
>>> again.
>>> Does suspend and resume reads affect performance;
>>
>> Again as far as I know Netty has no write to disk/somewhere else
>> related things included. If you expect big files you'd better not
>> buffer them in memory until complete (you may run out of it).
> 
> I think Rajks asked a different question.  Suspend/resume operation
> affects performance somewhat, but it will depend on various factors -
> you need to test by yourself in this case.
> 
>>> Also is this taken care of automatically by using the
>>> OrderedMemoryAwareExcecutor as I see some topics in this forum
>>> recommending
>>> to use the OrderedMemoryAwareExcecutor to handle the memory issue and as
>>> well for high latency back-end operations.
> 
> Yes.  A read operation will be suspended and resumed automatically
> depending on the size of the task queue in the executor.
> 
> So, I believe Netty is suitable for your use case IMHO.
> 
> HTH,
> 
> — Trustin Lee, http://gleamynode.net/
> 
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users
> 
> 

-- 
View this message in context: http://n2.nabble.com/Using-Netty-tp2877729p2906318.html
Sent from the Netty User Group mailing list archive at Nabble.com.