[jboss-user] [JBoss Messaging] - Re: Selector Help needed - Possible alternative suggestions

[anhminh_tran]

Hi Tim,

Thanks for looking at my question.

Yes, we have multiple consumers that might have the same selectors and or different selectors.

The idea of using the producer and consumer model for us is this.

There will be many requests that will be made concurrently.  These requests all go into a persisent queue for guaranteed delivery.   These requests are for a "job" to be worked on.  These jobs might be quick or long running jobs.  

What we want is to have a variable number of consumers.  Each consumer will sit on a different box, since each job is very CPU intensive.  Each consumer will look for certain types of jobs and process them one-at-a-time.

We want to have multiple consumers to have the *same* selectors because we want to scale side-ways;  just add more boxes to be able to process more messages (jobs).

We also want to have multiple consumers to have different selectors, since certain types of jobs are known to be much smaller than the other jobs.  So we want to have dedicated consumers to work on these types of jobs.  This is so that the long running jobs do not clog up all the consumers and queue up all the small jobs.  In our use case, it's ok to have the long running jobs take their time to do the work..  but it's mandatory to get all the small jobs through as soon as possible.

The other caveat is that certain consumers should be able to config to look for small jobs first, then if there are no small jobs in the queue, grab a big job.  When do, start looking for a small job again... and so on..

So, we have three types of consumers if you may.  One dedicated to handling small jobs.  One dedicated to handling big jobs.  One looks for small jobs, if non are there, look for big jobs to work on.

We actually do have requirements to have multiple consumers of big jobs to work on same and different types of of jobs..

We have all of these implemented and working with one problem that we didn't expect.  That is, the consumer that is applying a selector, seems to be holding hostage all the messages that fit it's selector even though he's only called receive() for one message.

What we need is for other consumers to be able to apply the same selector and be able to get the messages that the other consumer didn't call receive() on yet.  That way, we can have many consumers work on many of the same types of jobs.  We need this to scale the processing of these jobs.

My other thought is if there is a way to specify for the consumer applying the selector that he only wants 1 message to be returned that matches his criteria.  He will work on his job and then go ask for 1 more of the same.

Thanks,
Minh

View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4056923#4056923

Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4056923