[hibernate-dev] Re: [Search] making updates to the indexes concurrently

[Sanne Grinovero]

2008/11/21 Hardy Ferentschik <hibernate at ferentschik.de>:
> On Fri, 21 Nov 2008 00:58:53 +0100, Sanne Grinovero
> <sanne.grinovero at gmail.com> wrote:
>
>> I think the best option would be to have a separate
>> Executor for each directory provider: otherwise it could happen
>> that a slowly reacting index could block correct operation from others,
>> as many queues could pileup targeting the same DP and exausting
>> the threads, which would all stay locked and collapse to a single
>> threaded model.
>>
>> This makes me think we should need to create one executor
>> per DP, each one using just one thread: additional benefit is
>> that no locking would be needed, we can remove all barriers
>> in the backend (unless batch mode enables concurrent usage of the
>> IndexWriter)
>
> Hmm, sounds a little counterintuitive to me. Is the whole idea behind the
> Executor framework not
> to have a single Executor which is backed by a pool of threads? Then again,
> I am not
> so familiar with this. Just a thought.
>
think it this way: each Executor is (normally) backed by a concurrent queue
of jobs to do; but actually in this case you have this situation:

* jobs to be done on different indexes have nothing to do with each
other, this is actually what you want made parallel
(so you need as many threads as indexes).

* two jobs on the same index can't be run in parallel (they need the same lock)
(so you never want to assign to two threads tasks needing the same index, or one
thread would be blocked, and thus not helping)

so what you actually need is a queue for each index, and each one needs to be
performed by one only thread (as more concurrency is not possible on one index
in transaction mode).
You could create a ConcurrentBlockingQueue<Work> for each index, and then
rely on the current locking scheme and be very clever on the pool
size, but using
separate Executors you actually avoid another problem, you avoid this
situations:
index A can't be updated because all threads in pool are working on an index B,
and are all locked.
Also an Executor actually IS such a concurrent queue, you just make sure
each thread gets assigned a task it can actually perform.
Threads are not so expensive in Java, and anyway because of the
"one thread a time" requirement they couldn't even try helping each other;
> Removing barriers sounds good though :)
>
> --Hardy
>

-- Sanne