Hello, because of HSEARCH-268( optimize indexes in parallel ) but also for other purposes, I am in need to define a new ThreadPool in Hibernate Search's Lucene backend. The final effect will actually be that all changes to indexes are going to be performed in parallel (on different indexes). I consider this a major improvement, and is currently easy to implement, iff we solve the following problems. The question is about how to size it properly, and how should the parallel workers interact, especially regarding commit failures and rollbacks: about the size ========= I've considered some options: 1) "steal" the configuration setting from BatchedQueueingProcessor, transforming that implementation in singlethreaded, and reusing the parameter internally to the Lucene backend only (JMS doesn't need it AFAIK). I'm afraid this could break custom made backends configuration parsing. 2)add a new parameter to the environment 3)use a size equal to the number of DirectoryProviders (this is the optimal value, could be the default and be overriden by a parameter). 4)change the contract of BackendQueueProcessorFactory: instead of returning one Runnable it returns a list of Runnables, so it's possible to use the existing Executor. This needs some consideration about how different Runnables have to "join the same TX"; The JMS implementation could return just one Runnable, so no worry about that. about transactions ============ As you know Search is not using a two phase commit between DB and Index, but Emmanuel has a very cool vision about that: we could add that later. The problem is: what to do if a Lucene index update fails (e.g. index A is corrupted), should we cancel the tasks going to make changes to the other indexes, B and C? That would be possible, but I don't think that you like that: after all the database changes are committed already, so I should actually make a "best effort" to update all indexes which are still working correctly. Another option would be to make the changes to all indexes, and then IndexWriter.commit() them all after they are all done. This is the opposite of the previous example, and also more complex to implement. I personally don't like this, but would like to hear more voices as it is an important matter. I think Search should work on a "best effort" criteria for next release: update all indexes it is able to. In a future one we could add an option to make it "two phase" optionally) by playing with the new Lucene commit() capabilities, but this would only make sense if you actually wanted to rollback the database changes in case of an index failure. sharing IndexWriter in batch mode ===================== this is not needed for HSEARCH-268( optimize indexes in parallel ) but is needed to get a major boost in indexing performance. Currently the IndexWriter lifecycle is coupled to the operations done in a transaction; (also Emmanuel reminded me we need to release the file lock ASAP as a supported configuration is to use two Search instances sharing the same FS-based index). We already have the concept of "batch operation" and "transactional operation"; the only difference is currently about which tuning settings are applied to the IndexWriter. My idea is to extend the semantics of "batch mode" to mean a state which is globally affecting the way IndexWriters are aquired and released: when in batch mode, the IndexWriter is not closed at the end of each work queue, and the locks are not used: the IndexWriter could be shared across different threads. This is not transactionally safe of course, but that's why this is called "batch mode" opposing to "transactional mode": nobody would expect transactional behaviour. There should be taken care to revert the status to "transaction mode" and close the IndexWriter at the end, but this API would make me reindex the database using the "parallel scrollableresults" in the most efficient way, and nicely integrated. This isn't as complicated to implement as it is to explain;-) Sanne

On Fri, 21 Nov 2008 00:58:53 +0100, Sanne Grinovero <sanne.grinovero(a)gmail.com&gt; wrote: > I think the best option would be to have a separate > Executor for each directory provider: otherwise it could happen > that a slowly reacting index could block correct operation from others, > as many queues could pileup targeting the same DP and exausting > the threads, which would all stay locked and collapse to a single > threaded model. > > This makes me think we should need to create one executor > per DP, each one using just one thread: additional benefit is > that no locking would be needed, we can remove all barriers > in the backend (unless batch mode enables concurrent usage of the > IndexWriter) Hmm, sounds a little counterintuitive to me. Is the whole idea behind the Executor framework not to have a single Executor which is backed by a pool of threads? Then again, I am not so familiar with this. Just a thought.

Removing barriers sounds good though :) --Hardy

On Thu, 20 Nov 2008 21:14:16 +0100, Sanne Grinovero <sanne.grinovero(a)gmail.com&gt; wrote: > because of HSEARCH-268( optimize indexes in parallel ) but also for > other purposes, I am in need to define a new ThreadPool in Hibernate > Search's Lucene backend. > The final effect will actually be that all changes to indexes are > going to be performed in parallel (on different indexes). So really, we should rename the Jira task to "apply index modifications in parallel", unless we are first tackling the optimize problem only.

> about the size > ========= > I've considered some options: > 1) "steal" the configuration setting from BatchedQueueingProcessor, > transforming that implementation in singlethreaded, > and reusing the parameter internally to the Lucene backend only (JMS > doesn't need it AFAIK). > I'm afraid this could break custom made backends configuration parsing. > 2)add a new parameter to the environment Will this change change the way the work execution is configured? For example, if you set hibernate.search.worker.execution=async, does this mean that first of all the work queue itself gets processed by hibernate.search.worker.thread_pool.size number of threads and then there will be another thread (from another pool? same pool?) per directory provider applying the actual changes? If so we need two settings I believe since the default size of these two different thread pools will be different, right?

> about transactions > ============ > As you know Search is not using a two phase commit between DB and > Index, but Emmanuel has a very cool vision about that: we could add > that later. > The problem is: what to do if a Lucene index update fails (e.g. index > A is corrupted), > should we cancel the tasks going to make changes to the other indexes, B > and C? > That would be possible, but I don't think that you like that: after > all the database changes are committed already, so I should actually > make a "best effort" to update all indexes which are still working > correctly. I think a best effort approach is the best for now. I assume we still throw an Exception if the index operation fails, right? In this case the user will have to reindex one way or the other anyway.

Will we be able to get this change in for the GA release?