Re: [hibernate-dev] Hibernate Search and massive indexing

I'd like to stress that the second mode (adaptative) is not contrained into the BackendQueueProcessorFactory interface as it is not created by the usual BatchedQueueingProcessor but by a new kind of Worker. So we can apply some more pipelining optimizations.

About 5) the API is not going to be threadsafe/multithread but will use multiple threads. It's not clear to me at the moment when the new "Commit" LuceneWork should be used.

Sanne 2009/5/7 Emmanuel Bernard <emmanuel(a)hibernate.org&gt;: > I have synced with Sanne on his work on massive reindexing and here > is the > outcome of the discussion. > > 1. An exclusive batch mode is a mode where a node has exclusive > access to > the index and can optimize writings (not flushing, not committing at > specific times etc). > > 2. The node able to activate the exclusive batch has to be the > master in a > cluster (ie not the slaves). > > 3. The master will have two modes, a transactional mode (as today, > ie commit > at tx boundaries - potentially asyned) and an exclusive batch mode > called > the adaptative mode. > In this mode the BackendQueryProcessor can take some freedom in > when and how > it flushes changes to the Lucene index and when and how it commits. > One approach would be to be transactional (ie one queue of changes > = one > commit) for low thresholds and batch exclusive for higher threshold > (apply > several queues of changes before flushing or committing > this back end would somehow communicate with the master copy > process to only > copy changes at the right time. (I think it should work well > already but > needs to be verified). > A slave / client could force a commit by sending a Commit > LuceneWork if > needed. > > 4. There should be a way to switch at runtime from the tx mode to the > adaptative mode. When switching, the tx queue is forked, new > elements are > queued, old elements are processed. > When the old queue is emptied, the adaptative mode kicks in. > > 5. On top of that, the massive indexer API, reads data from the > database as > fast as possible and push index works to the adaptative engine. > This API > will be mono server but multi thread for now. Sanne can describe > that more > in details. > This api woudl have a start and waitTillDone() API that starts the > adaptative engine and stops it. > > That's it for the first step. > > Second steps (no particular order) > 6. Make the massive indexer API work in a cluster. > Slaves would read the DB and push index works to the queue > > 7. find a way to apply analyzing before the actual IndexWriter usage > That would allow to increase the index parallelism by allowing some > pipelining. > Or even better to analyze on the slaves and free cpu time for the > master > (would work nice with 6) > > Sanne please add anything I have missed, misinterpreted. > _______________________________________________ > hibernate-dev mailing list > hibernate-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hibernate-dev >