[hibernate-dev] Hibernate Search and massive indexing

I have synced with Sanne on his work on massive reindexing and here is the outcome of the discussion. 1. An exclusive batch mode is a mode where a node has exclusive access to the index and can optimize writings (not flushing, not committing at specific times etc). 2. The node able to activate the exclusive batch has to be the master in a cluster (ie not the slaves). 3. The master will have two modes, a transactional mode (as today, ie commit at tx boundaries - potentially asyned) and an exclusive batch mode called the adaptative mode. In this mode the BackendQueryProcessor can take some freedom in when and how it flushes changes to the Lucene index and when and how it commits. One approach would be to be transactional (ie one queue of changes = one commit) for low thresholds and batch exclusive for higher threshold (apply several queues of changes before flushing or committing this back end would somehow communicate with the master copy process to only copy changes at the right time. (I think it should work well already but needs to be verified). A slave / client could force a commit by sending a Commit LuceneWork if needed. 4. There should be a way to switch at runtime from the tx mode to the adaptative mode. When switching, the tx queue is forked, new elements are queued, old elements are processed. When the old queue is emptied, the adaptative mode kicks in. 5. On top of that, the massive indexer API, reads data from the database as fast as possible and push index works to the adaptative engine. This API will be mono server but multi thread for now. Sanne can describe that more in details. This api woudl have a start and waitTillDone() API that starts the adaptative engine and stops it. That's it for the first step. Second steps (no particular order) 6. Make the massive indexer API work in a cluster. Slaves would read the DB and push index works to the queue 7. find a way to apply analyzing before the actual IndexWriter usage That would allow to increase the index parallelism by allowing some pipelining. Or even better to analyze on the slaves and free cpu time for the master (would work nice with 6) Sanne please add anything I have missed, misinterpreted.