[hibernate-dev] Hibernate Search and massive indexing

[Emmanuel Bernard]

I have synced with Sanne on his work on massive reindexing and here is  
the outcome of the discussion.

1. An exclusive batch mode is a mode where a node has exclusive access  
to the index and can optimize writings (not flushing, not committing  
at specific times etc).

2. The node able to activate the exclusive batch has to be the master  
in a cluster (ie not the slaves).

3. The master will have two modes, a transactional mode (as today, ie  
commit at tx boundaries - potentially asyned) and an exclusive batch  
mode called the adaptative mode.
In this mode the BackendQueryProcessor can take some freedom in when  
and how it flushes changes to the Lucene index and when and how it  
commits.
One approach would be to be transactional (ie one queue of changes =  
one commit) for low thresholds and batch exclusive for higher  
threshold (apply several queues of changes before flushing or committing
this back end would somehow communicate with the master copy process  
to only copy changes at the right time. (I think it should work well  
already but needs to be verified).
A slave / client could force a commit by sending a Commit LuceneWork  
if needed.

4. There should be a way to switch at runtime from the tx mode to the  
adaptative mode. When switching, the tx queue is forked, new elements  
are queued, old elements are processed.
When the old queue is emptied, the adaptative mode kicks in.

5. On top of that, the massive indexer API, reads data from the  
database as fast as possible and push index works to the adaptative  
engine. This API will be mono server but multi thread for now. Sanne  
can describe that more in details.
This api woudl have a start and waitTillDone() API that starts the  
adaptative engine and stops it.

That's it for the first step.

Second steps (no particular order)
6. Make the massive indexer API work in a cluster.
Slaves would read the DB and push index works to the queue

7. find a way to apply analyzing before the actual IndexWriter usage
That would allow to increase the index parallelism by allowing some  
pipelining.
Or even better to analyze on the slaves and free cpu time for the  
master (would work nice with 6)

Sanne please add anything I have missed, misinterpreted.