Re: [hibernate-dev] Design: HSEARCH-1032 MassIndexer with a live update mechanism

Current priorities on Search are: - Infinispan IndexManager -> me - Metadata API -> Hardy - Multitenancy (aka dynamic Sharding) -> me + Emmanuel + Dimitrios Those are all important as they represent hard requirements for other projects, but I'd also like to consider at least the basic design for how the MassIndexer could operate in "update mode": a highly requested mode in which it re-synchronizes the index with the database but without wiping out the index, which creates a window in time of the application in which results are not complete. # Reminder on current design: 1- deletes the current index 2- scrolls on all entities and uses ADD index operations to add them all again There are two basic approaches on the table (other ideas welcome) : - #A Use UPDATE index operations instead, skipping the initial delete - #B Rebuild the index in a secondary directory, then switch Let's explore them: #A Use UPDATE index operations instead, skipping the initial delete ## what Technically an UPDATE operation is - in Lucene terms - an atomic (delete+add); the benefit is that each query will either see the previous document or the updated one, there is no possibility that the doc is skipped as there is no possibility to flush the changes between the delete and the add operation. ## performance The reason the current design deletes all elements at the start of the process, is that this is a very efficient operation: it targets a single term (the class name field) or in some cases targets the whole index, so just needs to delete all segments files. When doing a delete operation on a per-document base, instead of a class, that very likely needs a deletion on multiple terms (which is not efficient at all as it needs to IO to seek across multiple disk positions), and of course the worse point is that it triggers a delete operation for each and every entity. To compare, a single ADD doesn't need any disk seek as we can pack multiple operations in one - until buffer is full - but any single delete requires N disk seeks (N is not directly the number of fields but is proportional to it). Based on this, and on experience with the #index() method benchmarking, I'm expecting the UPDATE strategy to be approximately a thousand times slower than the current MassIndexer implementation.. considering for some it takes a couple of hours, going to 2000 hours is maybe not an option :-) (that's 3 months) ## left over entries Another problem is that if we scroll on all entities from the database, we're failing to delete documents in the index for which there is no match anymore. So we would need a final phase in which we run the inverse iteration: for each element in the index, verify if there is a match in the database; sounds like an ugly lot of queries, even if we batch it in verification blocks. bottomline, looks messy. #B Rebuild the index in a secondary directory, then switch ## performance No big concerns, but we assume there is enough space for at least four times the size of the index (because we normally need twice to be able to compact one, and we have two to manage). ## design The good part is that we can reuse most of the existing MassIndexer; but transactional changes (those applied by the application during a reindexing) need to be redirected to both the indexes: the one being used until the rebuild is complete so that the queries stay consistent, and also enqueued into the one being built so that they don't get lost in case they apply to documents which have already been indexed. The queue handling is tricky, because in such case further additions actually need to be updates, unless we can keep them on hold in a buffer to be applied on the pristine index: could take quite some memory, depending on the amount of changes flying in during the massindexing. If the queue grows beyond reason we'll need to either apply backpressure on the transactions or offload to disk or change to an update strategy for the remaining massindexing process.. none of these are desirable but I guess people could tune to make this condition unlikely. ## SPI changes With this design we need to be able to: - dynamically instantiate a second Directory in a different path - switch to delegate writes to both directories / one directory - control from where Readers are opened - make sure closed Readers go back to the original pool where they come from as their reference source could have been changed - be able to switch (permanently) to a different active index - destroy old index I'm afraid each of these can affect our SPIs; likely at least IndexManager. I hope we can have all the logic in "behind the scenes" code which drives the same SPIs as of today but I'd need a POC to verify this. ## Directory index path If we switch from one Directory to another - thinking about the FSDirectory - we're either violating the path configuration options from the user or we need to move the new index into the configured position when done. If the above sounds a bit complex, I'm actually more concerned about implementing such an atomic move on the filesystem. I guess we could agree that if the user configured an index to be in - say - "/var/lucene/persons" we could store the indexes in "/var/lucene/persons/index-a" and "/var/lucene/persons/index-b", alternating in similar way to the FSMasterDirectoryProvider, but that takes away some control on index position and is not backwards compatible. Would this be acceptable? # Timeline This might need to be moved to 5.0 because of the various backwards compatibility concerns - ideally if some community user feels to participate we could share some early code in experimental branches and work together. Comments and better ideas welcome :) Sanne _______________________________________________ hibernate-dev mailing list hibernate-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev