[hibernate-dev] inspecting performance issues in Hibernate Search

Hello, about HSEARCH-513 [Experiencing long delays when indexing entities saved in different transactions], it appears that the special use case is affected by a set of small issues which, when combined, create a huge performance mess. The difficulty in reproducing the issue was not related to Spring, but in the need to reproduce it with actual data triggering the situation. * when adding a new graph of objects, even if all are new the "collection" type of work prevales and all entities issue a delete+add operation on the index, while an add operation alone should be done. When the collections are commented, the generated operations are of "add" type only. I don't see this as critical, but this would be fixed too as a side effect of fixing HSEARCH-570. * The "depth" parameter of @IndexedEmbedded is working fine, but the counterpart @ContainedIn is not applying any limit in the other direction, effectively adding the largest possible graph in the workqueue. * While the @ContainedIn option "walks up" the tree, it adds other work which will recursively "walk down" and then up again, effectively exploding the number of objects unneccessarily. * Then when you have a huge amount of work, HSEARCH-570 [Optimize the o^2 loop in DocumentBuilderIndexedEntity.addWorkToQueue] kicks in as a coup de grace on the too large list, resulting in several minutes processing for the simple data in a unit test, hours are being reported on the issue when using real data. I consider this quite a critical issue, but the work I've been doing in the last few days is quite intrusive on the code of org.hibernate.search.backend.impl.BatchedQueueingProcessor, which is in the real core of mappings should I shelve my changes for after the release? I'd vote myself to try fixing it, provided Emmanuel or Hardy will have some time to review - I think we're well covered with unit tests to at least try and see how far I get. Sanne