[hibernate-dev] inspecting performance issues in Hibernate Search

[Sanne Grinovero]

Hello,
about HSEARCH-513 [Experiencing long delays when indexing entities
saved in different transactions],
it appears that the special use case is affected by a set of small
issues which, when combined, create a huge performance mess.
The difficulty in reproducing the issue was not related to Spring, but
in the need to reproduce it with actual data triggering the situation.

* when adding a new graph of objects, even if all are new the
"collection" type of work prevales and all entities issue a delete+add
operation on the index,
while an add operation alone should be done. When the collections are
commented, the generated operations are of "add" type only.
I don't see this as critical, but this would be fixed too as a side
effect of fixing HSEARCH-570.

* The "depth" parameter of @IndexedEmbedded is working fine, but the
counterpart @ContainedIn is not applying any limit in the other
direction, effectively adding the largest
possible graph in the workqueue.

* While the @ContainedIn option "walks up" the tree, it adds other
work which will recursively "walk down" and then up again, effectively
exploding the number
of objects unneccessarily.

* Then when you have a huge amount of work, HSEARCH-570 [Optimize the
o^2 loop in DocumentBuilderIndexedEntity.addWorkToQueue] kicks in as a
coup de grace
on the too large list, resulting in several minutes processing for the
simple data in a unit test, hours are being reported on the issue when
using real data.

I consider this quite a critical issue, but the work I've been doing
in the last few days is quite intrusive on the code of
org.hibernate.search.backend.impl.BatchedQueueingProcessor,
which is in the real core of mappings should I shelve my changes for
after the release?
I'd vote myself to try fixing it, provided Emmanuel or Hardy will have
some time to review - I think we're well covered
with unit tests to at least try and see how far I get.

Sanne