[hibernate-dev] Search: backend refactoring

[Sanne Grinovero]

The short question:
	may I add some methods to the implementations of LuceneWork?
	I'm refactoring the backends and it would help, but there
	is a warning there in the javadoc about not changing it freely.

	Sanne

The same question, a bit more verbose:

Hi,
I've been puzzling about several optimization in Search I would like
to implement,
but am needing to do some refactoring in the
org.hibernate.search.backend package.
(mostly done actually, but needing your ideas)

Most changes affect "lucene" implementation, but the code would be
greatly simplified,
more readable and (performing better too IMHO) if I'm permitted to change the
current implementations of LuceneWork; however there's a big warning there
about a requirement to be backwards compatible with the serialized form.
(btw OptimizeLuceneWork is missing the "magic serialization number")

I would like to add them some methods, and a single field which could actually
be transient so I could attempt to maintain the compatibility.
Additionally I've been thinking that iff you like to keep the LuceneWork as
a very simple transport and prefer to not add methods, it would be nicer to
have just one class and have the AddLuceneWork/DeleteLuceneWork/... to
differentiate
by a field (using org.hibernate.search.backend.WorkType ?)
to mark the different type of work; so I could add
the methods I'm needing to the enum.
Also I could see some use of having an UpdateLuceneWork too, so that it is
the backend implementation's business to decide if he wants to split it in a
delete+insert or do something more clever:
the receive order of messages would be less critical and some clever
optimizations
could be applied by the backend by reordering received Work(s) or repackaging
several queues in one.

What I've done already:
a)early division in different queues, basing on affected DirectoryProviders

b)refactoring/simplification of Workspace, no longer needed to keep track of
state for different DP as there is only one in the context.

c)shorter Lock times: no threads ever need more than one Lock;
work is sorted by DP, each lock is released before acquiring the next one.
(deadlockFreeQueue is removed as not needed anymore)
before if we needed lock on DP's A,B,C the time of acquisition looked like:
Alock *********
Block    ******
CLock       ***
now it is more like
Alock ***
Block    ***
Clock       ***
And my goal is to make this possible, in separate threads when async:
Alock ***
Block ***
Clock ***
(not implemented yet: will need a new backend, but I'm preparing the common
stuff to make this possible)

d)The QueueProcessor can ask the Work about if they need an indexwriter,
indexreader or have any preference about one for when there is
possibility to make a choice (when we open both a reader and writer
anyway because of strict requirement of other Work in the same queue).

e)basing on d), DeleteLuceneWork is able to run either on reader or writer
(when it's possible to do so, depending on (the number of different
classes using the same DP) == 1); In this last case the work is able to
tell it "prefers" to be executed on an IndexWriter, but will be able
to do it's task with an IndexReader too (or the opposite?)

f)"batch mode" is currently set on all DP if only one Work is of type batch,
the division of Workspace per DP does not need this any more and batch
mode can be set independently.

Another goal I have with this design is the possibility to aggregate
different committed queues in one, having the possibility to
optimize away work (insert then delete => noop) considering the original
order, but also call the strategy optimization again
to reorder the newly created work for best efficiency.
The final effect would be to obtain the same behavior of
my custom batch indexer, but optimizing not only indexing from scratch
but any type of load.
I hope to not scare you, the resulting code is quite simple and I
think there are actually less LOC than the current trunk has;
I've not prepared any special case Test, I just run all existing ones.

kind regards,
Sanne