[hibernate-dev] MassIndexer and Hibernate Core dependency

[Sanne Grinovero]

Hi Emmanuel,

I guess some classes would result similar in an Infinispan based mass
indexer, bit this is coupled to which are the efficient strategies to
get a continuous flow of Documents.
It mostly depends on how you're going to store structured entities in
Infinispan? The whole graph needed by a DocumentBuilder is found on
the same key? (i.e. no lazy loading?)
Sure a non-tuned implementation would not be hard, but we might find
out that to make it fast it needs a larger set of changes maybe up to
the API.

Some classes would be reused with minimal changes, as whatever the
source is I'd still recommend to use the multiple-parallel-queues
idea, so at least these:

ProducerConsumerQueue,
MassIndexerProgressMonitor,
Executors
even the coordinator BatchCoordinator could work unchanged;

All classes in org.hibernate.search.backend.impl.batchlucene would
also work unchanged.

I can't really abstract-away Hibernate from the MassIndexerImpl and
the 2 Runnable types it spawns, the strategies it uses are tailored
for Hibernate: exploit caches, fight delay on lazyloading,
detach-reattach to session at thread switches, pool the Session(s). I
guess you could rename it to HibernateMassIndexerImpl? Another
implementation could easily combine the other classes as common
utilities but I'd bet the main strategy would be different.

you say "aside from the MassIndexer interface", note I'm exposing
cacheMode(org.hibernate.CacheMode cacheMode) on the interface.
I've seen consistent performance differences in enabling/disabling the
caches, in some models clearly faster enabling it, in others disabling
it, so I need some way to set this.
We could change that to a boolean, as only IGNORE/NOT_IGNORE the
caches are interesting options. Other implementations could then
ignore this setting if caching doesn't make sense to them.

After having made the API "hibernate free" we can change the factory
to return a different impl whatever the storage engine is. (we call it
"storage engine" ?)

Sanne


2009/10/25 Emmanuel Bernard <emmanuel at hibernate.org>:
> Hey Sanne,
> MassIndexerImpl is 100% dependent of Hibernate Core (you pass a
> SessionFactory).
> How realistic and doable is it to abstract that a bit. For example, would an
> Infinispan based mass indexer make sense? If yes, what's reusable in your
> impl (I mean aside from the MassIndexer interface).
>
> Emmanuel
>