[hibernate-dev] [HSEARCH] Implementing mass indexer with JSR 352 batch application

[Sanne Grinovero]

It sounds interesting, but bear with me I'm not familiar with it so
I'll throw out some doubts.

Will it work without a JEE container?
I guess some implementations might be embeddable, but still how simple
would that be for the user?

Note that we already allow rebuilding the index via JMX commands,
that's quite standard too and it's hooked up by other integrators such
as Infinispan or the WildFly admin console, and both expose a CLI too.
I guess other stacks will hook up their favourite approach easily too.

The idea sounds good but I'm not convinced on this to have a
compelling benefits/efforts ratio.. especially not if it turns out to
be slower than our current implementation (heck it's not as fast as
I'd want it yet but it's ok, and I know how to make it much better as
long as we control the details)

Performance of this component is important. On the same database I've
had MassIndexer POCs which would take 6 months to complete, a week to
complete, or 3 minutes. The current implementation compared to those
tests is the one which takes approximately 4 hours as I could never
make the last optimisations generic enough for the general purpose but
it would be just a matter of fixing open JIRAs for a couple of very
concrete points.
My point being that people often need to be able to reindex in a
couple of hours - no matter the size. 2 hours is of course very human
arbitrary but it seems to be the general acceptable threshold for such
an operation: beyond a developer's tool it's also a tool for recovery
from critical failures so it's absolutely unacceptable that a system
potentially needs weeks to recover from an issue.


On 24 February 2016 at 09:56, Vlad Mihalcea <mihalcea.vlad at gmail.com> wrote:
> +1
>
> Sounds like a good idea.
>
> On Wed, Feb 24, 2016 at 11:39 AM, Gunnar Morling <gunnar at hibernate.org>
> wrote:
>
>> Hi,
>>
>> I've been contemplating the idea of creating a JSR-352-style batch
>> application for re-indexing one or more entity types in Hibernate
>> Search.
>>
>> Functionally, it'd be the same as the current mass indexer, but using
>> JSR 352 would provide some nice benefits:
>>
>> * Operation through standard batch interfaces (e.g. CLI, web console
>> or whatever servers provide)
>> * Standardized monitoring and logging
>> * Standardized error handling, restartability after failures
>>
>> I thought this might be an interesting GSoC idea.
>>
>> It's very isolated and also should not be too complex to do. If the
>> student is quick, one further idea could be to provide some UI
>> functionality for controlling this (I'm not sure what's already
>> available in the WF web console).
>>
>> Any thoughts?
>>
>> Thanks,
>>
>> --Gunnar
>> _______________________________________________
>> hibernate-dev mailing list
>> hibernate-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>>
> _______________________________________________
> hibernate-dev mailing list
> hibernate-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hibernate-dev