[hibernate-dev] Hibernate Search 421 [Exceptions happening in backend are unnoticed]

[Emmanuel Bernard]

I forgot about this behavior, I thought you were raising an exception during the sync mode (even if run with parallel processes). That definitely looks like the right thing to do.

So an exception strategy should:
 - know if it's a synchronized call or not
 - know the list of operations that are failing
 - know the list of operations that failed as a consequence of the primary failure (eg, one insert fails and rolls back many)

That means that UncaughtExceptionHandler is not rich enough to provide this info and that we need a specific interface / contract. We very likely need also some:
 - initialize
 - start
 - stop
 methods to accomodate the JMS case for example.

hibernate.search.indexing_error_handling=f.q.c.n (with maybe some shortcuts like default - described by Sanne - and jms even though I m not convinced of the concrete usefuness of the jms approach).
and then any property under hibernate.search.indexing_error_handling will be passed to the initialize method (like we do for many other things).

Question:
 - should we have a global strategy or one per backend?

On 9 janv. 2010, at 14:34, Sanne Grinovero wrote:

> Hi Amin,
> I've been looking a bit in this, but didn't take any action as we
> didn't discuss any strategy, so glad you join in propose something.
> I didn't think of jms, it's ok we provide some way for the end user to
> override whatever we provide, but I think we should just provide
> the basic stuff: logging the error or propagate it back when possible.
> And then add some extensibility like plugin loading the usual way.
> 
> Originally (Hibernate Search 3.0.x) the changes made to different
> indexes were serialized, so when using the "sync" backend the same
> process would have applied changes to the index and, in case of
> exceptions, this would have been propagated up to user's code making
> it impossible to go unnoticed, or at least moving responsibility of
> handling it to the developer.
> When using the "async" mode for backend a different thread would have
> handled the indexwriter, so in case of an exception the error would
> have killed the separate thread, go unnoticed, and as we use a
> ThreadPool a new thread would have been spawned to replace the failed
> one.
> 
> Since Search 3.1.x both "sync" and "async" use a separate thread - to
> be able to apply changes to different indexes in parallel - so
> exceptions are going unnoticed by user code even in "sync" mode. Look
> into "run" method of
> org.hibernate.search.backend.impl.lucene.LuceneBackendQueueProcessor,
> it creates a org.hibernate.search.backend.impl.lucene.QueueProcessors
> where most of this logic resides.
> This impl will use either runAllAsync() or runAllWaiting() depending
> on backend configuration (async/sync). The Async version just
> schedules the different tasks, the sync one is going to do the same
> but will wait for all of them to have finished before returning
> control, making sure to implement the sync behaviour even while using
> several threads to perform it. I was inspired by
> java.util.concurrent.AbstractExecutorService.invokeAll(Collection<Callable<T>>
> tasks), as mentioned in code comments, which is ignoring
> ExecutionException, so this empty catch block should be used now to do
> something.
> 
> I'd suggest to provide this defaults:
> 
> * for sync backend:
>   by default: rethrow the exception (like old behaviour)
>   configurable alternative: log it
> 
> * for async backend:
>   log it (it's not useful to rethrow as nobody is listening on it)
> 
> To handle these cases only you would add the log statement, an rethrow
> it, fixing the QueueProcessors code.
> To give full control to the end user of what to do, I'd suggest to let
> him specify an implementationf of
> java.lang.Thread.UncaughtExceptionHandler
> and so we can set that on the thread.
> The backend is using the default JVM "Executors.newFixedThreadPool( 1
> );"  (initialized at
> org.hibernate.search.backend.impl.lucene.PerDPResources:51)
> but we could change that to use the Search Executor factory
> "org.hibernate.search.batchindexing.Executors" I later added for the
> batchbackend.
> The nice thingy for this Executor is that it customizes the thread
> names so you can easily spot Search's threads in monitoring/debugging
> tools;
> we could have a setUncaughtExceptionHandler( userImplementation ) in
> org.hibernate.search.batchindexing.Executors:85.
> If you change it there, the same handler would be used to manage
> errors in the batch indexer, so you solve both problems at once.
> Our "log the error" implementation would be the nice default for an
> exception handler, but still I'd like to make sure the user code will
> get the error propagated when
> using async mode.
> 
> Cheers,
> Sanne
> 
> 
> 2010/1/9 Amin Mohammed-Coleman <aminmc at gmail.com>:
>> Hi All
>> 
>> Emmanuel asked me to look at this issue (HSearch 421) where exceptions
>> happening in backend process which are going unnoticed.  I was wondering if
>> I could get some advice/thoughts on how to tackle the problem.    The issue
>> mentioned providing the user the option to decide how to handle exceptions
>> (for example queues, logs, etc), so I'm guessing there needs to be some
>> custom option that the user will need to set up, maybe something like this?
>> 
>> exception_handling_strategy=jms
>> exception_handling_strategy_jms_queue=
>> 
>> or if they wanted to log the exceptions:
>> 
>> exception_handling_strategy=log
>> 
>> or the user could create a custom class which implements a particular
>> interface to handle exceptions
>> 
>> exception_handling_strategy=custom
>> exception_handling_custom_class=ExceptionHandling
>> 
>> I could be completely wrong in the above approach and therefore would be
>> grateful for any input.
>> 
>> 
>> Cheers
>> Amin
>> _______________________________________________
>> hibernate-dev mailing list
>> hibernate-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>> 
> 
> _______________________________________________
> hibernate-dev mailing list
> hibernate-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hibernate-dev