[Hibernate-JIRA] Created: (HSEARCH-601) Allow MassIndexer to recover gracefully when individual objects won't index

[Hibernate-JIRA] Created:...

Ben Dotte (JIRA)

Thursday, 30 September 2010 Thu, 30 Sep '10

2:34 p.m.

Allow MassIndexer to recover gracefully when individual objects won't index --------------------------------------------------------------------------- Key: HSEARCH-601 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601 Project: Hibernate Search Issue Type: Improvement Components: massindexer Affects Versions: 3.2.0.Final Environment: Hibernate 3.5.3, SQL Server 2005 Reporter: Ben Dotte Some of our sites take hours to index, so it is critical that initial indexing complete even if some of the individual entities fail to index properly. Right now, it appears that the MassIndexer dies if any individual entity throws an exception during indexing, and does not complete indexing for that type. I have a workaround for now by overriding EntityConsumerLuceneworkProducer with my own index() method that catches and logs exceptions from docBuilder.createAddWork(): private void index( Object entity, Session session ) throws InterruptedException { Serializable id = session.getIdentifier( entity ); Class clazz = Hibernate.getClass( entity ); DocumentBuilderIndexedEntity docBuilder = documentBuilders.get( clazz ); TwoWayFieldBridge idBridge = docBuilder.getIdBridge(); String idInString = idBridge.objectToString( id ); //depending on the complexity of the object graph going to be indexed it's possible //that we hit the database several times during work construction. try { AddLuceneWork addWork = docBuilder.createAddWork( clazz, entity, id, idInString, true ); backend.enqueueAsyncWork( addWork ); } catch (Exception e) { log.error("Error indexing " + clazz + " id " + id, e); } } This has the added benefit that the object type and id that errored is logged, where that can be tough to track down otherwise. That may not be the best solution in general since it could hide exceptions from Hibernate Search itself. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://opensource.atlassian.com/projects/hibernate/secure/Administrators.... - For more information on JIRA, see: http://www.atlassian.com/software/jira

Show replies by date

Sanne Grinovero (JIRA)

Thursday, 30 September Thu, 30 Sep

4 p.m.

New subject: [Hibernate-JIRA] Updated: (HSEARCH-601) Allow MassIndexer to recover gracefully when individual objects won't index

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601?pag... ] Sanne Grinovero updated HSEARCH-601: ------------------------------------ Fix Version/s: 3.3.0.Beta2 ok, something should be done but this requires some discussion. I don't like the idea that it could loop very long on many errors, say a NullPointerException because of a broken fieldbridge shouldn't log me ten millions of errors; we could add a maximum allowable errors threshold - not sure if this is brilliant as it implies another configuration option. The number of errors should be collected by the progress monitor; maybe we could think in delegating the exception to it so that the monitor implementation can decide to log or not, and take the decision if the indexing should be aborted or continued. What is the kind of errors that you think should cause Search to not abort indexing?

...

Allow MassIndexer to recover gracefully when individual objects won't index --------------------------------------------------------------------------- Key: HSEARCH-601 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601 Project: Hibernate Search Issue Type: Improvement Components: massindexer Affects Versions: 3.2.0.Final Environment: Hibernate 3.5.3, SQL Server 2005 Reporter: Ben Dotte Fix For: 3.3.0.Beta2 Some of our sites take hours to index, so it is critical that initial indexing complete even if some of the individual entities fail to index properly. Right now, it appears that the MassIndexer dies if any individual entity throws an exception during indexing, and does not complete indexing for that type. I have a workaround for now by overriding EntityConsumerLuceneworkProducer with my own index() method that catches and logs exceptions from docBuilder.createAddWork(): private void index( Object entity, Session session ) throws InterruptedException { Serializable id = session.getIdentifier( entity ); Class clazz = Hibernate.getClass( entity ); DocumentBuilderIndexedEntity docBuilder = documentBuilders.get( clazz ); TwoWayFieldBridge idBridge = docBuilder.getIdBridge(); String idInString = idBridge.objectToString( id ); //depending on the complexity of the object graph going to be indexed it's possible //that we hit the database several times during work construction. try { AddLuceneWork addWork = docBuilder.createAddWork( clazz, entity, id, idInString, true ); backend.enqueueAsyncWork( addWork ); } catch (Exception e) { log.error("Error indexing " + clazz + " id " + id, e); } } This has the added benefit that the object type and id that errored is logged, where that can be tough to track down otherwise. That may not be the best solution in general since it could hide exceptions from Hibernate Search itself.

-- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://opensource.atlassian.com/projects/hibernate/secure/Administrators.... - For more information on JIRA, see: http://www.atlassian.com/software/jira

Ben Dotte (JIRA)

4:30 p.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-601) Allow MassIndexer to recover gracefully when individual objects won't index

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601?pag... ] Ben Dotte commented on HSEARCH-601: ----------------------------------- I like the idea of the progress monitor tracking the errors. It is unfortunately tricky to pinpoint what errors might happen, that's why I took the unfortunate path of catching Exception in the workaround. I guess typically they involve invalid database state that may cause a field bridge to throw an NPE just for that object, while everything else indexes without a problem. Deciding how to handle the exception could be in the progress monitor or maybe some separate user-supplied exception handler. I do already have SimpleIndexingProgressMonitor overridden since the hibernate.search.worker.indexing.monitor configuration property documented in MassIndexerProgressMonitor does not actually work :)

...

Allow MassIndexer to recover gracefully when individual objects won't index --------------------------------------------------------------------------- Key: HSEARCH-601 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601 Project: Hibernate Search Issue Type: Improvement Components: massindexer Affects Versions: 3.2.0.Final Environment: Hibernate 3.5.3, SQL Server 2005 Reporter: Ben Dotte Fix For: 3.3.0.Beta2 Some of our sites take hours to index, so it is critical that initial indexing complete even if some of the individual entities fail to index properly. Right now, it appears that the MassIndexer dies if any individual entity throws an exception during indexing, and does not complete indexing for that type. I have a workaround for now by overriding EntityConsumerLuceneworkProducer with my own index() method that catches and logs exceptions from docBuilder.createAddWork(): private void index( Object entity, Session session ) throws InterruptedException { Serializable id = session.getIdentifier( entity ); Class clazz = Hibernate.getClass( entity ); DocumentBuilderIndexedEntity docBuilder = documentBuilders.get( clazz ); TwoWayFieldBridge idBridge = docBuilder.getIdBridge(); String idInString = idBridge.objectToString( id ); //depending on the complexity of the object graph going to be indexed it's possible //that we hit the database several times during work construction. try { AddLuceneWork addWork = docBuilder.createAddWork( clazz, entity, id, idInString, true ); backend.enqueueAsyncWork( addWork ); } catch (Exception e) { log.error("Error indexing " + clazz + " id " + id, e); } } This has the added benefit that the object type and id that errored is logged, where that can be tough to track down otherwise. That may not be the best solution in general since it could hide exceptions from Hibernate Search itself.

Emmanuel Bernard (JIRA)

Friday, 8 October Fri, 8 Oct

5:14 a.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-601) Allow MassIndexer to recover gracefully when individual objects won't index

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601?pag... ] Emmanuel Bernard commented on HSEARCH-601: ------------------------------------------ What's the status of this guys? Beta2 is for end of this week. Should we push that for later?

...

Allow MassIndexer to recover gracefully when individual objects won't index --------------------------------------------------------------------------- Key: HSEARCH-601 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601 Project: Hibernate Search Issue Type: Improvement Components: massindexer Affects Versions: 3.2.0.Final Environment: Hibernate 3.5.3, SQL Server 2005 Reporter: Ben Dotte Fix For: 3.3.0.Beta2 Some of our sites take hours to index, so it is critical that initial indexing complete even if some of the individual entities fail to index properly. Right now, it appears that the MassIndexer dies if any individual entity throws an exception during indexing, and does not complete indexing for that type. I have a workaround for now by overriding EntityConsumerLuceneworkProducer with my own index() method that catches and logs exceptions from docBuilder.createAddWork(): private void index( Object entity, Session session ) throws InterruptedException { Serializable id = session.getIdentifier( entity ); Class clazz = Hibernate.getClass( entity ); DocumentBuilderIndexedEntity docBuilder = documentBuilders.get( clazz ); TwoWayFieldBridge idBridge = docBuilder.getIdBridge(); String idInString = idBridge.objectToString( id ); //depending on the complexity of the object graph going to be indexed it's possible //that we hit the database several times during work construction. try { AddLuceneWork addWork = docBuilder.createAddWork( clazz, entity, id, idInString, true ); backend.enqueueAsyncWork( addWork ); } catch (Exception e) { log.error("Error indexing " + clazz + " id " + id, e); } } This has the added benefit that the object type and id that errored is logged, where that can be tough to track down otherwise. That may not be the best solution in general since it could hide exceptions from Hibernate Search itself.

Emmanuel Bernard (JIRA)

Wednesday, 13 October Wed, 13 Oct

8:52 a.m.

New subject: [Hibernate-JIRA] Assigned: (HSEARCH-601) Allow MassIndexer to recover gracefully when individual objects won't index

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601?pag... ] Emmanuel Bernard reassigned HSEARCH-601: ---------------------------------------- Assignee: Sanne Grinovero Assigning to SAnne so he can chose a proper version resolution.

...

Allow MassIndexer to recover gracefully when individual objects won't index --------------------------------------------------------------------------- Key: HSEARCH-601 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601 Project: Hibernate Search Issue Type: Improvement Components: massindexer Affects Versions: 3.2.0.Final Environment: Hibernate 3.5.3, SQL Server 2005 Reporter: Ben Dotte Assignee: Sanne Grinovero Fix For: 3.3.0.Beta2 Some of our sites take hours to index, so it is critical that initial indexing complete even if some of the individual entities fail to index properly. Right now, it appears that the MassIndexer dies if any individual entity throws an exception during indexing, and does not complete indexing for that type. I have a workaround for now by overriding EntityConsumerLuceneworkProducer with my own index() method that catches and logs exceptions from docBuilder.createAddWork(): private void index( Object entity, Session session ) throws InterruptedException { Serializable id = session.getIdentifier( entity ); Class clazz = Hibernate.getClass( entity ); DocumentBuilderIndexedEntity docBuilder = documentBuilders.get( clazz ); TwoWayFieldBridge idBridge = docBuilder.getIdBridge(); String idInString = idBridge.objectToString( id ); //depending on the complexity of the object graph going to be indexed it's possible //that we hit the database several times during work construction. try { AddLuceneWork addWork = docBuilder.createAddWork( clazz, entity, id, idInString, true ); backend.enqueueAsyncWork( addWork ); } catch (Exception e) { log.error("Error indexing " + clazz + " id " + id, e); } } This has the added benefit that the object type and id that errored is logged, where that can be tough to track down otherwise. That may not be the best solution in general since it could hide exceptions from Hibernate Search itself.

Sanne Grinovero (JIRA)

12:36 p.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-601) Allow MassIndexer to recover gracefully when individual objects won't index

...

Allow MassIndexer to recover gracefully when individual objects won't index --------------------------------------------------------------------------- Key: HSEARCH-601 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601 Project: Hibernate Search Issue Type: Improvement Components: massindexer Affects Versions: 3.2.0.Final Environment: Hibernate 3.5.3, SQL Server 2005 Reporter: Ben Dotte Assignee: Sanne Grinovero Fix For: 3.3.0.Beta2 Some of our sites take hours to index, so it is critical that initial indexing complete even if some of the individual entities fail to index properly. Right now, it appears that the MassIndexer dies if any individual entity throws an exception during indexing, and does not complete indexing for that type. I have a workaround for now by overriding EntityConsumerLuceneworkProducer with my own index() method that catches and logs exceptions from docBuilder.createAddWork(): private void index( Object entity, Session session ) throws InterruptedException { Serializable id = session.getIdentifier( entity ); Class clazz = Hibernate.getClass( entity ); DocumentBuilderIndexedEntity docBuilder = documentBuilders.get( clazz ); TwoWayFieldBridge idBridge = docBuilder.getIdBridge(); String idInString = idBridge.objectToString( id ); //depending on the complexity of the object graph going to be indexed it's possible //that we hit the database several times during work construction. try { AddLuceneWork addWork = docBuilder.createAddWork( clazz, entity, id, idInString, true ); backend.enqueueAsyncWork( addWork ); } catch (Exception e) { log.error("Error indexing " + clazz + " id " + id, e); } } This has the added benefit that the object type and id that errored is logged, where that can be tough to track down otherwise. That may not be the best solution in general since it could hide exceptions from Hibernate Search itself.

Sanne Grinovero (JIRA)

12:40 p.m.

New subject: [Hibernate-JIRA] Updated: (HSEARCH-601) Allow MassIndexer to recover gracefully when individual objects won't index

...

Allow MassIndexer to recover gracefully when individual objects won't index --------------------------------------------------------------------------- Key: HSEARCH-601 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601 Project: Hibernate Search Issue Type: Improvement Components: massindexer Affects Versions: 3.2.0.Final Environment: Hibernate 3.5.3, SQL Server 2005 Reporter: Ben Dotte Assignee: Sanne Grinovero Fix For: 3.3.0.Beta3 Some of our sites take hours to index, so it is critical that initial indexing complete even if some of the individual entities fail to index properly. Right now, it appears that the MassIndexer dies if any individual entity throws an exception during indexing, and does not complete indexing for that type. I have a workaround for now by overriding EntityConsumerLuceneworkProducer with my own index() method that catches and logs exceptions from docBuilder.createAddWork(): private void index( Object entity, Session session ) throws InterruptedException { Serializable id = session.getIdentifier( entity ); Class clazz = Hibernate.getClass( entity ); DocumentBuilderIndexedEntity docBuilder = documentBuilders.get( clazz ); TwoWayFieldBridge idBridge = docBuilder.getIdBridge(); String idInString = idBridge.objectToString( id ); //depending on the complexity of the object graph going to be indexed it's possible //that we hit the database several times during work construction. try { AddLuceneWork addWork = docBuilder.createAddWork( clazz, entity, id, idInString, true ); backend.enqueueAsyncWork( addWork ); } catch (Exception e) { log.error("Error indexing " + clazz + " id " + id, e); } } This has the added benefit that the object type and id that errored is logged, where that can be tough to track down otherwise. That may not be the best solution in general since it could hide exceptions from Hibernate Search itself.

Sanne Grinovero (JIRA)

Tuesday, 26 October Tue, 26 Oct

10:47 a.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-601) Allow MassIndexer to recover gracefully when individual objects won't index

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601?pag... ] Sanne Grinovero commented on HSEARCH-601: ----------------------------------------- thinking more about it, having a threshold of "acceptable errors" seems inconsistent, and complex: handling the errors makes sense, but pushing the responsibility to the progress monitor doesn't. So current behavior is to fail in case of errors, but being async this might not be noticed by user code. I'd propose to properly catch and log most RuntimeErrors errors, continuing the job and logging each error.

...

Allow MassIndexer to recover gracefully when individual objects won't index --------------------------------------------------------------------------- Key: HSEARCH-601 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601 Project: Hibernate Search Issue Type: Improvement Components: massindexer Affects Versions: 3.2.0.Final Environment: Hibernate 3.5.3, SQL Server 2005 Reporter: Ben Dotte Assignee: Sanne Grinovero Fix For: 3.3.0.Beta3 Some of our sites take hours to index, so it is critical that initial indexing complete even if some of the individual entities fail to index properly. Right now, it appears that the MassIndexer dies if any individual entity throws an exception during indexing, and does not complete indexing for that type. I have a workaround for now by overriding EntityConsumerLuceneworkProducer with my own index() method that catches and logs exceptions from docBuilder.createAddWork(): private void index( Object entity, Session session ) throws InterruptedException { Serializable id = session.getIdentifier( entity ); Class clazz = Hibernate.getClass( entity ); DocumentBuilderIndexedEntity docBuilder = documentBuilders.get( clazz ); TwoWayFieldBridge idBridge = docBuilder.getIdBridge(); String idInString = idBridge.objectToString( id ); //depending on the complexity of the object graph going to be indexed it's possible //that we hit the database several times during work construction. try { AddLuceneWork addWork = docBuilder.createAddWork( clazz, entity, id, idInString, true ); backend.enqueueAsyncWork( addWork ); } catch (Exception e) { log.error("Error indexing " + clazz + " id " + id, e); } } This has the added benefit that the object type and id that errored is logged, where that can be tough to track down otherwise. That may not be the best solution in general since it could hide exceptions from Hibernate Search itself.

Ben Dotte (JIRA)

11:33 a.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-601) Allow MassIndexer to recover gracefully when individual objects won't index

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601?pag... ] Ben Dotte commented on HSEARCH-601: ----------------------------------- That sounds fine for a general solution. It might be nice eventually to add a hook for client code to track these in some way since we display the number of errored objects on the UI. But even without that, allowing indexing to complete regardless of individual failures seems like the right behavior to me. Thanks Sanne.

...

Allow MassIndexer to recover gracefully when individual objects won't index --------------------------------------------------------------------------- Key: HSEARCH-601 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601 Project: Hibernate Search Issue Type: Improvement Components: massindexer Affects Versions: 3.2.0.Final Environment: Hibernate 3.5.3, SQL Server 2005 Reporter: Ben Dotte Assignee: Sanne Grinovero Fix For: 3.3.0.Beta3 Some of our sites take hours to index, so it is critical that initial indexing complete even if some of the individual entities fail to index properly. Right now, it appears that the MassIndexer dies if any individual entity throws an exception during indexing, and does not complete indexing for that type. I have a workaround for now by overriding EntityConsumerLuceneworkProducer with my own index() method that catches and logs exceptions from docBuilder.createAddWork(): private void index( Object entity, Session session ) throws InterruptedException { Serializable id = session.getIdentifier( entity ); Class clazz = Hibernate.getClass( entity ); DocumentBuilderIndexedEntity docBuilder = documentBuilders.get( clazz ); TwoWayFieldBridge idBridge = docBuilder.getIdBridge(); String idInString = idBridge.objectToString( id ); //depending on the complexity of the object graph going to be indexed it's possible //that we hit the database several times during work construction. try { AddLuceneWork addWork = docBuilder.createAddWork( clazz, entity, id, idInString, true ); backend.enqueueAsyncWork( addWork ); } catch (Exception e) { log.error("Error indexing " + clazz + " id " + id, e); } } This has the added benefit that the object type and id that errored is logged, where that can be tough to track down otherwise. That may not be the best solution in general since it could hide exceptions from Hibernate Search itself.

Sanne Grinovero (JIRA)

11:49 a.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-601) Allow MassIndexer to recover gracefully when individual objects won't index

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601?pag... ] Sanne Grinovero commented on HSEARCH-601: ----------------------------------------- so you're using a http://docs.jboss.org/hibernate/search/3.3/reference/en-US/html_single/#d... to show the errors on the UI? or would it work for you? We've been discussing this, and after a quick POC it seems the way to go. Errors would be handled by the error handler; Fail-fast will still be the default (handling the first error only and then abort), and a new option can let you "fail-never" and receive all errors in whatever error handler you might have configured.

...

Allow MassIndexer to recover gracefully when individual objects won't index --------------------------------------------------------------------------- Key: HSEARCH-601 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601 Project: Hibernate Search Issue Type: Improvement Components: massindexer Affects Versions: 3.2.0.Final Environment: Hibernate 3.5.3, SQL Server 2005 Reporter: Ben Dotte Assignee: Sanne Grinovero Fix For: 3.3.0.Beta3 Some of our sites take hours to index, so it is critical that initial indexing complete even if some of the individual entities fail to index properly. Right now, it appears that the MassIndexer dies if any individual entity throws an exception during indexing, and does not complete indexing for that type. I have a workaround for now by overriding EntityConsumerLuceneworkProducer with my own index() method that catches and logs exceptions from docBuilder.createAddWork(): private void index( Object entity, Session session ) throws InterruptedException { Serializable id = session.getIdentifier( entity ); Class clazz = Hibernate.getClass( entity ); DocumentBuilderIndexedEntity docBuilder = documentBuilders.get( clazz ); TwoWayFieldBridge idBridge = docBuilder.getIdBridge(); String idInString = idBridge.objectToString( id ); //depending on the complexity of the object graph going to be indexed it's possible //that we hit the database several times during work construction. try { AddLuceneWork addWork = docBuilder.createAddWork( clazz, entity, id, idInString, true ); backend.enqueueAsyncWork( addWork ); } catch (Exception e) { log.error("Error indexing " + clazz + " id " + id, e); } } This has the added benefit that the object type and id that errored is logged, where that can be tough to track down otherwise. That may not be the best solution in general since it could hide exceptions from Hibernate Search itself.

Ben Dotte (JIRA)

12:22 p.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-601) Allow MassIndexer to recover gracefully when individual objects won't index

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601?pag... ] Ben Dotte commented on HSEARCH-601: ----------------------------------- I didn't even know about that error handler. That sounds like a great way to go. Thanks again.

...

Allow MassIndexer to recover gracefully when individual objects won't index --------------------------------------------------------------------------- Key: HSEARCH-601 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601 Project: Hibernate Search Issue Type: Improvement Components: massindexer Affects Versions: 3.2.0.Final Environment: Hibernate 3.5.3, SQL Server 2005 Reporter: Ben Dotte Assignee: Sanne Grinovero Fix For: 3.3.0.Beta3 Some of our sites take hours to index, so it is critical that initial indexing complete even if some of the individual entities fail to index properly. Right now, it appears that the MassIndexer dies if any individual entity throws an exception during indexing, and does not complete indexing for that type. I have a workaround for now by overriding EntityConsumerLuceneworkProducer with my own index() method that catches and logs exceptions from docBuilder.createAddWork(): private void index( Object entity, Session session ) throws InterruptedException { Serializable id = session.getIdentifier( entity ); Class clazz = Hibernate.getClass( entity ); DocumentBuilderIndexedEntity docBuilder = documentBuilders.get( clazz ); TwoWayFieldBridge idBridge = docBuilder.getIdBridge(); String idInString = idBridge.objectToString( id ); //depending on the complexity of the object graph going to be indexed it's possible //that we hit the database several times during work construction. try { AddLuceneWork addWork = docBuilder.createAddWork( clazz, entity, id, idInString, true ); backend.enqueueAsyncWork( addWork ); } catch (Exception e) { log.error("Error indexing " + clazz + " id " + id, e); } } This has the added benefit that the object type and id that errored is logged, where that can be tough to track down otherwise. That may not be the best solution in general since it could hide exceptions from Hibernate Search itself.

Emmanuel Bernard (JIRA)

Friday, 29 October Fri, 29 Oct

11:10 a.m.

New subject: [Hibernate-JIRA] Updated: (HSEARCH-601) Allow MassIndexer to recover gracefully when individual objects won't index

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601?pag... ] Emmanuel Bernard updated HSEARCH-601: ------------------------------------- Fix Version/s: (was: 3.3.0.Beta3) 3.3.0.CR1

...

Allow MassIndexer to recover gracefully when individual objects won't index --------------------------------------------------------------------------- Key: HSEARCH-601 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601 Project: Hibernate Search Issue Type: Improvement Components: massindexer Affects Versions: 3.2.0.Final Environment: Hibernate 3.5.3, SQL Server 2005 Reporter: Ben Dotte Assignee: Sanne Grinovero Fix For: 3.3.0.CR1 Some of our sites take hours to index, so it is critical that initial indexing complete even if some of the individual entities fail to index properly. Right now, it appears that the MassIndexer dies if any individual entity throws an exception during indexing, and does not complete indexing for that type. I have a workaround for now by overriding EntityConsumerLuceneworkProducer with my own index() method that catches and logs exceptions from docBuilder.createAddWork(): private void index( Object entity, Session session ) throws InterruptedException { Serializable id = session.getIdentifier( entity ); Class clazz = Hibernate.getClass( entity ); DocumentBuilderIndexedEntity docBuilder = documentBuilders.get( clazz ); TwoWayFieldBridge idBridge = docBuilder.getIdBridge(); String idInString = idBridge.objectToString( id ); //depending on the complexity of the object graph going to be indexed it's possible //that we hit the database several times during work construction. try { AddLuceneWork addWork = docBuilder.createAddWork( clazz, entity, id, idInString, true ); backend.enqueueAsyncWork( addWork ); } catch (Exception e) { log.error("Error indexing " + clazz + " id " + id, e); } } This has the added benefit that the object type and id that errored is logged, where that can be tough to track down otherwise. That may not be the best solution in general since it could hide exceptions from Hibernate Search itself.

Emmanuel Bernard (JIRA)

Monday, 8 November Mon, 8 Nov

4:46 a.m.

New subject: [Hibernate-JIRA] Updated: (HSEARCH-601) Allow MassIndexer to recover gracefully when individual objects won't index

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601?pag... ] Emmanuel Bernard updated HSEARCH-601: ------------------------------------- Fix Version/s: (was: 3.3.0.CR1) 3.3.0

...

Allow MassIndexer to recover gracefully when individual objects won't index --------------------------------------------------------------------------- Key: HSEARCH-601 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601 Project: Hibernate Search Issue Type: Improvement Components: massindexer Affects Versions: 3.2.0.Final Environment: Hibernate 3.5.3, SQL Server 2005 Reporter: Ben Dotte Assignee: Sanne Grinovero Fix For: 3.3.0 Some of our sites take hours to index, so it is critical that initial indexing complete even if some of the individual entities fail to index properly. Right now, it appears that the MassIndexer dies if any individual entity throws an exception during indexing, and does not complete indexing for that type. I have a workaround for now by overriding EntityConsumerLuceneworkProducer with my own index() method that catches and logs exceptions from docBuilder.createAddWork(): private void index( Object entity, Session session ) throws InterruptedException { Serializable id = session.getIdentifier( entity ); Class clazz = Hibernate.getClass( entity ); DocumentBuilderIndexedEntity docBuilder = documentBuilders.get( clazz ); TwoWayFieldBridge idBridge = docBuilder.getIdBridge(); String idInString = idBridge.objectToString( id ); //depending on the complexity of the object graph going to be indexed it's possible //that we hit the database several times during work construction. try { AddLuceneWork addWork = docBuilder.createAddWork( clazz, entity, id, idInString, true ); backend.enqueueAsyncWork( addWork ); } catch (Exception e) { log.error("Error indexing " + clazz + " id " + id, e); } } This has the added benefit that the object type and id that errored is logged, where that can be tough to track down otherwise. That may not be the best solution in general since it could hide exceptions from Hibernate Search itself.

Emmanuel Bernard (JIRA)

Wednesday, 8 December Wed, 8 Dec

4:16 a.m.

New subject: [Hibernate-JIRA] Updated: (HSEARCH-601) Allow MassIndexer to recover gracefully when individual objects won't index

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601?pag... ] Emmanuel Bernard updated HSEARCH-601: ------------------------------------- Fix Version/s: (was: 3.3.0.CR2) 3.4.0

...

Allow MassIndexer to recover gracefully when individual objects won't index --------------------------------------------------------------------------- Key: HSEARCH-601 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601 Project: Hibernate Search Issue Type: Improvement Components: massindexer Affects Versions: 3.2.0.Final Environment: Hibernate 3.5.3, SQL Server 2005 Reporter: Ben Dotte Assignee: Sanne Grinovero Fix For: 3.4.0 Some of our sites take hours to index, so it is critical that initial indexing complete even if some of the individual entities fail to index properly. Right now, it appears that the MassIndexer dies if any individual entity throws an exception during indexing, and does not complete indexing for that type. I have a workaround for now by overriding EntityConsumerLuceneworkProducer with my own index() method that catches and logs exceptions from docBuilder.createAddWork(): private void index( Object entity, Session session ) throws InterruptedException { Serializable id = session.getIdentifier( entity ); Class clazz = Hibernate.getClass( entity ); DocumentBuilderIndexedEntity docBuilder = documentBuilders.get( clazz ); TwoWayFieldBridge idBridge = docBuilder.getIdBridge(); String idInString = idBridge.objectToString( id ); //depending on the complexity of the object graph going to be indexed it's possible //that we hit the database several times during work construction. try { AddLuceneWork addWork = docBuilder.createAddWork( clazz, entity, id, idInString, true ); backend.enqueueAsyncWork( addWork ); } catch (Exception e) { log.error("Error indexing " + clazz + " id " + id, e); } } This has the added benefit that the object type and id that errored is logged, where that can be tough to track down otherwise. That may not be the best solution in general since it could hide exceptions from Hibernate Search itself.

Sanne Grinovero (JIRA)

Wednesday, 9 February Wed, 9 Feb

7:35 a.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-601) Allow MassIndexer to recover gracefully when individual objects won't index

...

Allow MassIndexer to recover gracefully when individual objects won't index --------------------------------------------------------------------------- Key: HSEARCH-601 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601 Project: Hibernate Search Issue Type: Improvement Components: massindexer Affects Versions: 3.2.0.Final Environment: Hibernate 3.5.3, SQL Server 2005 Reporter: Ben Dotte Assignee: Sanne Grinovero Fix For: 3.4.0 Some of our sites take hours to index, so it is critical that initial indexing complete even if some of the individual entities fail to index properly. Right now, it appears that the MassIndexer dies if any individual entity throws an exception during indexing, and does not complete indexing for that type. I have a workaround for now by overriding EntityConsumerLuceneworkProducer with my own index() method that catches and logs exceptions from docBuilder.createAddWork(): private void index( Object entity, Session session ) throws InterruptedException { Serializable id = session.getIdentifier( entity ); Class clazz = Hibernate.getClass( entity ); DocumentBuilderIndexedEntity docBuilder = documentBuilders.get( clazz ); TwoWayFieldBridge idBridge = docBuilder.getIdBridge(); String idInString = idBridge.objectToString( id ); //depending on the complexity of the object graph going to be indexed it's possible //that we hit the database several times during work construction. try { AddLuceneWork addWork = docBuilder.createAddWork( clazz, entity, id, idInString, true ); backend.enqueueAsyncWork( addWork ); } catch (Exception e) { log.error("Error indexing " + clazz + " id " + id, e); } } This has the added benefit that the object type and id that errored is logged, where that can be tough to track down otherwise. That may not be the best solution in general since it could hide exceptions from Hibernate Search itself.

Emmanuel Bernard (JIRA)

Monday, 18 April Mon, 18 Apr

7:03 a.m.

New subject: [Hibernate-JIRA] Updated: (HSEARCH-601) Allow MassIndexer to recover gracefully when individual objects won't index

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601?pag... ] Emmanuel Bernard updated HSEARCH-601: ------------------------------------- Fix Version/s: (was: 3.4.0) 3.5

...

Allow MassIndexer to recover gracefully when individual objects won't index --------------------------------------------------------------------------- Key: HSEARCH-601 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601 Project: Hibernate Search Issue Type: Improvement Components: massindexer Affects Versions: 3.2.0.Final Environment: Hibernate 3.5.3, SQL Server 2005 Reporter: Ben Dotte Assignee: Sanne Grinovero Fix For: 3.5 Some of our sites take hours to index, so it is critical that initial indexing complete even if some of the individual entities fail to index properly. Right now, it appears that the MassIndexer dies if any individual entity throws an exception during indexing, and does not complete indexing for that type. I have a workaround for now by overriding EntityConsumerLuceneworkProducer with my own index() method that catches and logs exceptions from docBuilder.createAddWork(): private void index( Object entity, Session session ) throws InterruptedException { Serializable id = session.getIdentifier( entity ); Class clazz = Hibernate.getClass( entity ); DocumentBuilderIndexedEntity docBuilder = documentBuilders.get( clazz ); TwoWayFieldBridge idBridge = docBuilder.getIdBridge(); String idInString = idBridge.objectToString( id ); //depending on the complexity of the object graph going to be indexed it's possible //that we hit the database several times during work construction. try { AddLuceneWork addWork = docBuilder.createAddWork( clazz, entity, id, idInString, true ); backend.enqueueAsyncWork( addWork ); } catch (Exception e) { log.error("Error indexing " + clazz + " id " + id, e); } } This has the added benefit that the object type and id that errored is logged, where that can be tough to track down otherwise. That may not be the best solution in general since it could hide exceptions from Hibernate Search itself.

Emmanuel Bernard (JIRA)

Friday, 23 December Fri, 23 Dec

8:40 a.m.

New subject: [Hibernate-JIRA] Updated: (HSEARCH-601) Allow MassIndexer to recover gracefully when individual objects won't index

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601?pag... ] Emmanuel Bernard updated HSEARCH-601: ------------------------------------- Fix Version/s: (was: 4.1)

...

Allow MassIndexer to recover gracefully when individual objects won't index --------------------------------------------------------------------------- Key: HSEARCH-601 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-601 Project: Hibernate Search Issue Type: Improvement Components: massindexer Affects Versions: 3.2.0.Final Environment: Hibernate 3.5.3, SQL Server 2005 Reporter: Ben Dotte Assignee: Sanne Grinovero Some of our sites take hours to index, so it is critical that initial indexing complete even if some of the individual entities fail to index properly. Right now, it appears that the MassIndexer dies if any individual entity throws an exception during indexing, and does not complete indexing for that type. I have a workaround for now by overriding EntityConsumerLuceneworkProducer with my own index() method that catches and logs exceptions from docBuilder.createAddWork(): private void index( Object entity, Session session ) throws InterruptedException { Serializable id = session.getIdentifier( entity ); Class clazz = Hibernate.getClass( entity ); DocumentBuilderIndexedEntity docBuilder = documentBuilders.get( clazz ); TwoWayFieldBridge idBridge = docBuilder.getIdBridge(); String idInString = idBridge.objectToString( id ); //depending on the complexity of the object graph going to be indexed it's possible //that we hit the database several times during work construction. try { AddLuceneWork addWork = docBuilder.createAddWork( clazz, entity, id, idInString, true ); backend.enqueueAsyncWork( addWork ); } catch (Exception e) { log.error("Error indexing " + clazz + " id " + id, e); } } This has the added benefit that the object type and id that errored is logged, where that can be tough to track down otherwise. That may not be the best solution in general since it could hide exceptions from Hibernate Search itself.

-- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

4684

days inactive

5133

days old

hibernate-issues@lists.jboss.org

Manage subscription

16 comments

3 participants

tags (0)

participants (3)

Ben Dotte (JIRA)
Emmanuel Bernard (JIRA)
Sanne Grinovero (JIRA)

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

[Hibernate-JIRA] Created: (HSEARCH-601) Allow MassIndexer to recover gracefully when individual objects won't index