[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-867?pag... ] Sanne Grinovero commented on HSEARCH-867: ----------------------------------------- Hi Adam, there's this old article about lazy field extraction, I'm not sure how far it can still be applied to recent versions of Hibernate Search; it would be great if you could try it out, and then let us know if we need to change something or you're welcome to fix the article (it's a wiki). http://community.jboss.org/wiki/HibernateSearchAndOfflineTextExtraction -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-867?pag... ] adam edited comment on HSEARCH-867 at 8/24/11 9:39 AM: ------------------------------------------------------- Sanne, as I read the documentation originally, it uses the stringValue() method on Fieldable to get the string, so it just pushes the toString down the line so-to-speak. I'll look into what Fieldable can do and suggest changes to the documentation if you'd like. BTW, the first one is to modify the documentation to either: # more clearly link the source # provide the complete object definition of the LazyField was (Author: abrin): Sanne, as I read the documentation originally, it uses the stringValue() method on Fieldable to get the string, so it just pushes the toString down the line so-to-speak. I'll look into what Fieldable can do and suggest changes to the documentation if you'd like. BTW, the first one is to modify the documentation to define LazyField more directly to explicitly implement Fieldable. thanks, adam -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-867?pag... ] adam commented on HSEARCH-867: ------------------------------ Sanne, thanks for your comments. I'm trying to optimize an issue locally where we will be pulling in multiple files (sometimes large). Hence, trying to avoid string concatenation due to the memory issue. What I've found is that the Fieldable class (according to the documentation) should happily work with a reader if it's given one instead of the String. Documenting what I did (as this get's indexed in google) # changing the stringValue to return null and implementing a reader works well # using a SequenceInputStream allows me to wrap the FileInputStreams into a single reader # changing the FieldBridge to process a reader and pass it to the LazyField -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-867?pag... ] Sanne Grinovero commented on HSEARCH-867: ----------------------------------------- even generally Lucene will use a lot of file handles, and webservers do too. I wonder if you already have raised the limits? Server/enterprise Linux distributions come preconfigured with a generous amount, but desktop/developer oriented Linux distributions usually have an insufficient amount. Even if you already have a generous kernel limit, you make a good point that this design does not allow you to control the number of open streams. I think you should not open the stream initially when creating the LazyField, but you should rather pass enough information (file path?) to the LaziField implementation to open the reader only when it's needed, and close it too. So you avoid opening a resource in one thread and closing it in another, which is generally a bad idea, and also there won't be more readers open than the amount of workers in the thread_pool. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira