[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-135?pag... ] Emmanuel Bernard commented on HSEARCH-135: ------------------------------------------ Isn't it more or less what Lucene does when there is enough ram? If not then we can do that in a DirectoryProvider but we need to be sure to warn people about OutOfMemoryExceptions :) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://opensource.atlassian.com/projects/hibernate/secure/Administrators.... - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-135?pag... ] Hardy Ferentschik commented on HSEARCH-135: ------------------------------------------- Just want to get some feedback on the following idea. I would like to implement a custom DirectoryProvider where the Lucene Directory is backed up by two different Directory implementations. For index creation I want to use a FSDirectory and for use with a IndexReader I want to use a RAMDirectory. This way I get the best of both worlds. The FSDirectory in order to create a persistence index and the RAMDirectoy to speed up searches. As far as I can see this should work without major complications, however at the moment DirectoryProvider has only one single method to retrieve the Directory - getDirectory(). I would need a way to retrieve the directory depending on the usecase. For example you could have an additional boolean flag, getDirectory(boolean write). Or you could have two different methods - getReaderDirectory() and getWriterDirectory(). Any comments? -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://opensource.atlassian.com/projects/hibernate/secure/Administrators.... - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-135?pag... ] Sanne Grinovero commented on HSEARCH-135: ----------------------------------------- Did you try to compare RAMDirectory performance to FSDirectory performance? The filesystem is just slower the first time you have to read a segment/sector, after that the operating system knows some very good tricks about I/O optimization given enough free ram and you don't have to care (much) about memory requirements or have more work for your garbage collector. Also the OS is quite good at giving you transparently the correct version, so avoiding all the mess with flushing and reloading. My tests run on linux, but I think other oses should behave similarly. You can still get "the best of both worlds" with indexreader "warmup", having some query hit most segments in background before returning it to real searchers. Of course, this is only possible if you can somehow prepare the indexreaders beforehand. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://opensource.atlassian.com/projects/hibernate/secure/Administrators.... - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-135?pag... ] Hardy Ferentschik commented on HSEARCH-135: ------------------------------------------- [11:00am] hardy_: I was wondering whether you have some more experience now with RamDirectory vd FSDirectory in Lucene [11:00am] hardy_: https://forum.hibernate.org/viewtopic.php?f=9&t=1000523 [11:01am] hardy_: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-135 [11:01am] hardy_: I created HSEARCH-135 a while ago, because I also thought it would be a good idea to use a RAMDirectory, but initalise it with a FSDirectory [11:02am] sannegrinovero: ah yer I remember that [11:02am] sannegrinovero: like more than a year ago [11:02am] hardy_: I am not sure if this actually still makes snese [11:03am] sannegrinovero: my point is that RAMDirectory isn't faster than FSDirectory, when testing on non-trivial indexes [11:04am] hardy_: right. I think that was also Emmanuel's take on it [11:04am] sannegrinovero: but if you think the feature ads value, it's easy to implement, so no reason to not making it, besides dev time. [11:04am] hardy_: i see [11:04am] hardy_: I think the guy on the forum might have a point with the configuration [11:05am] sannegrinovero: the NIODirectory is also interesting, but is in high complexity [11:05am] hardy_: I wam wondering whether we should use the RAMDirectory constructor passing the file based index in case indexBase is specified [11:06am] sannegrinovero: "the index is allowed to take xyzMB in the VM and the rest you have to go to disk for." is really an Operating System problem [11:07am] hardy_: right [11:07am] sannegrinovero: that's what you get when using FSDirectory.. OS will use caches as it sees more fit [11:07am] sannegrinovero: that's why FSDirectory is as fast as memory, because it uses memory in clever ways.. on unix at least [11:09am] hardy_: ok, thanks [11:09am] sannegrinovero: so I see he's asking for a clever implementation combining FS+RAM approaches, I think the answer is FSDirectory, RAMDir being more like a toy. [11:10am] sannegrinovero: The feature you opened might be useful in case you want the index to be initialized at startup, but then loose changes and "reset state" at application startup. [11:11am] hardy_: makes sense -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://opensource.atlassian.com/projects/hibernate/secure/Administrators.... - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-135?pag... ] Sanne Grinovero updated HSEARCH-135: ------------------------------------ Priority: Minor (was: Trivial) Description: It would be nice to have the ability to use a Lucene RAM index which gets constucted from an existing Lucene file based indexed. For example in an JMS setup the master could create a file based Lucene index, share it out to the slaves which in turn use this file based index to populate a RAM index. This would give you the best of two worlds. Not sure how hard it would be to implement this in an unclustered environment. [*] read from FS at startup (optionally, should be configurabile) [*] write back to FS at shutdown (optionally, should be configurabile) [*] document the behaviour, especially warning about this case: some setups are doing local clustering: 2 instances sharing the directory. This obviously can't work, so they should avoid using this feature of writing back to FS. was: It would be nice to have the ability to use a Lucene RAM index which gets constucted from an existing Lucene file based indexed. For example in an JMS setup the master could create a file based Lucene index, share it out to the slaves which in turn use this file based index to populate a RAM index. This would give you the best of two worlds. Not sure how hard it would be to implement this in an unclustered environment. Fix Version/s: 3.3.0 -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://opensource.atlassian.com/projects/hibernate/secure/Administrators.... - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-135?pag... ] Emmanuel Bernard commented on HSEARCH-135: ------------------------------------------ Shouldn't these kind of operations be more on demand than every time? Copying a big directory can take a while. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://opensource.atlassian.com/projects/hibernate/secure/Administrators.... - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-135?pag... ] Emmanuel Bernard commented on HSEARCH-135: ------------------------------------------ The core problem is that there is pretty much no identified use case for it :) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://opensource.atlassian.com/projects/hibernate/secure/Administrators.... - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-135?pag... ] Emmanuel Bernard commented on HSEARCH-135: ------------------------------------------ It's Hardy's baby. I would not mind closing this issue. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://opensource.atlassian.com/projects/hibernate/secure/Administrators.... - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-135?pag... ] Hardy Ferentschik resolved HSEARCH-135. --------------------------------------- Resolution: Rejected I guess the whole issue is obsolete by now. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://opensource.atlassian.com/projects/hibernate/secure/Administrators.... - For more information on JIRA, see: http://www.atlassian.com/software/jira