[Hibernate-JIRA] Created: (HSEARCH-469) Filter caching using causes excessive memory use

[Hibernate-JIRA] Created:...

Dobes Vandermeer (JIRA)

Tuesday, 16 March 2010 Tue, 16 Mar '10

11:23 a.m.

Filter caching using causes excessive memory use ------------------------------------------------ Key: HSEARCH-469 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-469 Project: Hibernate Search Issue Type: Bug Components: query Affects Versions: 3.1.1.GA Environment: hibernate-core-3.3.2.GA Reporter: Dobes Vandermeer The CachingWrapperFilter uses the reader instance (CacheableMultiReader) as a key for the caching. However, the reader instance keeps pointers to byte arrays in its "normsCache" and in the "normsCache" of its sub-readers; each array has one byte for each document in the index and in some cases there will be multiple of these arrays associated with differet fields. For an index with millions of records this can result in an apparent "leak" of hundreds of megabytes of memory as those readers are not re-used and the MRU cache used by default will keep up to 128 hard references to the readers by default. The search system must either re-use or delete the normsCache, OR the cache key for these filters should be tied to something else that doesn't keep references to potentially huge data arrays. Otherwise the scalability of the search subsystem is significantly impacted when using filters, as you must have enough heap to accommodate up to 128 times as many copies of the norms arrays. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://opensource.atlassian.com/projects/hibernate/secure/Administrators.... - For more information on JIRA, see: http://www.atlassian.com/software/jira

Show replies by date

Dobes Vandermeer (JIRA)

Tuesday, 16 March Tue, 16 Mar

11:25 a.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-469) Filter caching using causes excessive memory use

...

-- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://opensource.atlassian.com/projects/hibernate/secure/Administrators.... - For more information on JIRA, see: http://www.atlassian.com/software/jira

Sanne Grinovero (JIRA)

12:11 p.m.

New subject: [Hibernate-JIRA] Updated: (HSEARCH-469) Filter caching using causes excessive memory use

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-469?pag... ] Sanne Grinovero updated HSEARCH-469: ------------------------------------ Description: The CachingWrapperFilter uses the reader instance (CacheableMultiReader) as a key for the caching. However, the reader instance keeps pointers to byte arrays in its "normsCache" and in the "normsCache" of its sub-readers; each array has one byte for each document in the index and in some cases there will be multiple of these arrays associated with differet fields. For an index with millions of records this can result in an apparent "leak" of hundreds of megabytes of memory as those readers are not re-used and the MRU cache used by default will keep up to 128 hard references to the readers by default. The search system must either re-use or delete the normsCache, OR the cache key for these filters should be tied to something else that doesn't keep references to potentially huge data arrays. Otherwise the scalability of the search subsystem is significantly impacted when using filters, as you must have enough heap to accommodate up to 128 times as many copies of the norms arrays. was: The CachingWrapperFilter uses the reader instance (CacheableMultiReader) as a key for the caching. However, the reader instance keeps pointers to byte arrays in its "normsCache" and in the "normsCache" of its sub-readers; each array has one byte for each document in the index and in some cases there will be multiple of these arrays associated with differet fields. For an index with millions of records this can result in an apparent "leak" of hundreds of megabytes of memory as those readers are not re-used and the MRU cache used by default will keep up to 128 hard references to the readers by default. The search system must either re-use or delete the normsCache, OR the cache key for these filters should be tied to something else that doesn't keep references to potentially huge data arrays. Otherwise the scalability of the search subsystem is significantly impacted when using filters, as you must have enough heap to accommodate up to 128 times as many copies of the norms arrays. Fix Version/s: 3.2.0.Beta3

...

Filter caching using causes excessive memory use ------------------------------------------------ Key: HSEARCH-469 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-469 Project: Hibernate Search Issue Type: Bug Components: query Affects Versions: 3.1.1.GA Environment: hibernate-core-3.3.2.GA Reporter: Dobes Vandermeer Fix For: 3.2.0.Beta3 The CachingWrapperFilter uses the reader instance (CacheableMultiReader) as a key for the caching. However, the reader instance keeps pointers to byte arrays in its "normsCache" and in the "normsCache" of its sub-readers; each array has one byte for each document in the index and in some cases there will be multiple of these arrays associated with differet fields. For an index with millions of records this can result in an apparent "leak" of hundreds of megabytes of memory as those readers are not re-used and the MRU cache used by default will keep up to 128 hard references to the readers by default. The search system must either re-use or delete the normsCache, OR the cache key for these filters should be tied to something else that doesn't keep references to potentially huge data arrays. Otherwise the scalability of the search subsystem is significantly impacted when using filters, as you must have enough heap to accommodate up to 128 times as many copies of the norms arrays.

Emmanuel Bernard (JIRA)

Friday, 2 April Fri, 2 Apr

12:40 p.m.

New subject: [Hibernate-JIRA] Assigned: (HSEARCH-469) Filter caching using causes excessive memory use

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-469?pag... ] Emmanuel Bernard reassigned HSEARCH-469: ---------------------------------------- Assignee: Sanne Grinovero

...

Filter caching using causes excessive memory use ------------------------------------------------ Key: HSEARCH-469 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-469 Project: Hibernate Search Issue Type: Bug Components: query Affects Versions: 3.1.1.GA Environment: hibernate-core-3.3.2.GA Reporter: Dobes Vandermeer Assignee: Sanne Grinovero Fix For: 3.2.0.Beta3 The CachingWrapperFilter uses the reader instance (CacheableMultiReader) as a key for the caching. However, the reader instance keeps pointers to byte arrays in its "normsCache" and in the "normsCache" of its sub-readers; each array has one byte for each document in the index and in some cases there will be multiple of these arrays associated with differet fields. For an index with millions of records this can result in an apparent "leak" of hundreds of megabytes of memory as those readers are not re-used and the MRU cache used by default will keep up to 128 hard references to the readers by default. The search system must either re-use or delete the normsCache, OR the cache key for these filters should be tied to something else that doesn't keep references to potentially huge data arrays. Otherwise the scalability of the search subsystem is significantly impacted when using filters, as you must have enough heap to accommodate up to 128 times as many copies of the norms arrays.

Emmanuel Bernard (JIRA)

Saturday, 3 April Sat, 3 Apr

6:26 a.m.

New subject: [Hibernate-JIRA] Updated: (HSEARCH-469) Filter caching using causes excessive memory use

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-469?pag... ] Emmanuel Bernard updated HSEARCH-469: ------------------------------------- Fix Version/s: (was: 3.2.0.Beta3) 3.2.0.Beta2

...

Filter caching using causes excessive memory use ------------------------------------------------ Key: HSEARCH-469 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-469 Project: Hibernate Search Issue Type: Bug Components: query Affects Versions: 3.1.1.GA Environment: hibernate-core-3.3.2.GA Reporter: Dobes Vandermeer Assignee: Sanne Grinovero Fix For: 3.2.0.Beta2 The CachingWrapperFilter uses the reader instance (CacheableMultiReader) as a key for the caching. However, the reader instance keeps pointers to byte arrays in its "normsCache" and in the "normsCache" of its sub-readers; each array has one byte for each document in the index and in some cases there will be multiple of these arrays associated with differet fields. For an index with millions of records this can result in an apparent "leak" of hundreds of megabytes of memory as those readers are not re-used and the MRU cache used by default will keep up to 128 hard references to the readers by default. The search system must either re-use or delete the normsCache, OR the cache key for these filters should be tied to something else that doesn't keep references to potentially huge data arrays. Otherwise the scalability of the search subsystem is significantly impacted when using filters, as you must have enough heap to accommodate up to 128 times as many copies of the norms arrays.

Sanne Grinovero (JIRA)

Sunday, 11 April Sun, 11 Apr

5:13 p.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-469) Filter caching using causes excessive memory use

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-469?pag... ] Sanne Grinovero commented on HSEARCH-469: ----------------------------------------- There is a bug which is that the {{hibernate.search.filter.cache_docidresults.size}} configuration property is being ignored. I'm considering several options: * change SoftLimitMRUCache to a {{ReferenceMap( ReferenceMap.SOFT, ReferenceMap.SOFT )}} * lower the default hard-referenced to a lower amount (there is a DEFAULT_SIZE defined in CachingWrapperFilter set to 5 instead of 128 but this value is not being used) * actively removing elements from the caches when the reader is closed I think this issue should be solved by either applying the configuration property to the SoftLimitMRUCache or converting it to a full-soft cache (removing the configuration property). We could schedule an improvement for a later version to actively remove elements relating to closed IndexReaders, to make best use of available memory. thoughts?

...

Filter caching using causes excessive memory use ------------------------------------------------ Key: HSEARCH-469 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-469 Project: Hibernate Search Issue Type: Bug Components: query Affects Versions: 3.1.1.GA Environment: hibernate-core-3.3.2.GA Reporter: Dobes Vandermeer Assignee: Sanne Grinovero Fix For: 3.2.0.CR1 The CachingWrapperFilter uses the reader instance (CacheableMultiReader) as a key for the caching. However, the reader instance keeps pointers to byte arrays in its "normsCache" and in the "normsCache" of its sub-readers; each array has one byte for each document in the index and in some cases there will be multiple of these arrays associated with differet fields. For an index with millions of records this can result in an apparent "leak" of hundreds of megabytes of memory as those readers are not re-used and the MRU cache used by default will keep up to 128 hard references to the readers by default. The search system must either re-use or delete the normsCache, OR the cache key for these filters should be tied to something else that doesn't keep references to potentially huge data arrays. Otherwise the scalability of the search subsystem is significantly impacted when using filters, as you must have enough heap to accommodate up to 128 times as many copies of the norms arrays.

Sanne Grinovero (JIRA)

5:33 p.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-469) Filter caching using causes excessive memory use

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-469?pag... ] Sanne Grinovero commented on HSEARCH-469: ----------------------------------------- sorry the {{cache_docidresults.size}} is not being ignored - was looking into wrong version. Dobes, would setting this property to a low value not be good enough to solve your problem? It's defaulting to 5, not 128. You could even set it to zero to have soft-only filter caches. Of course, actively removing unused readers would be better.

...

Filter caching using causes excessive memory use ------------------------------------------------ Key: HSEARCH-469 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-469 Project: Hibernate Search Issue Type: Bug Components: query Affects Versions: 3.1.1.GA Environment: hibernate-core-3.3.2.GA Reporter: Dobes Vandermeer Assignee: Sanne Grinovero Fix For: 3.2.0.CR1 The CachingWrapperFilter uses the reader instance (CacheableMultiReader) as a key for the caching. However, the reader instance keeps pointers to byte arrays in its "normsCache" and in the "normsCache" of its sub-readers; each array has one byte for each document in the index and in some cases there will be multiple of these arrays associated with differet fields. For an index with millions of records this can result in an apparent "leak" of hundreds of megabytes of memory as those readers are not re-used and the MRU cache used by default will keep up to 128 hard references to the readers by default. The search system must either re-use or delete the normsCache, OR the cache key for these filters should be tied to something else that doesn't keep references to potentially huge data arrays. Otherwise the scalability of the search subsystem is significantly impacted when using filters, as you must have enough heap to accommodate up to 128 times as many copies of the norms arrays.

Emmanuel Bernard (JIRA)

Monday, 12 April Mon, 12 Apr

5:19 a.m.

New subject: [Hibernate-JIRA] Updated: (HSEARCH-469) Filter caching using causes excessive memory use

...

Filter caching using causes excessive memory use ------------------------------------------------ Key: HSEARCH-469 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-469 Project: Hibernate Search Issue Type: Bug Components: query Affects Versions: 3.1.1.GA Environment: hibernate-core-3.3.2.GA Reporter: Dobes Vandermeer Assignee: Sanne Grinovero Fix For: 3.2.0 The CachingWrapperFilter uses the reader instance (CacheableMultiReader) as a key for the caching. However, the reader instance keeps pointers to byte arrays in its "normsCache" and in the "normsCache" of its sub-readers; each array has one byte for each document in the index and in some cases there will be multiple of these arrays associated with differet fields. For an index with millions of records this can result in an apparent "leak" of hundreds of megabytes of memory as those readers are not re-used and the MRU cache used by default will keep up to 128 hard references to the readers by default. The search system must either re-use or delete the normsCache, OR the cache key for these filters should be tied to something else that doesn't keep references to potentially huge data arrays. Otherwise the scalability of the search subsystem is significantly impacted when using filters, as you must have enough heap to accommodate up to 128 times as many copies of the norms arrays.

Dobes Vandermeer (JIRA)

11:14 a.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-469) Filter caching using causes excessive memory use

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-469?pag... ] Dobes Vandermeer commented on HSEARCH-469: ------------------------------------------ Hi Sanne, It would certainly reduce the size of the memory leak and probably avoid a server crash. However, it would still be wasting a fair amount of memory. Iin my quest for a speedy and reliable setup I've gotten rid of the use of filters and, subsequently, hibernate search itself, so there's really nothing further for you to do in order to help me specifically, but for the benefit of future users it may be good to reduce memory consumption by looking at the behavior of this filter cache. Cheers, Dobes

...

Filter caching using causes excessive memory use ------------------------------------------------ Key: HSEARCH-469 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-469 Project: Hibernate Search Issue Type: Bug Components: query Affects Versions: 3.1.1.GA Environment: hibernate-core-3.3.2.GA Reporter: Dobes Vandermeer Assignee: Sanne Grinovero Fix For: 3.2.0 The CachingWrapperFilter uses the reader instance (CacheableMultiReader) as a key for the caching. However, the reader instance keeps pointers to byte arrays in its "normsCache" and in the "normsCache" of its sub-readers; each array has one byte for each document in the index and in some cases there will be multiple of these arrays associated with differet fields. For an index with millions of records this can result in an apparent "leak" of hundreds of megabytes of memory as those readers are not re-used and the MRU cache used by default will keep up to 128 hard references to the readers by default. The search system must either re-use or delete the normsCache, OR the cache key for these filters should be tied to something else that doesn't keep references to potentially huge data arrays. Otherwise the scalability of the search subsystem is significantly impacted when using filters, as you must have enough heap to accommodate up to 128 times as many copies of the norms arrays.

Sanne Grinovero (JIRA)

12:06 p.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-469) Filter caching using causes excessive memory use

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-469?pag... ] Sanne Grinovero commented on HSEARCH-469: ----------------------------------------- {quote}It would certainly reduce the size of the memory leak and probably avoid a server crash{quote} why do you say reduce? if you set it to zero no memory would be leaked; and I wouldn't call it a leak as this value is being respected by use request: filters aren't cached by default. {quote}However, it would still be wasting a fair amount of memory.{quote} right, I still think it's important to do - just not classifying it as a bug but as a nice improvement. {quote}in my quest for a speedy and reliable setup I've gotten rid of the use of filters and, subsequently, hibernate search itself{quote} While I almost understand your position about the filters, what did you find in the rest of Search to work against your desire for speed and reliability? I have been in a similar position two years ago but I think I contributed all improvements I could spot, and I'm not finding evident areas which could have some exciting improvement. Sorry for OT, feel free to use the forum or mailing-list if you have good ideas.

...

Filter caching using causes excessive memory use ------------------------------------------------ Key: HSEARCH-469 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-469 Project: Hibernate Search Issue Type: Bug Components: query Affects Versions: 3.1.1.GA Environment: hibernate-core-3.3.2.GA Reporter: Dobes Vandermeer Assignee: Sanne Grinovero Fix For: 3.2.0 The CachingWrapperFilter uses the reader instance (CacheableMultiReader) as a key for the caching. However, the reader instance keeps pointers to byte arrays in its "normsCache" and in the "normsCache" of its sub-readers; each array has one byte for each document in the index and in some cases there will be multiple of these arrays associated with differet fields. For an index with millions of records this can result in an apparent "leak" of hundreds of megabytes of memory as those readers are not re-used and the MRU cache used by default will keep up to 128 hard references to the readers by default. The search system must either re-use or delete the normsCache, OR the cache key for these filters should be tied to something else that doesn't keep references to potentially huge data arrays. Otherwise the scalability of the search subsystem is significantly impacted when using filters, as you must have enough heap to accommodate up to 128 times as many copies of the norms arrays.

Dobes Vandermeer (JIRA)

1:22 p.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-469) Filter caching using causes excessive memory use

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-469?pag... ] Dobes Vandermeer commented on HSEARCH-469: ------------------------------------------ Hi Sanne, It's off-topic, yes. I think the most of the needed improvements are already in the system but not yet implemented or released. At the same time, our use of lucene is simple enough that it's easier to reimplement search using lucene directly than to patch hibernate-search. Hope that makes sense ... especially I am changing the system to store indexes in the database, and one set of indexes per tenant in our multi-tenant system. Each tenant has a relatively small set of records to search and searching is never done across tenants, so this reduces our memory footprint on a search considerably. Storing the indexes in the DB reduces IT complexity for backups, database clones, transactions, and unit tests and although performance may be slightly impacted, it seems worth the price in simplifying a bunch of other things.

...

Filter caching using causes excessive memory use ------------------------------------------------ Key: HSEARCH-469 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-469 Project: Hibernate Search Issue Type: Bug Components: query Affects Versions: 3.1.1.GA Environment: hibernate-core-3.3.2.GA Reporter: Dobes Vandermeer Assignee: Sanne Grinovero Fix For: 3.2.0 The CachingWrapperFilter uses the reader instance (CacheableMultiReader) as a key for the caching. However, the reader instance keeps pointers to byte arrays in its "normsCache" and in the "normsCache" of its sub-readers; each array has one byte for each document in the index and in some cases there will be multiple of these arrays associated with differet fields. For an index with millions of records this can result in an apparent "leak" of hundreds of megabytes of memory as those readers are not re-used and the MRU cache used by default will keep up to 128 hard references to the readers by default. The search system must either re-use or delete the normsCache, OR the cache key for these filters should be tied to something else that doesn't keep references to potentially huge data arrays. Otherwise the scalability of the search subsystem is significantly impacted when using filters, as you must have enough heap to accommodate up to 128 times as many copies of the norms arrays.

Sanne Grinovero (JIRA)

5:10 p.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-469) Filter caching using causes excessive memory use

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-469?pag... ] Sanne Grinovero commented on HSEARCH-469: ----------------------------------------- {quote}I think the most of the needed improvements are already in the system but not yet implemented or released{quote} feeling better :)

...

Filter caching using causes excessive memory use ------------------------------------------------ Key: HSEARCH-469 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-469 Project: Hibernate Search Issue Type: Bug Components: query Affects Versions: 3.1.1.GA Environment: hibernate-core-3.3.2.GA Reporter: Dobes Vandermeer Assignee: Sanne Grinovero Fix For: 3.2.0 The CachingWrapperFilter uses the reader instance (CacheableMultiReader) as a key for the caching. However, the reader instance keeps pointers to byte arrays in its "normsCache" and in the "normsCache" of its sub-readers; each array has one byte for each document in the index and in some cases there will be multiple of these arrays associated with differet fields. For an index with millions of records this can result in an apparent "leak" of hundreds of megabytes of memory as those readers are not re-used and the MRU cache used by default will keep up to 128 hard references to the readers by default. The search system must either re-use or delete the normsCache, OR the cache key for these filters should be tied to something else that doesn't keep references to potentially huge data arrays. Otherwise the scalability of the search subsystem is significantly impacted when using filters, as you must have enough heap to accommodate up to 128 times as many copies of the norms arrays.

Sanne Grinovero (JIRA)

Friday, 30 April Fri, 30 Apr

12:16 p.m.

New subject: [Hibernate-JIRA] Updated: (HSEARCH-469) Filter caching using causes excessive memory use

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-469?pag... ] Sanne Grinovero updated HSEARCH-469: ------------------------------------ Affects Version/s: 3.2.0.Beta1 3.2.0.CR1 Issue Type: Improvement (was: Bug) I changed it to "improvement" as it's caching as defined in configuration properties. I agree it would be nice to remove cached results as soon as they aren't needed, but it looks like quite a change, I'd suggest to move it to 3.3 unless there's going to be another CR: too many changes for a release imho.

...

Filter caching using causes excessive memory use ------------------------------------------------ Key: HSEARCH-469 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-469 Project: Hibernate Search Issue Type: Improvement Components: query Affects Versions: 3.1.1.GA, 3.2.0.Beta1, 3.2.0.CR1 Environment: hibernate-core-3.3.2.GA Reporter: Dobes Vandermeer Assignee: Sanne Grinovero Fix For: 3.2.0 The CachingWrapperFilter uses the reader instance (CacheableMultiReader) as a key for the caching. However, the reader instance keeps pointers to byte arrays in its "normsCache" and in the "normsCache" of its sub-readers; each array has one byte for each document in the index and in some cases there will be multiple of these arrays associated with differet fields. For an index with millions of records this can result in an apparent "leak" of hundreds of megabytes of memory as those readers are not re-used and the MRU cache used by default will keep up to 128 hard references to the readers by default. The search system must either re-use or delete the normsCache, OR the cache key for these filters should be tied to something else that doesn't keep references to potentially huge data arrays. Otherwise the scalability of the search subsystem is significantly impacted when using filters, as you must have enough heap to accommodate up to 128 times as many copies of the norms arrays.

Emmanuel Bernard (JIRA)

Tuesday, 4 May Tue, 4 May

3:47 a.m.

New subject: [Hibernate-JIRA] Updated: (HSEARCH-469) Filter caching using causes excessive memory use

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-469?pag... ] Emmanuel Bernard updated HSEARCH-469: ------------------------------------- Fix Version/s: (was: 3.2.0) 3.3.0

...

Filter caching using causes excessive memory use ------------------------------------------------ Key: HSEARCH-469 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-469 Project: Hibernate Search Issue Type: Improvement Components: query Affects Versions: 3.1.1.GA, 3.2.0.Beta1, 3.2.0.CR1 Environment: hibernate-core-3.3.2.GA Reporter: Dobes Vandermeer Assignee: Sanne Grinovero Fix For: 3.3.0 The CachingWrapperFilter uses the reader instance (CacheableMultiReader) as a key for the caching. However, the reader instance keeps pointers to byte arrays in its "normsCache" and in the "normsCache" of its sub-readers; each array has one byte for each document in the index and in some cases there will be multiple of these arrays associated with differet fields. For an index with millions of records this can result in an apparent "leak" of hundreds of megabytes of memory as those readers are not re-used and the MRU cache used by default will keep up to 128 hard references to the readers by default. The search system must either re-use or delete the normsCache, OR the cache key for these filters should be tied to something else that doesn't keep references to potentially huge data arrays. Otherwise the scalability of the search subsystem is significantly impacted when using filters, as you must have enough heap to accommodate up to 128 times as many copies of the norms arrays.

Emmanuel Bernard (JIRA)

Saturday, 6 November Sat, 6 Nov

9:59 a.m.

New subject: [Hibernate-JIRA] Updated: (HSEARCH-469) Filter caching using causes excessive memory use

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-469?pag... ] Emmanuel Bernard updated HSEARCH-469: ------------------------------------- Fix Version/s: (was: 3.3.0)

...

Filter caching using causes excessive memory use ------------------------------------------------ Key: HSEARCH-469 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-469 Project: Hibernate Search Issue Type: Improvement Components: query Affects Versions: 3.1.1.GA, 3.2.0.Beta1, 3.2.0.CR1 Environment: hibernate-core-3.3.2.GA Reporter: Dobes Vandermeer Assignee: Sanne Grinovero The CachingWrapperFilter uses the reader instance (CacheableMultiReader) as a key for the caching. However, the reader instance keeps pointers to byte arrays in its "normsCache" and in the "normsCache" of its sub-readers; each array has one byte for each document in the index and in some cases there will be multiple of these arrays associated with differet fields. For an index with millions of records this can result in an apparent "leak" of hundreds of megabytes of memory as those readers are not re-used and the MRU cache used by default will keep up to 128 hard references to the readers by default. The search system must either re-use or delete the normsCache, OR the cache key for these filters should be tied to something else that doesn't keep references to potentially huge data arrays. Otherwise the scalability of the search subsystem is significantly impacted when using filters, as you must have enough heap to accommodate up to 128 times as many copies of the norms arrays.

4972

days inactive

5207

days old

hibernate-issues@lists.jboss.org

Manage subscription

14 comments

3 participants

tags (0)

participants (3)

Dobes Vandermeer (JIRA)
Emmanuel Bernard (JIRA)
Sanne Grinovero (JIRA)

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

[Hibernate-JIRA] Created: (HSEARCH-469) Filter caching using causes excessive memory use