[hibernate-dev] Hibernate Search: FieldCache usage for classtype extraction from Documents

[Hardy Ferentschik]

On Wed, 02 Mar 2011 19:48:33 +0100, Sanne Grinovero <sanne at hibernate.org>  
wrote

> so it's quite possible that this approach takes hostage of a lot of
> memory.

I guess that was our fear from the beginning.

> So to cope with this, I've now built a custom fieldCache and a
> specialized Collector, which extract the data and keep it in memory
> not as String but as Class instances, which have the advantage of
> pointing to the same instances.

I will have a look at it tomorrow.

> profiling, it's all about an innocently looking Map.put() which I'm
> now using in the custom collector.
> Basically, I'm going to need to change the approach, or kill the
> feature. I'm now collecting values into an array and will see how it
> goes.
>
> In case someone wants to have a look, there's a working branch here:
> https://github.com/Sanne/hibernate-search/tree/HSEARCH-531
>
> and this is the line which is taking now 93% CPU time during a fulltext  
> search:
> https://github.com/Sanne/hibernate-search/commit/735c4644834cc516effa33f632075fc716eb3b49#L8R57
> (Hardy beware of your map usage! this is worse than we expected.. you
> definitely have to switch to a per-field collector instance if
> possible)

Right. Going to the chained Collector approach is next on my todo list.
Maybe we can have a chat tomorrow about this.

--Hardy