[hibernate-dev] Memory consumption

[Steve Ebersole]

actually, thinking about it more, not so sure the "reusing keys" 
approach would not work.

On Wed 16 May 2012 04:35:36 PM CDT, Steve Ebersole wrote:
> Well another option is to change the way batch load sql is
> created/handled which is something that has come up many times before,
> and someone was just discussing on the list last few weeks.
>
> Given a batch size, Hibernate currently generates a non-unintuitive
> number of loaders. Lets look at an example number. You gave 16 as the
> batch size. That means Hibernate would generate loaders for sizes:
> 16,10,9,8,7,6,5,4,3,2,1
>
> The idea being how to efficiently load batches when the number of
> things to load is less that 16.
>
> Deferring initialization of these batch loaders is one option, but
> that just really delays the memory consumption. Personally, if it were
> my app I'd rather that sizing be done up front rather than over the
> course of the next few hours or weeks as the system continues to run.
>
> Its been proposed a number of times to instead generate just a single
> loader for loading that number of items. This would mean generating a
> single SQL statement that has parameter holders for the full batch
> size and then somehow "filling out" the empty slots when the number of
> things to load is less that 16. If this could be made workable across
> all dialects, I personally think it would be the best approach as it
> probably gets close to the start up sizings you claimed but would
> never grow beyond that. But we would need to discuss this to see if it
> is a real possibility. Lets work an example with batch-size = 3. That
> would produce SQL like "select a.x, a.y, a.z from a a where a.x = ? or
> a.x = ? or a.x = ?"
>
> So for a case where we have 2 things to batch load, how do we best
> leverage that single statement? Which really comes down to how we
> handle the the 3rd parameter binding.
>
> I have seen simply reusing one of the keys as a proposal, but that
> will not work. Null should work, but would it work in all cases
> against all dialects?
>
>
> On 05/16/2012 03:14 PM, Andrej Golovnin wrote:
>> Hi Scott,
>>
>>>
>>> Nice find! Could you drill into one of the LockMode loaders and see
>>> what the larger objects referenced by that are?
>>>
>>
>> Largest objects are of type BatchingEntityLoader.
>>
>> The problem has to do with the value of
>> hibernate.default_batch_fetch_size.
>> In our case we use 16 as the value. In this case Hibernate creates 11
>> EntityLoaders
>> for every entity class involved in a collection association. So the
>> most memory
>> is consumed by generated SQL statements. If I compare memory snapshots,
>> I see ca. 100MB more of String objects in JBoss 7 as in JBoss 4.
>>
>> Changing the value of hibernate.default_batch_fetch_size from 16 to 4
>> reduces the size of SessionFactoryImpl in our case from ca. 370MB to
>> 180MB.
>> And if I use patched version of Hibernate as suggested by me, the
>> size of
>> SessionFactoryImpl is ca. 80MB. So we will now investigate, wether it is
>> really needed to set hibernate.default_batch_fetch_size to 16 or maybe
>> the lower value would also give us decent performance.
>>
>> Best regards
>> Andrej Golovnin
>> _______________________________________________
>> hibernate-dev mailing list
>> hibernate-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>

--
steve at hibernate.org
http://hibernate.org