[hibernate-dev] Memory consumption

[Steve Ebersole]

Well another option is to change the way batch load sql is 
created/handled which is something that has come up many times before, 
and someone was just discussing on the list last few weeks.

Given a batch size, Hibernate currently generates a non-unintuitive 
number of loaders.  Lets look at an example number.  You gave 16 as the 
batch size.  That means Hibernate would generate loaders for sizes: 
16,10,9,8,7,6,5,4,3,2,1

The idea being how to efficiently load batches when the number of things 
to load is less that 16.

Deferring initialization of these batch loaders is one option, but that 
just really delays the memory consumption.  Personally, if it were my 
app I'd rather that sizing be done up front rather than over the course 
of the next few hours or weeks as the system continues to run.

Its been proposed a number of times to instead generate just a single 
loader for loading that number of items.  This would mean generating a 
single SQL statement that has parameter holders for the full batch size 
and then somehow "filling out" the empty slots when the number of things 
to load is less that 16.  If this could be made workable across all 
dialects, I personally think it would be the best approach as it 
probably gets close to the start up sizings you claimed but would never 
grow beyond that.  But we would need to discuss this to see if it is a 
real possibility.  Lets work an example with batch-size = 3.  That would 
produce SQL like "select a.x, a.y, a.z from a a where a.x = ? or a.x = ? 
or a.x = ?"

So for a case where we have 2 things to batch load, how do we best 
leverage that single statement?  Which really comes down to how we 
handle the the 3rd parameter binding.

I have seen simply reusing one of the keys as a proposal, but that will 
not work.  Null should work, but would it work in all cases against all 
dialects?


On 05/16/2012 03:14 PM, Andrej Golovnin wrote:
> Hi Scott,
>
>>
>> Nice find!  Could you drill into one of the LockMode loaders and see what the larger objects referenced by that are?
>>
>
> Largest objects are of type BatchingEntityLoader.
>
> The problem has to do with the value of hibernate.default_batch_fetch_size.
> In our case we use 16 as the value. In this case Hibernate creates 11 EntityLoaders
> for every entity class involved in a collection association. So the most memory
> is consumed by generated SQL statements. If I compare memory snapshots,
> I see ca. 100MB more of String objects  in JBoss 7 as in JBoss 4.
>
> Changing the value of hibernate.default_batch_fetch_size from 16 to 4
> reduces the size of SessionFactoryImpl in our case from ca. 370MB to 180MB.
> And if I use patched version of Hibernate as suggested by me, the size of
> SessionFactoryImpl is ca. 80MB. So we will now investigate, wether it is
> really needed to set hibernate.default_batch_fetch_size to 16 or maybe
> the lower value would also give us decent performance.
>
> Best regards
> Andrej Golovnin
> _______________________________________________
> hibernate-dev mailing list
> hibernate-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hibernate-dev

-- 
steve at hibernate.org
http://hibernate.org