[hibernate-dev] Memory consumption

[Darryl Miles]

Markus Heiden wrote:
> After executing actions, the actions (e.g. EntityInsertAction) will be 
> moved from ActionQueue.inserts etc. to ActionQueue.executions to stay 
> there until the transaction completes. Because they do have a reference 
> to the instance being saved, they prevent that instance from being  
> garbage collected. Isn't it possible to clear the instance field after 
> execution? For example the EntityInsertAction needs the entity just for 
> getting the id. Can't the id be retrieved direcly after execution?
> 
> I ask, because I tried to insert many new objects to the database and 
> tried with Session.flush() and Session.evict() to get rid of these 
> objects each 1000 objects, because the memory usage was too high. But 
> for the above reason I had no chance to succeed. The workaround was to 
> commit the session more often, but this breaks transactional behaviour.
> 
> Any thoughts?

Yes I have noticed O(N) performance in ActionQueue looking for work to 
do (or dirtyness or something) when there is a large active set of 
Objects in play.  Don't forget that your underlying DB has to also track 
individual items too.

Is your issue memory or O(N) performance ?  I found the performance was 
the killer over a few 1000 objects.


Break the transaction down, i.e. have a 2 stage pattern with your data, 
you initially create a persistable entity with a status=1 and commit 
batches of items in this state.

Any code that picks up that persitable entity (in DAO layer) ignores any 
instances where state=1 (conceptually they don't really exist yet since 
they are non-active).

Once your large transaction has finished creating all these new entites 
with state=1 then you run a new transaction which marks them all with 
state=2 (conceptually this is an active state).  You can maintain your 
own active object list by simply holding the primary key value in a 
list, and this is what you walk.

If your list is really huge then you may need to persist that separately 
too!  Until you can get the business logic within the constraints of the 
CPU/memory resources you have for the task.


At some point given a large enough working set you will need to break 
your problem up into smaller transactions and redesign your approach.

Maybe the above pattern has a name, its a bit like a Java builder 
pattern but for the construction of large amounts of data in a 
transactional way.


Maybe this pattern is no use to you and you really want the moon on a 
stick to happen on your 386 with 640Kb.

Darryl