Re: [hibernate-dev] Memory consumption
Markus Heiden wrote:
After executing actions, the actions (e.g. EntityInsertAction) will
be
moved from ActionQueue.inserts etc. to ActionQueue.executions to stay
there until the transaction completes. Because they do have a reference
to the instance being saved, they prevent that instance from being
garbage collected. Isn't it possible to clear the instance field after
execution? For example the EntityInsertAction needs the entity just for
getting the id. Can't the id be retrieved direcly after execution?
I ask, because I tried to insert many new objects to the database and
tried with Session.flush() and Session.evict() to get rid of these
objects each 1000 objects, because the memory usage was too high. But
for the above reason I had no chance to succeed. The workaround was to
commit the session more often, but this breaks transactional behaviour.
Any thoughts?
Yes I have noticed O(N) performance in ActionQueue looking for work to
do (or dirtyness or something) when there is a large active set of
Objects in play. Don't forget that your underlying DB has to also track
individual items too.
Is your issue memory or O(N) performance ? I found the performance was
the killer over a few 1000 objects.
Break the transaction down, i.e. have a 2 stage pattern with your data,
you initially create a persistable entity with a status=1 and commit
batches of items in this state.
Any code that picks up that persitable entity (in DAO layer) ignores any
instances where state=1 (conceptually they don't really exist yet since
they are non-active).
Once your large transaction has finished creating all these new entites
with state=1 then you run a new transaction which marks them all with
state=2 (conceptually this is an active state). You can maintain your
own active object list by simply holding the primary key value in a
list, and this is what you walk.
If your list is really huge then you may need to persist that separately
too! Until you can get the business logic within the constraints of the
CPU/memory resources you have for the task.
At some point given a large enough working set you will need to break
your problem up into smaller transactions and redesign your approach.
Maybe the above pattern has a name, its a bit like a Java builder
pattern but for the construction of large amounts of data in a
transactional way.
Maybe this pattern is no use to you and you really want the moon on a
stick to happen on your 386 with 640Kb.
Darryl