[hibernate-dev] SingleTableEntityPersister memory footprint in 5.3

[Sanne Grinovero]

Hi Guillaume,

thanks for investigating. Steve in chat mentioned the option of
creating these EntityLoaders lazily: as while they need to be reused,
it's possible many of them are never really used.

I think that's a very interesting observation, is that what you plan to explore?

I see two difficulties:
 - you'd need thredsafe code, yet we don't want barriers on this area.
Will probably need features such as lazy set, yet these would need
even more objects allocated if you're not careful.
 - we always prefer fail-at-boot validation, so we'd need to make sure
even if you don't build them all, we can at least make sure we'll be
able to build them later.

It's worth remembering that the JVM can actually optimise ORM quite
well as there are not many memory barriers in the most complex areas.

Additionally there's the idea to see if any such Loader instance could
be reused - probably using a cache which we clear at end of bootstrap
- yet since Steve mentioned this would be a significant change let's
not try to push it in 5.3.

Finally, a simple one: I have the impression that loader's Map could
be made into a plain array, or a custom structure which would then be
ideal to support lazy creation of loaders. With the exception of two
special strings - which will needt to be treated specially - all other
keys are enums.

Whatever you choose to try, let's keep significant changes for 6 only :)

Thanks,
Sanne



On 4 May 2018 at 10:06, Guillaume Smet <guillaume.smet at gmail.com> wrote:
> Hi,
>
> So, I analyzed the dumps yesterday evening. The problem is real, meaning his
> SessionFactory is consuming more than 1GB of memory for 600+ tables, some
> with a lot of attributes. So for sure, the model is a big one, but it would
> be nice to be more gentle with this type of configuration. I don't think
> it's something new to 5.3 though as it's not the first time we have this
> type of reports.
>
> From my observations, the problem mostly comes from:
> LegacyBatchingEntityLoader
>  \_ loaders - EntityLoader
>      \_ staticLoadQuery - EntityLoadQueryDetails
>          \_ rootReturn - EntityReturnImpl
>
> The largest LegacyBatchingEntityLoader I have in the dump takes more than 2
> MB.
>
> Keep in mind that with a batch size of 50, we have 13 EntityLoaders for
> batch sizes of 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 20 and 50, each loader
> taking ~ 200 KB.
>
> We discussed some ideas yesterday with Steve.
>
> I'll try to experiment a few ideas next week to see if I can come up with
> some mitigation. I'm not that familiar with the ORM source code so I'm full
> of crazy ideas to try - most of them will probably prove to be bad (Steve
> already shut some of them) but maybe one will survive and be beneficial to
> the problem at hand.
>
> I'll keep you posted if I can come up with something.
>
> --
> Guillaume
>