[hibernate-dev] SingleTableEntityPersister memory footprint in 5.3

[Steve Ebersole]

Loaders were designed to be an encapsulation of the query (the SQL, how to
build and execute the PreparedStatement) and the "load plan" (how the
ResultSet is processed).  4.2 changed how this last part is represented,
but overall the design is the same it always has been.  The problem is that
Loader was designed such that it itself builds all of these pieces.

This is what you'd have to change - specifically to split the creation of
the "load plan" outside of the Loader, or at least have the ability to pass
in an externally built one

On Fri, May 4, 2018 at 3:50 PM Guillaume Smet <guillaume.smet at gmail.com>
wrote:

> As already mentioned, I plan to take a look at this next week.
>
> I also thought we could share most of the info between loaders but Steve
> was thinking it might not be possible without massive changes.
>
> I'll take a look and report back to all of you. I might be way over my head
> as I don't know the codebase very well but I already did a very similar
> work for HV with great results so it's worth a shot.
>
> --
> Guillaume
>
> On Fri, May 4, 2018 at 10:31 PM, Gail Badner <gbadner at redhat.com> wrote:
>
> > I *think* that most, if not all, of the loaders for a particular entity
> > could be reworked to share most of the same information.
> >
> > IIRC, the only real differences are the batch size and the actual query
> > that gets generated.
> >
> > I'm happy to look into this, but I don't want to duplicate what others
> are
> > doing. It a bit difficult collaborating on this since our time zones
> don't
> > overlap very well (and, unfortunately, I'm not a morning person).
> >
> > I have some other work I need to attend to today. Please let me know if I
> > should dig into this next week.
> >
> > On Fri, May 4, 2018 at 12:30 PM, Guillaume Smet <
> guillaume.smet at gmail.com>
> > wrote:
> >
> >> Don't know about the history but with the entity I checked out in the
> >> OP's case, each EntityLoader is approximately 200 KB.
> >>
> >> The entity contains a lot of columns so that's no surprise.
> >>
> >> What doesn't help is that I don't think the 13 EntityLoaders/entity
> share
> >> a lot of things so they are just adding up.
> >>
> >> On Fri, May 4, 2018 at 9:15 PM, Sanne Grinovero <sanne at hibernate.org>
> >> wrote:
> >>
> >>> On 4 May 2018 at 19:45, Gail Badner <gbadner at redhat.com> wrote:
> >>> > I don't think the batching strategies actually changed when we moved
> >>> to load
> >>> > plans.
> >>> >
> >>> > Is the problem that each loader is taking considerably more memory?
> >>>
> >>> Yes that's right.
> >>>
> >>> >
> >>> > On Fri, May 4, 2018 at 5:41 AM, Guillaume Smet <
> >>> guillaume.smet at gmail.com>
> >>> > wrote:
> >>> >>
> >>> >> On Fri, May 4, 2018 at 11:48 AM, Sanne Grinovero <
> sanne at hibernate.org
> >>> >
> >>> >> wrote:
> >>> >>
> >>> >> > thanks for investigating. Steve in chat mentioned the option of
> >>> >> > creating these EntityLoaders lazily: as while they need to be
> >>> reused,
> >>> >> > it's possible many of them are never really used.
> >>> >> >
> >>> >> > I think that's a very interesting observation, is that what you
> >>> plan to
> >>> >> > explore?
> >>> >> >
> >>> >>
> >>> >> Yes, it's one of the things we should explore.
> >>> >>
> >>> >>
> >>> >> > I see two difficulties:
> >>> >> >  - you'd need thredsafe code, yet we don't want barriers on this
> >>> area.
> >>> >> > Will probably need features such as lazy set, yet these would need
> >>> >> > even more objects allocated if you're not careful.
> >>> >> >  - we always prefer fail-at-boot validation, so we'd need to make
> >>> sure
> >>> >> > even if you don't build them all, we can at least make sure we'll
> be
> >>> >> > able to build them later.
> >>> >> >
> >>> >>
> >>> >> Well, the idea is to prototype something on the memory usage front,
> >>> then
> >>> >> we
> >>> >> can discuss further what we want to do with it and if it's worth
> >>> refining
> >>> >> it and spending more time on it.
> >>> >>
> >>> >> If we end up choosing this path, be sure we'll be bugging you for
> >>> advice
> >>> >> on
> >>> >> concurrency and performance :).
> >>> >>
> >>> >>
> >>> >> > Additionally there's the idea to see if any such Loader instance
> >>> could
> >>> >> > be reused - probably using a cache which we clear at end of
> >>> bootstrap
> >>> >> > - yet since Steve mentioned this would be a significant change
> let's
> >>> >> > not try to push it in 5.3.
> >>> >> >
> >>> >> > Finally, a simple one: I have the impression that loader's Map
> could
> >>> >> > be made into a plain array, or a custom structure which would then
> >>> be
> >>> >> > ideal to support lazy creation of loaders. With the exception of
> two
> >>> >> > special strings - which will needt to be treated specially - all
> >>> other
> >>> >> > keys are enums.
> >>> >> >
> >>> >>
> >>> >> Yeah, I'll play with a few ideas and see what I can do. I'm not yet
> >>> sure
> >>> >> if
> >>> >> not knowing the code base will be a thing in my favor or not. At
> >>> least I
> >>> >> don't have preconceived ideas :].
> >>> >>
> >>> >>
> >>> >> > Whatever you choose to try, let's keep significant changes for 6
> >>> only :)
> >>> >> >
> >>> >>
> >>> >> Steve was saying that this whole thing wouldn't be an issue in 6.
> >>> >>
> >>> >> --
> >>> >> Guillaume
> >>> >> _______________________________________________
> >>> >> hibernate-dev mailing list
> >>> >> hibernate-dev at lists.jboss.org
> >>> >> https://lists.jboss.org/mailman/listinfo/hibernate-dev
> >>> >
> >>> >
> >>> _______________________________________________
> >>> hibernate-dev mailing list
> >>> hibernate-dev at lists.jboss.org
> >>> https://lists.jboss.org/mailman/listinfo/hibernate-dev
> >>>
> >>
> >>
> >
> _______________________________________________
> hibernate-dev mailing list
> hibernate-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>