Loaders were designed to be an encapsulation of the query (the SQL, how to
build and execute the PreparedStatement) and the "load plan" (how the
ResultSet is processed). 4.2 changed how this last part is represented,
but overall the design is the same it always has been. The problem is that
Loader was designed such that it itself builds all of these pieces.
This is what you'd have to change - specifically to split the creation of
the "load plan" outside of the Loader, or at least have the ability to pass
in an externally built one
On Fri, May 4, 2018 at 3:50 PM Guillaume Smet <guillaume.smet(a)gmail.com>
wrote:
As already mentioned, I plan to take a look at this next week.
I also thought we could share most of the info between loaders but Steve
was thinking it might not be possible without massive changes.
I'll take a look and report back to all of you. I might be way over my head
as I don't know the codebase very well but I already did a very similar
work for HV with great results so it's worth a shot.
--
Guillaume
On Fri, May 4, 2018 at 10:31 PM, Gail Badner <gbadner(a)redhat.com> wrote:
> I *think* that most, if not all, of the loaders for a particular entity
> could be reworked to share most of the same information.
>
> IIRC, the only real differences are the batch size and the actual query
> that gets generated.
>
> I'm happy to look into this, but I don't want to duplicate what others
are
> doing. It a bit difficult collaborating on this since our time zones
don't
> overlap very well (and, unfortunately, I'm not a morning person).
>
> I have some other work I need to attend to today. Please let me know if I
> should dig into this next week.
>
> On Fri, May 4, 2018 at 12:30 PM, Guillaume Smet <
guillaume.smet(a)gmail.com>
> wrote:
>
>> Don't know about the history but with the entity I checked out in the
>> OP's case, each EntityLoader is approximately 200 KB.
>>
>> The entity contains a lot of columns so that's no surprise.
>>
>> What doesn't help is that I don't think the 13 EntityLoaders/entity
share
>> a lot of things so they are just adding up.
>>
>> On Fri, May 4, 2018 at 9:15 PM, Sanne Grinovero <sanne(a)hibernate.org>
>> wrote:
>>
>>> On 4 May 2018 at 19:45, Gail Badner <gbadner(a)redhat.com> wrote:
>>> > I don't think the batching strategies actually changed when we
moved
>>> to load
>>> > plans.
>>> >
>>> > Is the problem that each loader is taking considerably more memory?
>>>
>>> Yes that's right.
>>>
>>> >
>>> > On Fri, May 4, 2018 at 5:41 AM, Guillaume Smet <
>>> guillaume.smet(a)gmail.com>
>>> > wrote:
>>> >>
>>> >> On Fri, May 4, 2018 at 11:48 AM, Sanne Grinovero <
sanne(a)hibernate.org
>>> >
>>> >> wrote:
>>> >>
>>> >> > thanks for investigating. Steve in chat mentioned the option
of
>>> >> > creating these EntityLoaders lazily: as while they need to be
>>> reused,
>>> >> > it's possible many of them are never really used.
>>> >> >
>>> >> > I think that's a very interesting observation, is that
what you
>>> plan to
>>> >> > explore?
>>> >> >
>>> >>
>>> >> Yes, it's one of the things we should explore.
>>> >>
>>> >>
>>> >> > I see two difficulties:
>>> >> > - you'd need thredsafe code, yet we don't want
barriers on this
>>> area.
>>> >> > Will probably need features such as lazy set, yet these would
need
>>> >> > even more objects allocated if you're not careful.
>>> >> > - we always prefer fail-at-boot validation, so we'd need
to make
>>> sure
>>> >> > even if you don't build them all, we can at least make
sure we'll
be
>>> >> > able to build them later.
>>> >> >
>>> >>
>>> >> Well, the idea is to prototype something on the memory usage
front,
>>> then
>>> >> we
>>> >> can discuss further what we want to do with it and if it's
worth
>>> refining
>>> >> it and spending more time on it.
>>> >>
>>> >> If we end up choosing this path, be sure we'll be bugging you
for
>>> advice
>>> >> on
>>> >> concurrency and performance :).
>>> >>
>>> >>
>>> >> > Additionally there's the idea to see if any such Loader
instance
>>> could
>>> >> > be reused - probably using a cache which we clear at end of
>>> bootstrap
>>> >> > - yet since Steve mentioned this would be a significant
change
let's
>>> >> > not try to push it in 5.3.
>>> >> >
>>> >> > Finally, a simple one: I have the impression that loader's
Map
could
>>> >> > be made into a plain array, or a custom structure which would
then
>>> be
>>> >> > ideal to support lazy creation of loaders. With the exception
of
two
>>> >> > special strings - which will needt to be treated specially -
all
>>> other
>>> >> > keys are enums.
>>> >> >
>>> >>
>>> >> Yeah, I'll play with a few ideas and see what I can do. I'm
not yet
>>> sure
>>> >> if
>>> >> not knowing the code base will be a thing in my favor or not. At
>>> least I
>>> >> don't have preconceived ideas :].
>>> >>
>>> >>
>>> >> > Whatever you choose to try, let's keep significant changes
for 6
>>> only :)
>>> >> >
>>> >>
>>> >> Steve was saying that this whole thing wouldn't be an issue in
6.
>>> >>
>>> >> --
>>> >> Guillaume
>>> >> _______________________________________________
>>> >> hibernate-dev mailing list
>>> >> hibernate-dev(a)lists.jboss.org
>>> >>
https://lists.jboss.org/mailman/listinfo/hibernate-dev
>>> >
>>> >
>>> _______________________________________________
>>> hibernate-dev mailing list
>>> hibernate-dev(a)lists.jboss.org
>>>
https://lists.jboss.org/mailman/listinfo/hibernate-dev
>>>
>>
>>
>
_______________________________________________
hibernate-dev mailing list
hibernate-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hibernate-dev