[hibernate-dev] Mass indexer and lazy initialization exceptions S03E04

[Sanne Grinovero]

Hi Guillaume,
sorry I wanted to POC the following idea before replying, but couldn't
get to it yet, so rather than keeping you in the dark for longer I'll
share the idea: maybe you can try it yourself?

What you could try is to merge the second and first phases of the
MassIndexer into a single phase, so that you don't need to lock the
loaded entity again to "re-attach" it to the Session.

There currently is this pipeline, I guess you're familiar with it:
1- an initial phase of loading a stream of primary keys
2- transformation from batches of keys to batches of entitties
3- DocumentBuilder phase, which implicitly triggers all custom
fieldbridges and processes recursively all @IndexedEmbedded
4- Feeding it all to the backend, which applies it to the IndexWriters

Phase 1 is rather ok; Phase 2 is also quite simple; Phase 3 is
"tricky" because it needs to attach the state which was disconnected
from the Session used by threads in Phase 2.. and this is where we get
in trouble as apparently the old trick of using a Lock mode is not too
reliable (anymore?)

This pipeline was designed this way for various performance reasons,
but I suspect that merging Phase 2 and 3 in one could nowadays be
relatively as fast (as other things changed and some of the original
design decisions aren't necessarily applicable anymore).
When you merge these phases, the idea is of course that you use a
single Session to load the primary key to an entity AND transform it
fully using the DocumentBuilder, only after that you clear the Session
for the current batch.

Then, you can remove all the previous workarounds too: given we'd be
removing the re-attach processing and all the proxy re-attaching code,
it might even win you something in performance.. I don't know that for
sure, but seems worth to try out?

Cheers,
Sanne



On 8 January 2014 13:13, Guillaume Smet <guillaume.smet at gmail.com> wrote:
> Hi Sanne,
>
> On Wed, Jan 8, 2014 at 1:43 PM, Sanne Grinovero <sanne at hibernate.org> wrote:
>> As far as I remember, you and your team reported (and helped fixing!)
>> a depressingly long list of issues since approximately the migration
>> to Hibernate ORM 4 of your applications (adopting AFAIR early betas of
>> Search 4.2 at the time - thanks JIRA for the records).
>> So that data point means to me that you are in the sweet spot
>> intersection of the sets {people who reported issues} V {people who
>> actually use a non trivial Domain model}, and in the current era it
>> looks like we don't have many contributors matching your team on these
>> qualities.
>
> We started contributing before the 4.x era but, after taking a look at
> the bugs we reported, I agree that the lazy initialization exceptions
> series started with 4.x.
>
> And AFAIR, I was quite surprised to find those bugs back then. I
> thought it was because we were working on more complex projects but
> you might be right about the fact that they were not present before.
>
>> Why I'm sharing this data points?
>> The MassIndexer code didn't change significantly since when it worked
>> fine on my app, so I'm thinking now that we have been fixing the wrong
>> project. Would you agree that it looks like related to how ORM changed
>> some internal details of how proxies and lazy collections get
>> re-attached to ORM?
>
> That's what I thought this morning while poking this issue although my
> thoughts were not as clear as yours. We installed workarounds in the
> mass indexer for a lot of cases but our current problem isn't fixable
> with a workaround AFAICS.
>
> Maybe there's something wrong/missing in the way the mass indexer is
> initializing the Hibernate session stuff or something we cannot do
> with the new 4.x architecture.
>
> Anyway, it might be a good idea to have the thoughts of an ORM guy about this.
>
> This looks like long term work so if anyone has any idea on how we can
> make our code work, it would be much appreciated.
>
> --
> Guilllaume