10:29 a.m.
Hi all,
It's been quite a long time since I came up with lazy initialization
problems in the mass indexer for the last time.
Unfortunately, this time, I don't see exactly how to fix this one and
I would like to ask for your advice about it.
The problem is quite simple:
I want to index the result of a method which invokes proxies along the
way. Something like:
@Field
public String getInformationWithProxy() {
return getField().getFieldWhichIsProxyfied().getValue();
}
As the fieldWhichIsProxyfied is not unproxyfied by an
objectInitializer, it doesn't have any session attached thus the lazy
initialization exception.
Note that the method is of course more complex than that and it mixes
fields from different fields and levels so we cannot use
@IndexedEmbedded.
As we use it to sort the results, we cannot put the information in
different fields either.
All in all, we're kinda stuck with this one.
Any bright idea out there?
Thanks.
--
Guillaume
1:43 p.m.
Hi Guillaume,
would you agree this looks like similar to HSEARCH-1260 ?
I'd like to step back a moment and re-evaluate how we faced all these
lazyinitialization issues.
As far as I remember, you and your team reported (and helped fixing!)
a depressingly long list of issues since approximately the migration
to Hibernate ORM 4 of your applications (adopting AFAIR early betas of
Search 4.2 at the time - thanks JIRA for the records).
So that data point means to me that you are in the sweet spot
intersection of the sets {people who reported issues} V {people who
actually use a non trivial Domain model}, and in the current era it
looks like we don't have many contributors matching your team on these
qualities.
In a previous era, when Hibernate Search was based on ORM 3.x, I was
working on a different project which had an extremely complex domain
model (about 400 Hibernate mapped entities, of which at least 50 where
@Indexed and with fairly advanced usage of custom types, custom
fieldbridges, Filters, etc...). I'm mentioning this nice domain model
I had, as I'm pretty sure that all these cases where working fine back
then :-)
I designed and implemented the MassIndexer back then, specifically
tailored for this complex domain, and it worked fine.
From the looks of all these issues, I'm pretty sure that my own
model
would have failed all the same tests you've described, and maybe even
more. Sadly, I don't have access to the staging DB of that application
anymore :-(
Why I'm sharing this data points?
The MassIndexer code didn't change significantly since when it worked
fine on my app, so I'm thinking now that we have been fixing the wrong
project. Would you agree that it looks like related to how ORM changed
some internal details of how proxies and lazy collections get
re-attached to ORM?
We probably should *remove* all the "reattaching proxies" logic
workarounds from the MassIndexer, let it fail hard and see if we can
get Hibernate ORM to behave like it did with Hibernate 3. That's
probably the better way of how we should have fixed these issues from
the beginning.
Hope I'm making sense ? It would also be great for use cases like HSEARCH-1260.
Sanne
On 8 January 2014 09:29, Guillaume Smet <guillaume.smet(a)gmail.com> wrote:
Hi all,
It's been quite a long time since I came up with lazy initialization
problems in the mass indexer for the last time.
Unfortunately, this time, I don't see exactly how to fix this one and
I would like to ask for your advice about it.
The problem is quite simple:
I want to index the result of a method which invokes proxies along the
way. Something like:
@Field
public String getInformationWithProxy() {
return getField().getFieldWhichIsProxyfied().getValue();
}
As the fieldWhichIsProxyfied is not unproxyfied by an
objectInitializer, it doesn't have any session attached thus the lazy
initialization exception.
Note that the method is of course more complex than that and it mixes
fields from different fields and levels so we cannot use
@IndexedEmbedded.
As we use it to sort the results, we cannot put the information in
different fields either.
All in all, we're kinda stuck with this one.
Any bright idea out there?
Thanks.
--
Guillaume
_______________________________________________
hibernate-dev mailing list
hibernate-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hibernate-dev
2:13 p.m.
Hi Sanne,
On Wed, Jan 8, 2014 at 1:43 PM, Sanne Grinovero <sanne(a)hibernate.org> wrote:
As far as I remember, you and your team reported (and helped
fixing!)
a depressingly long list of issues since approximately the migration
to Hibernate ORM 4 of your applications (adopting AFAIR early betas of
Search 4.2 at the time - thanks JIRA for the records).
So that data point means to me that you are in the sweet spot
intersection of the sets {people who reported issues} V {people who
actually use a non trivial Domain model}, and in the current era it
looks like we don't have many contributors matching your team on these
qualities.
We started contributing before the 4.x era but, after taking a look at
the bugs we reported, I agree that the lazy initialization exceptions
series started with 4.x.
And AFAIR, I was quite surprised to find those bugs back then. I
thought it was because we were working on more complex projects but
you might be right about the fact that they were not present before.
Why I'm sharing this data points?
The MassIndexer code didn't change significantly since when it worked
fine on my app, so I'm thinking now that we have been fixing the wrong
project. Would you agree that it looks like related to how ORM changed
some internal details of how proxies and lazy collections get
re-attached to ORM?
That's what I thought this morning while poking this issue although my
thoughts were not as clear as yours. We installed workarounds in the
mass indexer for a lot of cases but our current problem isn't fixable
with a workaround AFAICS.
Maybe there's something wrong/missing in the way the mass indexer is
initializing the Hibernate session stuff or something we cannot do
with the new 4.x architecture.
Anyway, it might be a good idea to have the thoughts of an ORM guy about this.
This looks like long term work so if anyone has any idea on how we can
make our code work, it would be much appreciated.
--
Guilllaume
1:06 a.m.
Hi Guillaume,
sorry I wanted to POC the following idea before replying, but couldn't
get to it yet, so rather than keeping you in the dark for longer I'll
share the idea: maybe you can try it yourself?
What you could try is to merge the second and first phases of the
MassIndexer into a single phase, so that you don't need to lock the
loaded entity again to "re-attach" it to the Session.
There currently is this pipeline, I guess you're familiar with it:
1- an initial phase of loading a stream of primary keys
2- transformation from batches of keys to batches of entitties
3- DocumentBuilder phase, which implicitly triggers all custom
fieldbridges and processes recursively all @IndexedEmbedded
4- Feeding it all to the backend, which applies it to the IndexWriters
Phase 1 is rather ok; Phase 2 is also quite simple; Phase 3 is
"tricky" because it needs to attach the state which was disconnected
from the Session used by threads in Phase 2.. and this is where we get
in trouble as apparently the old trick of using a Lock mode is not too
reliable (anymore?)
This pipeline was designed this way for various performance reasons,
but I suspect that merging Phase 2 and 3 in one could nowadays be
relatively as fast (as other things changed and some of the original
design decisions aren't necessarily applicable anymore).
When you merge these phases, the idea is of course that you use a
single Session to load the primary key to an entity AND transform it
fully using the DocumentBuilder, only after that you clear the Session
for the current batch.
Then, you can remove all the previous workarounds too: given we'd be
removing the re-attach processing and all the proxy re-attaching code,
it might even win you something in performance.. I don't know that for
sure, but seems worth to try out?
Cheers,
Sanne
On 8 January 2014 13:13, Guillaume Smet <guillaume.smet(a)gmail.com> wrote:
Hi Sanne,
On Wed, Jan 8, 2014 at 1:43 PM, Sanne Grinovero <sanne(a)hibernate.org> wrote:
> As far as I remember, you and your team reported (and helped fixing!)
> a depressingly long list of issues since approximately the migration
> to Hibernate ORM 4 of your applications (adopting AFAIR early betas of
> Search 4.2 at the time - thanks JIRA for the records).
> So that data point means to me that you are in the sweet spot
> intersection of the sets {people who reported issues} V {people who
> actually use a non trivial Domain model}, and in the current era it
> looks like we don't have many contributors matching your team on these
> qualities.
We started contributing before the 4.x era but, after taking a look at
the bugs we reported, I agree that the lazy initialization exceptions
series started with 4.x.
And AFAIR, I was quite surprised to find those bugs back then. I
thought it was because we were working on more complex projects but
you might be right about the fact that they were not present before.
> Why I'm sharing this data points?
> The MassIndexer code didn't change significantly since when it worked
> fine on my app, so I'm thinking now that we have been fixing the wrong
> project. Would you agree that it looks like related to how ORM changed
> some internal details of how proxies and lazy collections get
> re-attached to ORM?
That's what I thought this morning while poking this issue although my
thoughts were not as clear as yours. We installed workarounds in the
mass indexer for a lot of cases but our current problem isn't fixable
with a workaround AFAICS.
Maybe there's something wrong/missing in the way the mass indexer is
initializing the Hibernate session stuff or something we cannot do
with the new 4.x architecture.
Anyway, it might be a good idea to have the thoughts of an ORM guy about this.
This looks like long term work so if anyone has any idea on how we can
make our code work, it would be much appreciated.
--
Guilllaume
11:01 a.m.
Hi Sanne,
On Mon, Jan 13, 2014 at 1:06 AM, Sanne Grinovero <sanne(a)hibernate.org> wrote:
Then, you can remove all the previous workarounds too: given we'd
be
removing the re-attach processing and all the proxy re-attaching code,
it might even win you something in performance.. I don't know that for
sure, but seems worth to try out?
I agree that the separation of work between 2 phases seems to be the
root of all evil and we might be able to simplify a lot of code by
removing this.
I'm kinda swamped atm but I would like to take a look at this in
february if it's OK with you? I will probably need some guidance.
--
Guillaume
12:11 a.m.
On 13 January 2014 11:01, Guillaume Smet <guillaume.smet(a)gmail.com> wrote:
Hi Sanne,
On Mon, Jan 13, 2014 at 1:06 AM, Sanne Grinovero <sanne(a)hibernate.org> wrote:
> Then, you can remove all the previous workarounds too: given we'd be
> removing the re-attach processing and all the proxy re-attaching code,
> it might even win you something in performance.. I don't know that for
> sure, but seems worth to try out?
I agree that the separation of work between 2 phases seems to be the
root of all evil and we might be able to simplify a lot of code by
removing this.
I'm kinda swamped atm but I would like to take a look at this in
february if it's OK with you? I will probably need some guidance.
Of course I'm ok with that. If any I was under the impression you
needed a solution urgently; if that's the case, be aware I suspect
this could be a trivial change.
-- Sanne
10:09 a.m.
On Tue, Jan 14, 2014 at 12:11 AM, Sanne Grinovero <sanne(a)hibernate.org> wrote:
Of course I'm ok with that. If any I was under the impression
you
needed a solution urgently; if that's the case, be aware I suspect
this could be a trivial change.
It's for a sort on a table (but an important one) so I can make the
customer wait if I mention a solution in the near future.
I also suspect a trivial change but I think the cleanup might take more time.
--
Guillaume
3760
days inactive
3766
days old
6 comments
2 participants
participants (2)
-
Guillaume Smet -
Sanne Grinovero