[Tracked now as HSEARCH-1474]
thanks that's an interesting path to explore; still one problem is
consistency during on going changes: the way it works today most of
the work is processed by parallel and short lived transactions, but
the main stream of primary keys is a forward-only scrollable which
needs to select the primary keys only once, or otherwise pagination
out of transaction would require a fully locked database.
I simply can't load all those primary keys in memory: I need to page
on it while the backend threads make progress and consume the stream
to keep the memory usage under control.
On 23 December 2013 16:38, Emmanuel Bernard <emmanuel(a)hibernate.org> wrote:
I guess you could do it mostly transparently for the user.
Set a reasonably high timeout like say 2 mins, then every n operation look how much
you've consumed. If you are close to the limit, you commit and restart a
transaction.
So your option would be semi public.
hibernate.search.massindexer.transaction_timeout auto|time in (m)s
If a user is in trouble with the auto system, we give him the option.
Emmanuel
On Mon 2013-12-23 14:54, Sanne Grinovero wrote:
> Thanks that explains why I was missing the correlation.
> To avoid the transaction manager to kill our indexing process, I guess
> our only answer today is to reconfigure the transactions manager to
> use larger timeouts?
>
> That's quite limiting, especially as it requires to change it
> globally; ideally I'd want to ping the TM regularly to let it know
> we're still busy (and doing fine) ?
>
> I couldn't find a method to do that by looking at the standard
> javax.transaction API, so I'm wondering if we should take advantage of
> some proprietary extensions.
> Other than that, on the standard API I see we could set the TX timeout
> for this specific transaction; we should probably add a new option to
> the MassIndexer API to have the user suggest a
> very-high-but-reasonable timeout?
>
> Sanne
>
> On 23 December 2013 10:09, Emmanuel Bernard <emmanuel(a)hibernate.org> wrote:
> > Sorry late here.
> > OptionallyWrapInJTATransaction is not about timeout. It's about
> > explicitly starting a JTA transaction like a CMT entity bean would to so
> > that Hibernate ORM does not complain.
> >
> > On Mon 2013-11-11 17:29, Sanne Grinovero wrote:
> >> Hi Emmanuel,
> >> in case you get very bored at Devoxx :)
> >>
> >> I remember you implementing a quite complex fix for my initial
> >> MassIndexer which involved avoiding the transactions we use from
> >> timing out.
> >> This is probably more than a year old, but there is a user on the
> >> forums now using 4.4.0.Final and having a suspiciously similar
> >> problem:
> >>
> >>
https://forum.hibernate.org/viewtopic.php?f=9&t=1029562
> >>
> >>
> >> I've looked into our code, but I'm not understanding how the class
> >> org.hibernate.search.batchindexing.impl.OptionallyWrapInJTATransaction
> >> is supposed to prevent the transaction from timing out?
> >>
> >> Do you have any idea on the problem?
> >> I recently had to apply some refactoring so I might have introduced a
> >> regression but I need another pair of eyes.
> >>
> >> Tia,
> >> Sanne