[hibernate-dev] Hibernate Search: Transactions timeout on MassIndexer

[Sanne Grinovero]

[Tracked now as HSEARCH-1474]

thanks that's an interesting path to explore; still one problem is
consistency during on going changes: the way it works today most of
the work is processed by parallel and short lived transactions, but
the main stream of primary keys is a forward-only scrollable which
needs to select the primary keys only once, or otherwise pagination
out of transaction would require a fully locked database.

I simply can't load all those primary keys in memory: I need to page
on it while the backend threads make progress and consume the stream
to keep the memory usage under control.



On 23 December 2013 16:38, Emmanuel Bernard <emmanuel at hibernate.org> wrote:
> I guess you could do it mostly transparently for the user.
> Set a reasonably high timeout like say 2 mins, then every n operation look how much
> you've consumed. If you are close to the limit, you commit and restart a
> transaction.
> So your option would be semi public.
> hibernate.search.massindexer.transaction_timeout auto|time in (m)s
>
> If a user is in trouble with the auto system, we give him the option.
>
> Emmanuel
>
> On Mon 2013-12-23 14:54, Sanne Grinovero wrote:
>> Thanks that explains why I was missing the correlation.
>> To avoid the transaction manager to kill our indexing process, I guess
>> our only answer today is to reconfigure the transactions manager to
>> use larger timeouts?
>>
>> That's quite limiting, especially as it requires to change it
>> globally; ideally I'd want to ping the TM regularly to let it know
>> we're still busy (and doing fine) ?
>>
>> I couldn't find a method to do that by looking at the standard
>> javax.transaction API, so I'm wondering if we should take advantage of
>> some proprietary extensions.
>> Other than that, on the standard API I see we could set the TX timeout
>> for this specific transaction; we should probably add a new option to
>> the MassIndexer API to have the user suggest a
>> very-high-but-reasonable timeout?
>>
>> Sanne
>>
>> On 23 December 2013 10:09, Emmanuel Bernard <emmanuel at hibernate.org> wrote:
>> > Sorry late here.
>> > OptionallyWrapInJTATransaction is not about timeout. It's about
>> > explicitly starting a JTA transaction like a CMT entity bean would to so
>> > that Hibernate ORM does not complain.
>> >
>> > On Mon 2013-11-11 17:29, Sanne Grinovero wrote:
>> >> Hi Emmanuel,
>> >> in case you get very bored at Devoxx :)
>> >>
>> >> I remember you implementing a quite complex fix for my initial
>> >> MassIndexer which involved avoiding the transactions we use from
>> >> timing out.
>> >> This is probably more than a year old, but there is a user on the
>> >> forums now using 4.4.0.Final and having a suspiciously similar
>> >> problem:
>> >>
>> >> https://forum.hibernate.org/viewtopic.php?f=9&t=1029562
>> >>
>> >>
>> >> I've looked into our code, but I'm not understanding how the class
>> >> org.hibernate.search.batchindexing.impl.OptionallyWrapInJTATransaction
>> >> is supposed to prevent the transaction from timing out?
>> >>
>> >> Do you have any idea on the problem?
>> >> I recently had to apply some refactoring so I might have introduced a
>> >> regression but I need another pair of eyes.
>> >>
>> >> Tia,
>> >> Sanne