[hibernate-dev] [infinispan-dev] Infinispan tx, config and multithreading

Thu Aug 13 17:55:58 EDT 2009

IndexWriter processes index update and delegates some job to other
threads and waits when they finish. These "other" threads works on
data modified
in IndexWriter transaction. So I think if I use transaction per
thread, "others" would not see data modified by IndexWriter until
commit.

2009/8/13, Emmanuel Bernard <emmanuel at hibernate.org>:
> Ah I thought it was using multiple threads because of your mass
> indexing. I did not know some threads were span specifically for the
> Infinispan directory.
>
> On 13 août 09, at 17:34, Sanne Grinovero wrote:
>
>> Hi Łukasz,
>> what is your usage of these threads? did you consider using one
>> transaction per thread?
>>
>> Sanne
>>
>> 2009/8/13 Łukasz Moreń <lukasz.moren at gmail.com>:
>>> Newly created threads were not associated with any transaction, so I
>>> suppose it was a problem. Sharing transaction between threads seems
>>> to
>>> be a good solution.
>>> Thanks for help!
>>>
>>> 2009/8/13, Jason T. Greene <jason.greene at redhat.com>:
>>>> Correct. Also there could be read races as well, so if you are
>>>> going to
>>>> share a tx between threads, i would use some shared lock to
>>>> gaurantee
>>>> that only one thread can use it at a time. BTW this means you have
>>>> to
>>>> properly suspend/resume the TX via the TM API as well.
>>>>
>>>> Emmanuel Bernard wrote:
>>>>> Modifying a transaction means applying muations (like SQL INSERT /
>>>>> UPDATE / DELETE) to the transactional resource?
>>>>>
>>>>> On 13 août 09, at 15:07, Jason T. Greene wrote:
>>>>>
>>>>>> When using transactions, the context is bound to the
>>>>>> transaction, and
>>>>>> you can move a transaction between threads. However, you should
>>>>>> only
>>>>>> be modifying a transaction with one thread at a time.
>>>>>>
>>>>>> Emmanuel Bernard wrote:
>>>>>>> Could it be that you are not using the same transaction between
>>>>>>> different threads (ie you physically start different ones or
>>>>>>> different  "Infinispan contexts")?
>>>>>>> Infini guys, do you support transactional operation spanning
>>>>>>> several
>>>>>>> concurrent threads?
>>>>>>> On 13 août 09, at 14:04, Łukasz Moreń wrote:
>>>>>>>> I've tried with JBoss AS transaction manager and
>>>>>>>> JBossStandaloneTM.
>>>>>>>> The result is this same in all cases - error during merge.
>>>>>>>>
>>>>>>>> 2009/8/12, Emmanuel Bernard <emmanuel at hibernate.org>:
>>>>>>>>> Ok I understand better now.
>>>>>>>>> Do your tests in JBoss AS with it's decent transaction manager
>>>>>>>>> (infinispan should have a config for it)
>>>>>>>>> For unit testing, force the indexing process in hibernate to
>>>>>>>>> use a
>>>>>>>>> single thread (I ghnk it's possible ask Sanne of you don't
>>>>>>>>> know how).
>>>>>>>>>
>>>>>>>>> Exposing some configuration to infinispan makes sense. can you
>>>>>>>>> start a
>>>>>>>>> thread explainig what is configurable and which one you think
>>>>>>>>> we
>>>>>>>>> should expose to hsearch users. Ideally I would like to offer
>>>>>>>>> one or
>>>>>>>>> two defaut config scenarios and allow to fallback to a custom
>>>>>>>>> config.
>>>>>>>>>
>>>>>>>>> Emmanuel
>>>>>>>>>
>>>>>>>>> On 12 août 2009, at 11:58, Łukasz Moreń
>>>>>>>>> <lukasz.moren at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Sorry, but my wifi does not work well today. I will try to
>>>>>>>>>> explain
>>>>>>>>>> it more clear.
>>>>>>>>>>
>>>>>>>>>> I'm using DummyTransactionManager available for Infinispan.
>>>>>>>>>> It associates transaction with the calling thread.
>>>>>>>>>>
>>>>>>>>>> Steps to update index:
>>>>>>>>>>
>>>>>>>>>> 1. index writer acquires lock - begin of transaction
>>>>>>>>>>
>>>>>>>>>> 2. if it is necessary, index writer delegates new threads to
>>>>>>>>>> do
>>>>>>>>>> merge work.
>>>>>>>>>> Those merge threads do not see changes made so far from
>>>>>>>>>> begin of
>>>>>>>>>> transaction,
>>>>>>>>>> and are looking for segments which are not yet in index.
>>>>>>>>>> Changes will be visible when AD.3 is completed.
>>>>>>>>>> For tests i tried to commit transaction when merge starts
>>>>>>>>>> and then
>>>>>>>>>> everything worked well. But then i need to start it again.
>>>>>>>>>>
>>>>>>>>>> 3. index writer releases lock - transaction is commited, all
>>>>>>>>>> changes
>>>>>>>>>> made in this transaction are visible for other threads.
>>>>>>>>>>
>>>>>>>>>> Maybe using some other transaction manager could help?
>>>>>>>>>>
>>>>>>>>>> What about Infinispan cache configuration? Some configuration
>>>>>>>>>> mechanism should be exposed to the user,
>>>>>>>>>> or we can hardcoded one in InfinispanDirectoryProvider is
>>>>>>>>>> enough?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2009/8/12 Emmanuel Bernard <emmanuel at hibernate.org>
>>>>>>>>>> why?
>>>>>>>>>> Emmanuel Bernard
>>>>>>>>>> Pending
>>>>>>>>>> you there?
>>>>>>>>>> Emmanuel Bernard
>>>>>>>>>> Pending
>>>>>>>>>> Ok please describe in details what is going on. From what
>>>>>>>>>> you are
>>>>>>>>>> describing the tx cannot see all segments which looks like an
>>>>>>>>>> infinispan bug to me.
>>>>>>>>>> Pending
>>>>>>>>>>
>>>>>>>>>> As a back up you can try wo transaction and see if that works
>>>>>>>>>> Emmanuel Bernard
>>>>>>>>>> Pending
>>>>>>>>>> technically the lucene index should cope with that
>>>>>>>>>> Emmanuel Bernard
>>>>>>>>>> 11:16
>>>>>>>>>> but I like this approach less
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Let's try and chat by email IF I'm not online, I need to run
>>>>>>>>>> on some
>>>>>>>>>> errands today.
>>>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> infinispan-dev mailing list
>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Jason T. Greene
>>>>>> JBoss, a division of Red Hat
>>>>>
>>>>
>>>>
>>>> --
>>>> Jason T. Greene
>>>> JBoss, a division of Red Hat
>>>>
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> _______________________________________________
>> hibernate-dev mailing list
>> hibernate-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>
>