[infinispan-dev] [hibernate-dev] Infinispan tx, config and multithreading

[Sanne Grinovero]

what are these "other" threads? Are you speaking about the MergeSchedulers?

2009/8/13 Łukasz Moreń <lukasz.moren at gmail.com>:
> IndexWriter processes index update and delegates some job to other
> threads and waits when they finish. These "other" threads works on
> data modified
> in IndexWriter transaction. So I think if I use transaction per
> thread, "others" would not see data modified by IndexWriter until
> commit.
>
> 2009/8/13, Emmanuel Bernard <emmanuel at hibernate.org>:
>> Ah I thought it was using multiple threads because of your mass
>> indexing. I did not know some threads were span specifically for the
>> Infinispan directory.
>>
>> On 13 août 09, at 17:34, Sanne Grinovero wrote:
>>
>>> Hi Łukasz,
>>> what is your usage of these threads? did you consider using one
>>> transaction per thread?
>>>
>>> Sanne
>>>
>>> 2009/8/13 Łukasz Moreń <lukasz.moren at gmail.com>:
>>>> Newly created threads were not associated with any transaction, so I
>>>> suppose it was a problem. Sharing transaction between threads seems
>>>> to
>>>> be a good solution.
>>>> Thanks for help!
>>>>
>>>> 2009/8/13, Jason T. Greene <jason.greene at redhat.com>:
>>>>> Correct. Also there could be read races as well, so if you are
>>>>> going to
>>>>> share a tx between threads, i would use some shared lock to
>>>>> gaurantee
>>>>> that only one thread can use it at a time. BTW this means you have
>>>>> to
>>>>> properly suspend/resume the TX via the TM API as well.
>>>>>
>>>>> Emmanuel Bernard wrote:
>>>>>> Modifying a transaction means applying muations (like SQL INSERT /
>>>>>> UPDATE / DELETE) to the transactional resource?
>>>>>>
>>>>>> On 13 août 09, at 15:07, Jason T. Greene wrote:
>>>>>>
>>>>>>> When using transactions, the context is bound to the
>>>>>>> transaction, and
>>>>>>> you can move a transaction between threads. However, you should
>>>>>>> only
>>>>>>> be modifying a transaction with one thread at a time.
>>>>>>>
>>>>>>> Emmanuel Bernard wrote:
>>>>>>>> Could it be that you are not using the same transaction between
>>>>>>>> different threads (ie you physically start different ones or
>>>>>>>> different  "Infinispan contexts")?
>>>>>>>> Infini guys, do you support transactional operation spanning
>>>>>>>> several
>>>>>>>> concurrent threads?
>>>>>>>> On 13 août 09, at 14:04, Łukasz Moreń wrote:
>>>>>>>>> I've tried with JBoss AS transaction manager and
>>>>>>>>> JBossStandaloneTM.
>>>>>>>>> The result is this same in all cases - error during merge.
>>>>>>>>>
>>>>>>>>> 2009/8/12, Emmanuel Bernard <emmanuel at hibernate.org>:
>>>>>>>>>> Ok I understand better now.
>>>>>>>>>> Do your tests in JBoss AS with it's decent transaction manager
>>>>>>>>>> (infinispan should have a config for it)
>>>>>>>>>> For unit testing, force the indexing process in hibernate to
>>>>>>>>>> use a
>>>>>>>>>> single thread (I ghnk it's possible ask Sanne of you don't
>>>>>>>>>> know how).
>>>>>>>>>>
>>>>>>>>>> Exposing some configuration to infinispan makes sense. can you
>>>>>>>>>> start a
>>>>>>>>>> thread explainig what is configurable and which one you think
>>>>>>>>>> we
>>>>>>>>>> should expose to hsearch users. Ideally I would like to offer
>>>>>>>>>> one or
>>>>>>>>>> two defaut config scenarios and allow to fallback to a custom
>>>>>>>>>> config.
>>>>>>>>>>
>>>>>>>>>> Emmanuel
>>>>>>>>>>
>>>>>>>>>> On 12 août 2009, at 11:58, Łukasz Moreń
>>>>>>>>>> <lukasz.moren at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Sorry, but my wifi does not work well today. I will try to
>>>>>>>>>>> explain
>>>>>>>>>>> it more clear.
>>>>>>>>>>>
>>>>>>>>>>> I'm using DummyTransactionManager available for Infinispan.
>>>>>>>>>>> It associates transaction with the calling thread.
>>>>>>>>>>>
>>>>>>>>>>> Steps to update index:
>>>>>>>>>>>
>>>>>>>>>>> 1. index writer acquires lock - begin of transaction
>>>>>>>>>>>
>>>>>>>>>>> 2. if it is necessary, index writer delegates new threads to
>>>>>>>>>>> do
>>>>>>>>>>> merge work.
>>>>>>>>>>> Those merge threads do not see changes made so far from
>>>>>>>>>>> begin of
>>>>>>>>>>> transaction,
>>>>>>>>>>> and are looking for segments which are not yet in index.
>>>>>>>>>>> Changes will be visible when AD.3 is completed.
>>>>>>>>>>> For tests i tried to commit transaction when merge starts
>>>>>>>>>>> and then
>>>>>>>>>>> everything worked well. But then i need to start it again.
>>>>>>>>>>>
>>>>>>>>>>> 3. index writer releases lock - transaction is commited, all
>>>>>>>>>>> changes
>>>>>>>>>>> made in this transaction are visible for other threads.
>>>>>>>>>>>
>>>>>>>>>>> Maybe using some other transaction manager could help?
>>>>>>>>>>>
>>>>>>>>>>> What about Infinispan cache configuration? Some configuration
>>>>>>>>>>> mechanism should be exposed to the user,
>>>>>>>>>>> or we can hardcoded one in InfinispanDirectoryProvider is
>>>>>>>>>>> enough?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2009/8/12 Emmanuel Bernard <emmanuel at hibernate.org>
>>>>>>>>>>> why?
>>>>>>>>>>> Emmanuel Bernard
>>>>>>>>>>> Pending
>>>>>>>>>>> you there?
>>>>>>>>>>> Emmanuel Bernard
>>>>>>>>>>> Pending
>>>>>>>>>>> Ok please describe in details what is going on. From what
>>>>>>>>>>> you are
>>>>>>>>>>> describing the tx cannot see all segments which looks like an
>>>>>>>>>>> infinispan bug to me.
>>>>>>>>>>> Pending
>>>>>>>>>>>
>>>>>>>>>>> As a back up you can try wo transaction and see if that works
>>>>>>>>>>> Emmanuel Bernard
>>>>>>>>>>> Pending
>>>>>>>>>>> technically the lucene index should cope with that
>>>>>>>>>>> Emmanuel Bernard
>>>>>>>>>>> 11:16
>>>>>>>>>>> but I like this approach less
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Let's try and chat by email IF I'm not online, I need to run
>>>>>>>>>>> on some
>>>>>>>>>>> errands today.
>>>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> infinispan-dev mailing list
>>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Jason T. Greene
>>>>>>> JBoss, a division of Red Hat
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Jason T. Greene
>>>>> JBoss, a division of Red Hat
>>>>>
>>>>
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>> _______________________________________________
>>> hibernate-dev mailing list
>>> hibernate-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>>
>>
>