[infinispan-dev] JGroupsDistSync and ISPN-83

Tue May 17 09:17:26 EDT 2011

In conclusion this is a what we know so far regarding ISPN-83:

One of the reader threads (gets R lock on JGroupsDistSync) fails to 
complete (release R lock) in 10 sec stateTransferTimeout and therefore 
prevents state receiver from properly installing the state thus throwing 
ISPN00095 error code. You can observe this phenomena by running 
StateTransferCacheLoaderFunctionalTest

ISPN00095 is what the users where reporting after they have removed 
FLUSH from the stack

Unfortunately I would say that since ISPN00095 is a symptom of a problem 
but not the cause I have to continue to dig to see why it is happening 
to begin with an I will continue to do so as soon as the master 
testsuite is stabilized.

Let me know if you have some comments and suggestions!

Regards,
Vladimir

On 11-05-17 8:27 AM, Vladimir Blagojevic wrote:
> Apparently I did not understand semantics of RWL. I thought that writer
> can obtain a write lock even though read lock has not been released. But
> no, after all read locks have been released only then a writer can
> obtain a lock.
>
>
>
> On 11-05-16 11:18 PM, Sanne Grinovero wrote:
>> Same result here - where you expecting something different?
>>
>> Cheers,
>> Sanne
>>
>> 2011/5/16 Erik Salter<esalter at bnivideo.com>:
>>> EDIT:  Originally posted in the wrong thread.  I blame Outlook.
>>>
>>> I guess I qualify as "others", since I'm looking at similar issues.
>>>
>>> Got read lock
>>> java.util.concurrent.TimeoutException: Thread-1 could not obtain exclusive processing lock after 3 seconds.  Locks in question are java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock at 16aa37a6[Read locks = 1] and java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock at 12b7eea[Unlocked]
>>>
>>> This is on the latest master.
>>>
>>> Erik
>>>
>>> -----Original Message-----
>>> From: infinispan-dev-bounces at lists.jboss.org [mailto:infinispan-dev-bounces at lists.jboss.org] On Behalf Of Vladimir Blagojevic
>>> Sent: Monday, May 16, 2011 4:25 PM
>>> To: infinispan -Dev List
>>> Cc: Manik Surtani
>>> Subject: Re: [infinispan-dev] JGroupsDistSync and ISPN-83
>>>
>>> Manik (and others),
>>>
>>> Can you run this code on your laptops and let me know what happened!
>>>
>>> Vladimir
>>>
>>>      public static void main (String [] arg) throws Exception {
>>>         final JGroupsDistSync ds = new JGroupsDistSync();
>>>
>>>         ds.acquireProcessingLock(false, 3, TimeUnit.SECONDS);
>>>         System.out.println("Got read lock");
>>>
>>>         Thread t = new Thread(){
>>>           public void run (){
>>>              try {
>>>               ds.acquireProcessingLock(true, 3, TimeUnit.SECONDS);
>>>               System.out.println("Got write lock");
>>>            } catch (TimeoutException e) {
>>>               System.out.println(e);
>>>            }
>>>           }
>>>         };
>>>         t.start();
>>>      }
>>>
>>>
>>>
>>> On 11-05-13 4:53 PM, Manik Surtani wrote:
>>>> Yes, please have a look. If we are relying on lock upgrades then that's really bad. I am aware of the inability to (safely) upgrade a RWL and I'm pretty sure we don't try, but the dist sync codebase has evolved a lot and could do with some careful analysis.
>>>>
>>>> Sent from my mobile phone
>>>>
>>>> On 12 May 2011, at 09:24, Vladimir Blagojevic<vblagoje at redhat.com>     wrote:
>>>>
>>>>> On 11-05-11 11:23 AM, Dan Berindei wrote:
>>>>>> If ReentrantReadWriteLock would allow upgrades then you would get a
>>>>>> deadlock when two threads both hold the read lock and try to upgrade
>>>>>> to a write lock at the same time.
>>>>>> There's always a trade-off...
>>>>>>
>>>>>> I'm not familiar with the code, but are you sure the read lock is
>>>>>> being held by the same thread that is trying to acquire the write
>>>>>> lock?
>>>>>>
>>>>> Not sure and it sounds counter intuitive that a thread holding a read
>>>>> lock from cluster invocation is doing state generation for state
>>>>> transfer as well. But this lock business is fishy and I plan to get
>>>>> to the bottom of it...
>>>>>
>>>>> _______________________________________________
>>>>> infinispan-dev mailing list
>>>>> infinispan-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>> The information contained in this message is legally privileged and confidential, and is intended for the individual or entity to whom it is addressed (or their designee). If this message is read by anyone other than the intended recipient, please be advised that distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please notify the sender immediately and delete or destroy all copies of this message.
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev