[infinispan-issues] [JBoss JIRA] (ISPN-4497) Race condition in LocalLockMergingSegmentReadLocker results in file content being deleted

[Anuj Shah (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-4497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12983967#comment-12983967 ] 

Anuj Shah commented on ISPN-4497:
---------------------------------

I have a 'test' of sorts to demonstrate the problem:

{code:java}
  @Test
  public void testMultiThreaded() throws InterruptedException {
    EmbeddedCacheManager cacheManager = new DefaultCacheManager();
    Cache<Object, Object> metadata = cacheManager.getCache("metadata");
    Cache<Object, Object> chunks = cacheManager.getCache("chunks");
    Cache<Object, Integer> locks = cacheManager.getCache("locks");

    metadata.put(new FileCacheKey("indexName", "fileName"), new FileMetadata(10));

    int numThreads = 10;
    final LocalLockMergingSegmentReadLocker llmsrl = new LocalLockMergingSegmentReadLocker(locks, chunks, metadata, "indexName");

    for (int i = 0; i < numThreads; i++) {
      Thread thread = new Thread(new Runnable() {
        int counter = 0;
        @Override
        public void run() {
          try {
            while (true) {
              llmsrl.acquireReadLock("fileName");
              Thread.sleep(10);
              llmsrl.deleteOrReleaseReadLock("fileName");

              // Take a break every now and a again to try and avoid the same LocalReadLock being used constantly
              if (counter++ % 10 == 0) {
                Thread.sleep(100);
              }
            }
          } catch (InterruptedException e) {
          }
        }
      });

      thread.setDaemon(true);
      thread.start();
    }

    // Keep checking every 100ms. The file should never be deleted.
    while (true) {
      Thread.sleep(100);
      assertNotNull(metadata.get(new FileCacheKey("indexName", "fileName")));
    }
  }
{code}

For me this fails in around 8min

> Race condition in LocalLockMergingSegmentReadLocker results in file content being deleted
> -----------------------------------------------------------------------------------------
>
>                 Key: ISPN-4497
>                 URL: https://issues.jboss.org/browse/ISPN-4497
>             Project: Infinispan
>          Issue Type: Bug
>      Security Level: Public(Everyone can see) 
>          Components: Lucene Directory
>    Affects Versions: 5.2.6.Final
>            Reporter: Anuj Shah
>            Assignee: Sanne Grinovero
>
> There is a race condition in LocalLockMergingSegmentReadLocker which can lead to more calls delete on the underlying DistributedSegmentReadLocker which results in the file being removed from the caches. 
> This happens with three or more threads acquiring and releasing locks on the same file simultaneously:
> # Thread 1 (T1) acquires a lock and creates a {{LocalReadLock}}, call it L1 - the underlying lock is acquired
> # T2 starts to acquire and holds a reference to L1
> # T3 starts to acquire and holds a reference to L1
> # T1 releases - L1 at this stage only has value of 1 - so the underlying lock is released, and L1 is removed from the map
> # T2 continues - finds L1 with value 0 and acquires the underlying lock
> # T3 continues - increments L1 value to 2
> # T2 releases - creates a new {{LocalReadLock}}, L2 - this has zero value so the underlying lock is released, and L2 is removed from the map
> # T3 releases - creates a new {{LocalReadLock}}, L3 - this has zero value so the underlying lock is released, and L3 is removed from the map
> # The final step triggers a real file delete as underlying lock is released one too many times



--
This message was sent by Atlassian JIRA
(v6.2.6#6264)