[jbosscache-dev] JBoss Cache Lucene Directory

[Andrew Duckworth]

Hello,

I have been working on a Lucene Directory provider based on JBoss Cache, my starting point was an implementation Manik had already written which pretty much worked with a few minor tweaks. Our use case was to  cluster a Lucene index being used with Hibernate Search in our application, with the requirements that searching needed to be fast, there was no shared file system and it was important that the index was consistent across the cluster in a relatively short time frame. 

Maniks code used a token node in the cache to implement the distributed lock. During my testing I set up multiple cache copies with multiple threads reading/writing to each cache copy. I was finding a lot of transactions to acquire or release this lock were timing out, not understanding JBC well  I modified the distributed lock to use JGroups DistrubutedLockManager. This worked quite well, however the time taken to acquire/release the lock (~100 ms for both) dwarfed the time to process the index update, lowering throughput. Even using Hibernate Search with an async worker thread, there was still a lot of contention for the single lock which seemed to limit the scalability of the solution. I thinkl part of the problem was that our use of HB Search generates a lot of small units of work (remove index entry, add index entry) and each of these UOW acquire a new IndexWriter and new write lock on the underlying Lucene Directory implementation. 


 Out of curiosity, I created an alternative implementation based on the Hibernate Search JMS clustering strategy. Inside JBoss Cache I created a queue node and each slave node in the cluster creates a separate queue underneath where indexing work is written:

  /queue/slave1/[work0, work1, work2 ....]
             /slave2
             /slave3

etc

In each cluster member a background thread runs continuously when it wakes up, it decides if it is the master node or not (currently checks if it is the view coordinator, but I'm considering changing it to use a  longer lived distributed lock). If it is the master it merges the tasks from each slave queue, and updates the JBCDirectory in one go, it can safely do this with only local VM  locking. This approach means that in all the slave nodes they can write to their queue without needing a global lock that any other slave or the master would be using. On the master, it can perform multiple updates in the context of a single Lucene index writer. With a cache loader configured, work that is written into the slave queue is persistent, so it can survive the master node crashing with automatic fail over to a new master meaning that eventually all updates should be applied to the index. Each work element in the queue is time stamped to allow them to be processed in order (requires time synchronisation across the cluster) by the master. For our workload the master/slave pattern seems to improve the throughput of the system.


Currently I'm refining the code and I have a few JBoss Cache questions which I hope you can help me with:

1) I have noticed that under high load I get LockTimeoutExceptions writing to /queue/slave0 when the lock owner is a transaction working on /queue/slave1 , i.e. the same lock seems to be used for 2 unrelated nodes in the cache. I'm assuming this is a result of the lock striping algorithm, if you could give me some insight into how this works that would be very helpful. Bumping up the cache concurrency level from 500 to 2000 seemed to reduce this problem, however I'm not sure if it just reduces the probability of a random event of if there is some level that will be sufficient to eliminate the issue.

2) Is there a reason to use separate nodes for each slave queue ? Will it help with locking, or can each slave safely insert to the same parent node in separate transactions without interfering or blocking each other ? If I can reduce it to a single queue I thin that would be a more elegant solution. I am setting the lockParentForChildInsertRemove to false for the queue nodes.

3) Similarly, is there any reason why the master should/shouldn't take responsibility for removing work nodes that have been processed ?

Thanks in advance for help, I hope to make this solution general purpose enough to be able to contribute back to Hibernate Search and JBC teams.

Kind Regards,

Andrew