11:18 p.m.
Hello,
I have been working on a Lucene Directory provider based on JBoss Cache, my starting point
was an implementation Manik had already written which pretty much worked with a few minor
tweaks. Our use case was to cluster a Lucene index being used with Hibernate Search in
our application, with the requirements that searching needed to be fast, there was no
shared file system and it was important that the index was consistent across the cluster
in a relatively short time frame.
Maniks code used a token node in the cache to implement the distributed lock. During my
testing I set up multiple cache copies with multiple threads reading/writing to each cache
copy. I was finding a lot of transactions to acquire or release this lock were timing out,
not understanding JBC well I modified the distributed lock to use JGroups
DistrubutedLockManager. This worked quite well, however the time taken to acquire/release
the lock (~100 ms for both) dwarfed the time to process the index update, lowering
throughput. Even using Hibernate Search with an async worker thread, there was still a lot
of contention for the single lock which seemed to limit the scalability of the solution. I
thinkl part of the problem was that our use of HB Search generates a lot of small units of
work (remove index entry, add index entry) and each of these UOW acquire a new IndexWriter
and new write lock on the underlying Lucene Directory implementation.
Out of curiosity, I created an alternative implementation based on the Hibernate Search
JMS clustering strategy. Inside JBoss Cache I created a queue node and each slave node in
the cluster creates a separate queue underneath where indexing work is written:
/queue/slave1/[work0, work1, work2 ....]
/slave2
/slave3
etc
In each cluster member a background thread runs continuously when it wakes up, it decides
if it is the master node or not (currently checks if it is the view coordinator, but
I'm considering changing it to use a longer lived distributed lock). If it is the
master it merges the tasks from each slave queue, and updates the JBCDirectory in one go,
it can safely do this with only local VM locking. This approach means that in all the
slave nodes they can write to their queue without needing a global lock that any other
slave or the master would be using. On the master, it can perform multiple updates in the
context of a single Lucene index writer. With a cache loader configured, work that is
written into the slave queue is persistent, so it can survive the master node crashing
with automatic fail over to a new master meaning that eventually all updates should be
applied to the index. Each work element in the queue is time stamped to allow them to be
processed in order (requires time synchronisation across the cluster) by the master. For
our workload the master/slave pattern seems to improve the throughput of the system.
Currently I'm refining the code and I have a few JBoss Cache questions which I hope
you can help me with:
1) I have noticed that under high load I get LockTimeoutExceptions writing to
/queue/slave0 when the lock owner is a transaction working on /queue/slave1 , i.e. the
same lock seems to be used for 2 unrelated nodes in the cache. I'm assuming this is a
result of the lock striping algorithm, if you could give me some insight into how this
works that would be very helpful. Bumping up the cache concurrency level from 500 to 2000
seemed to reduce this problem, however I'm not sure if it just reduces the probability
of a random event of if there is some level that will be sufficient to eliminate the
issue.
2) Is there a reason to use separate nodes for each slave queue ? Will it help with
locking, or can each slave safely insert to the same parent node in separate transactions
without interfering or blocking each other ? If I can reduce it to a single queue I thin
that would be a more elegant solution. I am setting the lockParentForChildInsertRemove to
false for the queue nodes.
3) Similarly, is there any reason why the master should/shouldn't take responsibility
for removing work nodes that have been processed ?
Thanks in advance for help, I hope to make this solution general purpose enough to be able
to contribute back to Hibernate Search and JBC teams.
Kind Regards,
Andrew