JBoss Cache Lucene Directory
by Andrew Duckworth
Hello,
I have been working on a Lucene Directory provider based on JBoss Cache, my starting point was an implementation Manik had already written which pretty much worked with a few minor tweaks. Our use case was to cluster a Lucene index being used with Hibernate Search in our application, with the requirements that searching needed to be fast, there was no shared file system and it was important that the index was consistent across the cluster in a relatively short time frame.
Maniks code used a token node in the cache to implement the distributed lock. During my testing I set up multiple cache copies with multiple threads reading/writing to each cache copy. I was finding a lot of transactions to acquire or release this lock were timing out, not understanding JBC well I modified the distributed lock to use JGroups DistrubutedLockManager. This worked quite well, however the time taken to acquire/release the lock (~100 ms for both) dwarfed the time to process the index update, lowering throughput. Even using Hibernate Search with an async worker thread, there was still a lot of contention for the single lock which seemed to limit the scalability of the solution. I thinkl part of the problem was that our use of HB Search generates a lot of small units of work (remove index entry, add index entry) and each of these UOW acquire a new IndexWriter and new write lock on the underlying Lucene Directory implementation.
Out of curiosity, I created an alternative implementation based on the Hibernate Search JMS clustering strategy. Inside JBoss Cache I created a queue node and each slave node in the cluster creates a separate queue underneath where indexing work is written:
/queue/slave1/[work0, work1, work2 ....]
/slave2
/slave3
etc
In each cluster member a background thread runs continuously when it wakes up, it decides if it is the master node or not (currently checks if it is the view coordinator, but I'm considering changing it to use a longer lived distributed lock). If it is the master it merges the tasks from each slave queue, and updates the JBCDirectory in one go, it can safely do this with only local VM locking. This approach means that in all the slave nodes they can write to their queue without needing a global lock that any other slave or the master would be using. On the master, it can perform multiple updates in the context of a single Lucene index writer. With a cache loader configured, work that is written into the slave queue is persistent, so it can survive the master node crashing with automatic fail over to a new master meaning that eventually all updates should be applied to the index. Each work element in the queue is time stamped to allow them to be processed in order (requires time synchronisation across the cluster) by the master. For our workload the master/slave pattern seems to improve the throughput of the system.
Currently I'm refining the code and I have a few JBoss Cache questions which I hope you can help me with:
1) I have noticed that under high load I get LockTimeoutExceptions writing to /queue/slave0 when the lock owner is a transaction working on /queue/slave1 , i.e. the same lock seems to be used for 2 unrelated nodes in the cache. I'm assuming this is a result of the lock striping algorithm, if you could give me some insight into how this works that would be very helpful. Bumping up the cache concurrency level from 500 to 2000 seemed to reduce this problem, however I'm not sure if it just reduces the probability of a random event of if there is some level that will be sufficient to eliminate the issue.
2) Is there a reason to use separate nodes for each slave queue ? Will it help with locking, or can each slave safely insert to the same parent node in separate transactions without interfering or blocking each other ? If I can reduce it to a single queue I thin that would be a more elegant solution. I am setting the lockParentForChildInsertRemove to false for the queue nodes.
3) Similarly, is there any reason why the master should/shouldn't take responsibility for removing work nodes that have been processed ?
Thanks in advance for help, I hope to make this solution general purpose enough to be able to contribute back to Hibernate Search and JBC teams.
Kind Regards,
Andrew
15 years, 7 months
- 3
- 5
Re: [jbosscache-dev] Ease of extension of Cache and Node
by Brian Stansberry
Yep, that's it exactly.
Logically the same thing could be done for Node, although I personally
don't have a need for it.
Mircea Markus wrote:
> If I understood it correctly, what you need is a default cache(node)
> delegate: a class that extends Cache interface, wraps a Cache instance
> and delegate all calls to it.
> Whenever you want to extend it, all you would have to do is only
> override the methods you're interested in, so that if new methods are
> added your code will still be compilable.
> Am I right?
>
> Brian Stansberry wrote:
>> I've got a couple places where I've actually implemented the Cache
>> interface in order add a bit of specialized behavior (one is in a
>> test, one is to disable calls to stop/destroy by anyone but the AS
>> CacheManager that created the cache). Simple stuff, just pass the real
>> cache to the constructor and delegate to it.
>>
>> Problem is the Cache API changes via addition of new methods, which
>> breaks these things. Can the next release of JBC (not asking for it in
>> 3.1.0) include a class that does this?
>>
>> I include Node in $subject since the same conceptual issue applies
>> there as well.
>>
>> Same request would apply to horizon as well.
>>
>
--
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberry(a)redhat.com
15 years, 8 months
- 1
- 0
Ease of extension of Cache and Node
by Brian Stansberry
I've got a couple places where I've actually implemented the Cache
interface in order add a bit of specialized behavior (one is in a test,
one is to disable calls to stop/destroy by anyone but the AS
CacheManager that created the cache). Simple stuff, just pass the real
cache to the constructor and delegate to it.
Problem is the Cache API changes via addition of new methods, which
breaks these things. Can the next release of JBC (not asking for it in
3.1.0) include a class that does this?
I include Node in $subject since the same conceptual issue applies there
as well.
Same request would apply to horizon as well.
--
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberry(a)redhat.com
15 years, 8 months
- 1
- 0
- ← Newer
- 1
- Older →