[infinispan-dev] Re: [hibernate-dev] Re: Pushing indexes through JGroups

Hi, I talked with Łukasz about this last wekk. Definitely, #1 and #3. #2 I don't like either. The befefit of #3 would also be that one could drop the requirement of having a shared file system (NFS, NAS, ...) #3 should be quite easy to implement. Maybe easy to get started with. --Hardy On Mon, 25 May 2009 10:55:52 +0200, Emmanuel Bernard <emmanuel(a)hibernate.org > wrote: > Hello > I am not sure this is where we should go, or at least, it depends. > here are three scenarii > > > #1 JMS replacement > If you want to use JGroups as a replacement for the JMS backend, > then I think you should write a jgroups backend. Check > org.hibernate.search.backend.impl.jms > In this case all changes are sent via JGroups to a "master". The > master could be voted by the cluster possibly dynamically but > that's not necessary for the first version. > > #2 apply indexing on all nodes > JGroups could send the work queue to all nodes and each node could > apply the change. > for various reasons I am not fan of this solution as it creates > overhead in CPU / memory usage and does nto scale very well from a > theoretical PoV. > > #3 Index copy > this is what you are describing, copying the index using JGroups > instead of my file system approach. This might have merits esp as > we could diminish network traffic using multicast but it also > require to rethink the master / slave modus operandi. > Today the master copy on a regular basis a clean index to a shared > directory > On a regular basis, the slave go and copy the clean index from the > shared directory. > In your approach, the master would send changes to the slaves and > slaves would have to apply them "right away" (on their passive > version) > > I think #1 is more interesting than #3, we probably should start > with that. #3 might be interesting too, thoughts? > > Emmanuel > > PS: refactoring is a fact of life, so feel free to do so. Just > don't break public contracts. > > On May 21, 2009, at 22:14, Łukasz Moreń wrote: > >> Hi, >> >> I have few questions that concern using JGroups to copy index >> files. I think to create sender(for master) and receiver(slave) >> directory providers. >> Sender class mainly based on existing FSMasterDirectoryProvider, >> first create local index copy and send later to slave nodes >> (or send without copying, but that may cause lower performance?). >> To avoid code redundancy it would be good to refactor a little >> FSMasterDirectoryProvider class, so then I can use copying >> functionality in new DirectoryProvider and add sending one; or >> rather I should work around it? >> >> I do not understand completely how does the multithreading access >> to index file work. Does FileChannel class assure that, when index >> is copied and new Lucene works are pushed?