[hibernate-dev] Re: Pushing indexes through JGroups

[Hardy Ferentschik]

Hi,

I talked with Łukasz about this last wekk. Definitely, #1 and #3.
#2 I don't like either.

The befefit of #3 would also be that one could drop the requirement of
having a shared file system (NFS, NAS, ...) #3 should be quite easy to
implement. Maybe easy to get started with.

--Hardy

On Mon, 25 May 2009 10:55:52 +0200, Emmanuel Bernard  
<emmanuel at hibernate.org> wrote:

> Hello
> I am not sure this is where we should go, or at least, it depends. here  
> are three scenarii
>
>
> #1 JMS replacement
> If you want to use JGroups as a replacement for the JMS backend, then I  
> think you should write a jgroups backend. Check  
> org.hibernate.search.backend.impl.jms
> In this case all changes are sent via JGroups to a "master". The master  
> could be voted by the cluster possibly dynamically but that's not  
> necessary for the first version.
>
> #2 apply indexing on all nodes
> JGroups could send the work queue to all nodes and each node could apply  
> the change.
> for various reasons I am not fan of this solution as it creates overhead  
> in CPU / memory usage and does nto scale very well from a theoretical  
> PoV.
>
> #3 Index copy
> this is what you are describing, copying the index using JGroups instead  
> of my file system approach. This might have merits esp as we could  
> diminish network traffic using multicast but it also require to rethink  
> the master / slave modus operandi.
> Today the master copy on a regular basis a clean index to a shared  
> directory
> On a regular basis, the slave go and copy the clean index from the  
> shared directory.
> In your approach, the master would send changes to the slaves and slaves  
> would have to apply them "right away" (on their passive version)
>
> I think #1 is more interesting than #3, we probably should start with  
> that. #3 might be interesting too, thoughts?
>
> Emmanuel
>
> PS: refactoring is a fact of life, so feel free to do so. Just don't  
> break public contracts.
>
> On  May 21, 2009, at 22:14, Łukasz Moreń wrote:
>
>> Hi,
>>
>> I have few questions that concern using JGroups to copy index files. I  
>> think to create sender(for master) and receiver(slave) directory  
>> providers.
>> Sender class mainly based on existing FSMasterDirectoryProvider, first  
>> create local index copy and send later to slave nodes
>> (or send without copying, but that may cause lower performance?).
>> To avoid code redundancy it would be good to refactor a little  
>> FSMasterDirectoryProvider class, so then I can use copying  
>> functionality in new DirectoryProvider and add sending one; or rather I  
>> should work around it?
>>
>> I do not understand completely how does the multithreading access to  
>> index file work. Does FileChannel class assure that, when index is  
>> copied and new Lucene works are pushed?