[infinispan-dev] [hibernate-dev] Distributed queries

[Emmanuel Bernard]

Neither 1 nor 2 imply *distributed* queries.

The hard parts with distributed queries (ie executed on a grid and  
recomposed) are:
  - making sure you ask all the nodes where the index is distributed  
(you can't miss a node)
  - find a way to index only a subset of the data in a given index (on  
a given node). Applying the Infinispan distribution routine to the  
InfinispanDirectory does not do that, it chunks data arbitrarily.
  - be able to rebuild a given index on a givne node (ie remember  
which element were indexed)
  - you need to find a way to distribute your data without  
duplication. If a key is indexed multiple times, then you end up with  
duplicated results that can't trivially be de-duplicated.

Happy thinking.

On 17 sept. 09, at 10:32, Sanne Grinovero wrote:

> 2009/9/17 Michael Neale <michael.neale at gmail.com>:
>> I am still not entirely sure what I am asking, but look forward for
>> your merged in changes (they are in another branch right now yes?).
>>
>> Yes I mean querying objects - I was under the impression that lucene
>> was used for the indexing of the data to service these queries?
>
> Sure, to clarify: there's work going on on two different aspects,  
> which
> complement each other in the ideal setup:
>
> 1) Be able to query a Lucene index (wherever you store that) to find  
> objects
> which are located inside Infinispan; this is about how to search  
> them and how
> to maintain the index in synch with Infinispan's content.
>
> 2) Store a Lucene index inside Infinispan, instead of, for example,  
> filesystem.
> In this case we're not concerned about what you index, the Lucene  
> interface
> is the usual one and you should be able to replace the Directory
> implementation in existing applications.
>
> So 1) is the branch you've found, and Navin is working on that, 2)  
> is not yet
> in subversion, the latest patch is attached to other thread by  
> Łukasz,
> and is to be applied
> on Hibernate Search's trunk (and depends on Infinispan).
>
>>
>> On Wed, Sep 16, 2009 at 10:32 PM, Navin Surtani  
>> <nsurtani at redhat.com> wrote:
>>>
>>> On 16 Sep 2009, at 12:25, Michael Neale wrote:
>>>
>>>> oh ok nice - could you point me at which branch to try to find some
>>>> tests to play with?
>>>
>>> If you're talking about Querying objects in Infinispan: -
>>>
>>> The eventual goal is to be able to have different configurations on
>>> how you want to index your data. Manik has given me the 'OK' to  
>>> push a
>>> simple query interface for CR1 for Monday/Tuesday.
>>>
>>> I'm kind-of pressed with getting the code working for this and also
>>> between moving house and lack of internet there I'll be a bit quiet.
>>> However, I'll get a wiki up by the end of the week about how this  
>>> all
>>> works.
>>>
>>> However if you're not then I assume you're talking about using  
>>> Lucene
>>> to index into Infinispan?
>>>
>>>>
>>>> On Wed, Sep 16, 2009 at 6:05 PM, Sanne Grinovero
>>>> <sanne.grinovero at gmail.com> wrote:
>>>>> 2009/9/16 Michael Neale <michael.neale at gmail.com>:
>>>>>> regarding indexing and queries - is the current aim to not  
>>>>>> require
>>>>>> that the index for the entire data grid exist on a single node?
>>>>>>
>>>>>> (asking as a potential user who is wrestling with lucene  
>>>>>> indexes at
>>>>>> the moment is curious).
>>>>>
>>>>> Yes the concept is to store the Lucene index itself in the grid,  
>>>>> so
>>>>> it will
>>>>> be distributed, and the segments you use most get cached locally.
>>>>> At the moment you have to select only one node to write to the  
>>>>> index,
>>>>> but all other nodes should be able to read.
>>>>> Feel free to test it as we are needing feedback.
>>>>>
>>>>>>
>>>>>> --
>>>>>> Michael D Neale
>>>>>> home: www.michaelneale.net
>>>>>> blog: michaelneale.blogspot.com
>>>>>> _______________________________________________
>>>>>> infinispan-dev mailing list
>>>>>> infinispan-dev at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>
>>>>> _______________________________________________
>>>>> infinispan-dev mailing list
>>>>> infinispan-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Michael D Neale
>>>> home: www.michaelneale.net
>>>> blog: michaelneale.blogspot.com
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>> Navin Surtani
>>>
>>> Intern Infinispan
>>> Intern JBoss Cache Searchable
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>
>>
>>
>> --
>> Michael D Neale
>> home: www.michaelneale.net
>> blog: michaelneale.blogspot.com
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>
> _______________________________________________
> hibernate-dev mailing list
> hibernate-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hibernate-dev