[infinispan-dev] [hibernate-dev] Distributed queries

[Michael Neale]

I think you just stuck a pin in the bubble that normally says "magic
happens here" ;)

How much of this did you tackle regarding hibernate search that could
be applied here?

(you final point re duplication may have some "flexibility" I think ?)

On Fri, Sep 18, 2009 at 6:18 PM, Emmanuel Bernard
<emmanuel at hibernate.org> wrote:
> Neither 1 nor 2 imply *distributed* queries.
>
> The hard parts with distributed queries (ie executed on a grid and
> recomposed) are:
>  - making sure you ask all the nodes where the index is distributed
> (you can't miss a node)
>  - find a way to index only a subset of the data in a given index (on
> a given node). Applying the Infinispan distribution routine to the
> InfinispanDirectory does not do that, it chunks data arbitrarily.
>  - be able to rebuild a given index on a givne node (ie remember
> which element were indexed)
>  - you need to find a way to distribute your data without
> duplication. If a key is indexed multiple times, then you end up with
> duplicated results that can't trivially be de-duplicated.
>
> Happy thinking.
>
> On 17 sept. 09, at 10:32, Sanne Grinovero wrote:
>
>> 2009/9/17 Michael Neale <michael.neale at gmail.com>:
>>> I am still not entirely sure what I am asking, but look forward for
>>> your merged in changes (they are in another branch right now yes?).
>>>
>>> Yes I mean querying objects - I was under the impression that lucene
>>> was used for the indexing of the data to service these queries?
>>
>> Sure, to clarify: there's work going on on two different aspects,
>> which
>> complement each other in the ideal setup:
>>
>> 1) Be able to query a Lucene index (wherever you store that) to find
>> objects
>> which are located inside Infinispan; this is about how to search
>> them and how
>> to maintain the index in synch with Infinispan's content.
>>
>> 2) Store a Lucene index inside Infinispan, instead of, for example,
>> filesystem.
>> In this case we're not concerned about what you index, the Lucene
>> interface
>> is the usual one and you should be able to replace the Directory
>> implementation in existing applications.
>>
>> So 1) is the branch you've found, and Navin is working on that, 2)
>> is not yet
>> in subversion, the latest patch is attached to other thread by
>> Łukasz,
>> and is to be applied
>> on Hibernate Search's trunk (and depends on Infinispan).
>>
>>>
>>> On Wed, Sep 16, 2009 at 10:32 PM, Navin Surtani
>>> <nsurtani at redhat.com> wrote:
>>>>
>>>> On 16 Sep 2009, at 12:25, Michael Neale wrote:
>>>>
>>>>> oh ok nice - could you point me at which branch to try to find some
>>>>> tests to play with?
>>>>
>>>> If you're talking about Querying objects in Infinispan: -
>>>>
>>>> The eventual goal is to be able to have different configurations on
>>>> how you want to index your data. Manik has given me the 'OK' to
>>>> push a
>>>> simple query interface for CR1 for Monday/Tuesday.
>>>>
>>>> I'm kind-of pressed with getting the code working for this and also
>>>> between moving house and lack of internet there I'll be a bit quiet.
>>>> However, I'll get a wiki up by the end of the week about how this
>>>> all
>>>> works.
>>>>
>>>> However if you're not then I assume you're talking about using
>>>> Lucene
>>>> to index into Infinispan?
>>>>
>>>>>
>>>>> On Wed, Sep 16, 2009 at 6:05 PM, Sanne Grinovero
>>>>> <sanne.grinovero at gmail.com> wrote:
>>>>>> 2009/9/16 Michael Neale <michael.neale at gmail.com>:
>>>>>>> regarding indexing and queries - is the current aim to not
>>>>>>> require
>>>>>>> that the index for the entire data grid exist on a single node?
>>>>>>>
>>>>>>> (asking as a potential user who is wrestling with lucene
>>>>>>> indexes at
>>>>>>> the moment is curious).
>>>>>>
>>>>>> Yes the concept is to store the Lucene index itself in the grid,
>>>>>> so
>>>>>> it will
>>>>>> be distributed, and the segments you use most get cached locally.
>>>>>> At the moment you have to select only one node to write to the
>>>>>> index,
>>>>>> but all other nodes should be able to read.
>>>>>> Feel free to test it as we are needing feedback.
>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Michael D Neale
>>>>>>> home: www.michaelneale.net
>>>>>>> blog: michaelneale.blogspot.com
>>>>>>> _______________________________________________
>>>>>>> infinispan-dev mailing list
>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>
>>>>>> _______________________________________________
>>>>>> infinispan-dev mailing list
>>>>>> infinispan-dev at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Michael D Neale
>>>>> home: www.michaelneale.net
>>>>> blog: michaelneale.blogspot.com
>>>>> _______________________________________________
>>>>> infinispan-dev mailing list
>>>>> infinispan-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>
>>>> Navin Surtani
>>>>
>>>> Intern Infinispan
>>>> Intern JBoss Cache Searchable
>>>>
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>
>>>
>>>
>>>
>>> --
>>> Michael D Neale
>>> home: www.michaelneale.net
>>> blog: michaelneale.blogspot.com
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>
>> _______________________________________________
>> hibernate-dev mailing list
>> hibernate-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev



-- 
Michael D Neale
home: www.michaelneale.net
blog: michaelneale.blogspot.com