[infinispan-dev] ISPN 200

[Sanne Grinovero]

Hi Israel,
thank you very much, I've been very eager to read some code on this.

I've started to add some very minor comments to github; it would
greatly help if you could add some
heading javadoc to each class you created, for example what's the
purpose and role of QueryBox?

About sorting, it's good to have a starter, but I don't think we can
use reflection so massively, nor
rely on Comparator.

Sort fields aren't mandated to be stored, so it's not an option to
pre-extract them from the index either;
bottom line, it's not enough to retrieve only the matching objects
from the remote node, you have to fetch
also some index lowlevel metadata to be able to properly merge the
results using the appropriate sorting.

I just found this code which might be very useful as inspiration, it's
designed to merge results
from searches performed in different threads, but if you re-think the
executor as remote node executors,
you should be set:
org.apache.lucene.search.ParallelMultiSearcher.search(Weight, Filter, int, Sort)

So you have each node perform a Query on his local index, but then the
final sort is performed on the requesting
node again processing the same TopFieldDocs which each remote node created.
When returning the matching value objects, return the TopFieldDocs
too. Actually I'd say it should be fine to retrieve
the values at a later (second phase) step, as it's quite possible
people want to retrieve only the "top 100" or similar queries.

Finally, it would be great to see some test. We don't need extreme
code coverage at this point, but it would greatly help to
have at least a two nodes scenario with a simple sort being performed,
it's also nice for people
wanting to look into it, as an example of how to use your code and
what are the entry points.

thank you, great job.
Feel free to ask anything, by email or on IRC.

Sanne


2011/1/7 Israel Lacerra <israeldl at gmail.com>:
> I've pushed some code in https://github.com/israeldl/infinispan/
>
> This is not the final version. Is just to you take a look and tell me what
> you think. This commit have some problems (I'm fighting with Git yet).
> Ignore the changes in demos/*.
>
> To use the distributed query, we have to call
> QueryFactory.getDistributedQuery.
>
> There must be a lot things that I have to improve in the code, but I hope it
> is not in a totally wrong way!
>
> Some thoughts:
> - In the QueryBox, I made a simple eviction policy, and probably not the
> best. I maintain up to 300 lazyIterators. What you think is a good policy??
> - To make the sort, I had to force the fields used in that sort to implement
> Comparable. I could not use comparator methods behind the Sort class of
> Lucene. One option would be to create another way to user create the sort
> method and then from there we would create the lucene Sort. What you think?
> - I created a ClusteredQueryFacade on org.infinispan.commands, because I
> initialize this command in CommandsFactoryImpl.initializeReplicableCommand.
> There is a better way??
> - The sort is made in a naive way. If we have too many nodes, probably we
> will have problems. Should I improve this? In a real case, the number of
> nodes will be too big?
>
>
> thanks!
>
> Israel
>
> On Tue, Jan 4, 2011 at 2:49 PM, Manik Surtani <manik at jboss.org> wrote:
>>
>> On 30 Dec 2010, at 18:34, Israel Lacerra wrote:
>>
>> > Manik,
>> >
>> > I'm gonna need a few more days. I'll probably send you the code on the
>> > middle of the next week (about jan/6).
>> >
>> > Guys,
>> >
>> > About the way I'll send the code, I think the best is to make a fork on
>> > github and then I push my changes on this fork and send you the url of the
>> > fork. Right?
>>
>> Yes.  Pls post the link to this mail list so others can take a look as
>> well.
>>
>> Cheers
>> Manik
>>
>> --
>> Manik Surtani
>> manik at jboss.org
>> twitter.com/maniksurtani
>>
>> Lead, Infinispan
>> http://www.infinispan.org
>>
>>
>>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>