Manik, What you mean by: " * The calling node returns a CacheQuery impl that lazily fetches and collates results from the cluster." (JIRA) Is enough if each node returns a list of keys and then, we lazily get the values using the keys? Or the process has to be more lazy yet?

Ok, Manik! I've already have a code working, but not too "lazy" (and without the "sort" part). I get the keys on all nodes, and then I use them. So I have a EagerIterator, but not the LazyIterator. :/ I'll think about it... thanks On Wed, Aug 25, 2010 at 10:23 AM, Manik Surtani <manik(a)jboss.org&gt; wrote: On 24 Aug 2010, at 17:20, Israel Lacerra wrote: > Manik, > > What you mean by: > " * The calling node returns a CacheQuery impl that lazily fetches > and collates results from the cluster." (JIRA) > > Is enough if each node returns a list of keys and then, we lazily get the values using the keys? Or the process has to be more lazy yet? I think it can be "more lazy" as you said. :) -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Hi Israel, Just a quick question to your issue. What do you mean by you do not have the LazyIterator? I suppose I'm not really understanding what your issue is. Is it just that you don't want the lazy version or you don't know how to use it? :S On 25/08/10 14:39, Israel Lacerra wrote: > Ok, Manik! > > I've already have a code working, but not too "lazy" (and without the > "sort" part). I get the keys on all nodes, and then I use them. So I > have a EagerIterator, but not the LazyIterator. :/ > > I'll think about it... > > thanks > > On Wed, Aug 25, 2010 at 10:23 AM, Manik Surtani <manik(a)jboss.org > <mailto:manik@jboss.org>> wrote: > > > On 24 Aug 2010, at 17:20, Israel Lacerra wrote: > > > Manik, > > > > What you mean by: > > " * The calling node returns a CacheQuery impl that lazily fetches > > and collates results from the cluster." (JIRA) > > > > Is enough if each node returns a list of keys and then, we lazily > get the values using the keys? Or the process has to be more lazy yet? > > I think it can be "more lazy" as you said. :) > -- > Manik Surtani > manik(a)jboss.org <mailto:manik@jboss.org> > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org <mailto: infinispan-dev(a)lists.jboss.org&gt; > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Navin Surtani Intern Infinispan _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Hi guys, I'm still thinking about it, but I don't have a really good idea about the lazy iterator yet. The only way (that I see) I could make it more lazily is: - Broadcast the query. - Every node creates a QueryHits instance with the query and keep it in a simple little cache (array, hash, etc) - A "state" of the query is created and every lazyIterator.next() must send a command to a node and get the next hit (the next key). - After a certain time, the instances of queryHits "dies". It seems to me that this is not too efficient. But I don't have any other ideas. Do you have any suggestions about it? thanks! Israel AnyoneOn Tue, Aug 31, 2010 at 9:14 AM, Israel Lacerra <israeldl(a)gmail.com&gt; wrote: Hi Navin, I'm trying to do ISPN-200 (https://jira.jboss.org/browse/ISPN-200) and I don't know how to implement a good lazy iterator in a "distributed way". (Sorry... my english is not soo good. If you don't understand again, please just ask again! :) Israel On Tue, Aug 31, 2010 at 6:27 AM, Navin Surtani <nsurtani(a)redhat.com&gt; wrote: Hi Israel, Just a quick question to your issue. What do you mean by you do not have the LazyIterator? I suppose I'm not really understanding what your issue is. Is it just that you don't want the lazy version or you don't know how to use it? :S On 25/08/10 14:39, Israel Lacerra wrote: > Ok, Manik! > > I've already have a code working, but not too "lazy" (and without the > "sort" part). I get the keys on all nodes, and then I use them. So I > have a EagerIterator, but not the LazyIterator. :/ > > I'll think about it... > > thanks > > On Wed, Aug 25, 2010 at 10:23 AM, Manik Surtani <manik(a)jboss.org > <mailto:manik@jboss.org>> wrote: > > > On 24 Aug 2010, at 17:20, Israel Lacerra wrote: > > > Manik, > > > > What you mean by: > > " * The calling node returns a CacheQuery impl that lazily fetches > > and collates results from the cluster." (JIRA) > > > > Is enough if each node returns a list of keys and then, we lazily > get the values using the keys? Or the process has to be more lazy yet? > > I think it can be "more lazy" as you said. :) > -- > Manik Surtani > manik(a)jboss.org <mailto:manik@jboss.org> > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org <mailto:infinispan-dev@lists.jboss.org> > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Navin Surtani Intern Infinispan _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

You give up any hope of implementing sorting (explicit or using relevance). On 15 sept. 2010, at 12:16, Manik Surtani wrote: > How about something like: > > - Broadcast the query > - Every node creates the QueryHits inst, runs the query and collects > results. Starts streaming the results back immediately. > - The lazy iterator returns to the user immediately, and maintains an > internal cache of results coming in from N remote QueryHits instances > - iterator.next() blocks until this cache has available entries to return. > - on an implementation level, the GetHItsCommand (or something like > that) could return with a single hit, or N hits, with a flag of > whether more hits are available or not. > > Does that help? > > Manik > > > On 14 Sep 2010, at 20:46, Israel Lacerra wrote: > >> Hi guys, >> >> I'm still thinking about it, but I don't have a really good idea >> about the lazy iterator yet. The only way (that I see) I could make >> it more lazily is: >> >> - Broadcast the query. >> - Every node creates a QueryHits instance with the query and keep it >> in a simple little cache (array, hash, etc) >> - A "state" of the query is created and every lazyIterator.next() >> must send a command to a node and get the next hit (the next key). >> - After a certain time, the instances of queryHits "dies". >> >> It seems to me that this is not too efficient. But I don't have any >> other ideas. >> >> Do you have any suggestions about it? >> >> thanks! >> Israel >>

You give up any hope of implementing sorting (explicit or using relevance).

Isn't that an issue that comes with lots of hits in a lazyIterator anyway? I mean, if you're only going to load up a 'few hits at a time' then you can in theory only sort the few hits that you have loaded up anyway right? :S On 15/09/10 12:01, Emmanuel Bernard wrote: > You give up any hope of implementing sorting (explicit or using relevance). > > On 15 sept. 2010, at 12:16, Manik Surtani wrote: > >> How about something like: >> >> - Broadcast the query >> - Every node creates the QueryHits inst, runs the query and collects >> results. Starts streaming the results back immediately. >> - The lazy iterator returns to the user immediately, and maintains an >> internal cache of results coming in from N remote QueryHits instances >> - iterator.next() blocks until this cache has available entries to return. >> - on an implementation level, the GetHItsCommand (or something like >> that) could return with a single hit, or N hits, with a flag of >> whether more hits are available or not. >> >> Does that help? >> >> Manik >> >> >> On 14 Sep 2010, at 20:46, Israel Lacerra wrote: >> >>> Hi guys, >>> >>> I'm still thinking about it, but I don't have a really good idea >>> about the lazy iterator yet. The only way (that I see) I could make >>> it more lazily is: >>> >>> - Broadcast the query. >>> - Every node creates a QueryHits instance with the query and keep it >>> in a simple little cache (array, hash, etc) >>> - A "state" of the query is created and every lazyIterator.next() >>> must send a command to a node and get the next hit (the next key). >>> - After a certain time, the instances of queryHits "dies". >>> >>> It seems to me that this is not too efficient. But I don't have any >>> other ideas. >>> >>> Do you have any suggestions about it? >>> >>> thanks! >>> Israel >>> -- Navin Surtani Intern Infinispan _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

I think if we allways block iterator.next() until we have at least a value

>You give up any hope of implementing sorting (explicit or using relevance). I think if we allways block iterator.next() until we have at least a value of each node , we can apply sort in this values (since every node already made this). That's what you said, Navin? On Wed, Sep 15, 2010 at 8:16 AM, Navin Surtani <nsurtani(a)redhat.com&gt;wrote: > Isn't that an issue that comes with lots of hits in a lazyIterator anyway? > > I mean, if you're only going to load up a 'few hits at a time' then you > can in theory only sort the few hits that you have loaded up anyway > right? :S > > On 15/09/10 12:01, Emmanuel Bernard wrote: > > You give up any hope of implementing sorting (explicit or using > relevance). > > > > On 15 sept. 2010, at 12:16, Manik Surtani wrote: > > > >> How about something like: > >> > >> - Broadcast the query > >> - Every node creates the QueryHits inst, runs the query and collects > >> results. Starts streaming the results back immediately. > >> - The lazy iterator returns to the user immediately, and maintains an > >> internal cache of results coming in from N remote QueryHits instances > >> - iterator.next() blocks until this cache has available entries to > return. > >> - on an implementation level, the GetHItsCommand (or something like > >> that) could return with a single hit, or N hits, with a flag of > >> whether more hits are available or not. > >> > >> Does that help? > >> > >> Manik > >> > >> > >> On 14 Sep 2010, at 20:46, Israel Lacerra wrote: > >> > >>> Hi guys, > >>> > >>> I'm still thinking about it, but I don't have a really good idea > >>> about the lazy iterator yet. The only way (that I see) I could make > >>> it more lazily is: > >>> > >>> - Broadcast the query. > >>> - Every node creates a QueryHits instance with the query and keep it > >>> in a simple little cache (array, hash, etc) > >>> - A "state" of the query is created and every lazyIterator.next() > >>> must send a command to a node and get the next hit (the next key). > >>> - After a certain time, the instances of queryHits "dies". > >>> > >>> It seems to me that this is not too efficient. But I don't have any > >>> other ideas. > >>> > >>> Do you have any suggestions about it? > >>> > >>> thanks! > >>> Israel > >>> > > > -- > Navin Surtani > Intern Infinispan > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev >

>I think if we allways block iterator.next() until we have at least a value of each node , we can apply sort in this values (since every node already made this). I mean, the idea is to make a "merge" of the results... and the next() blocks until we have some results already sorted.

Emmanuel, do you think this can work?

>I think if we allways block iterator.next() until we have at least a value of each node , we can apply sort in this values (since every node already made this). I mean, the idea is to make a "merge" of the results... and the next() blocks until we have some results already sorted. Emmanuel, do you think this can work? On Wed, Sep 15, 2010 at 8:54 AM, Israel Lacerra <israeldl(a)gmail.com&gt; wrote: >You give up any hope of implementing sorting (explicit or using relevance). I think if we allways block iterator.next() until we have at least a value of each node , we can apply sort in this values (since every node already made this). That's what you said, Navin? On Wed, Sep 15, 2010 at 8:16 AM, Navin Surtani <nsurtani(a)redhat.com&gt; wrote: Isn't that an issue that comes with lots of hits in a lazyIterator anyway? I mean, if you're only going to load up a 'few hits at a time' then you can in theory only sort the few hits that you have loaded up anyway right? :S On 15/09/10 12:01, Emmanuel Bernard wrote: > You give up any hope of implementing sorting (explicit or using relevance). > > On 15 sept. 2010, at 12:16, Manik Surtani wrote: > >> How about something like: >> >> - Broadcast the query >> - Every node creates the QueryHits inst, runs the query and collects >> results. Starts streaming the results back immediately. >> - The lazy iterator returns to the user immediately, and maintains an >> internal cache of results coming in from N remote QueryHits instances >> - iterator.next() blocks until this cache has available entries to return. >> - on an implementation level, the GetHItsCommand (or something like >> that) could return with a single hit, or N hits, with a flag of >> whether more hits are available or not. >> >> Does that help? >> >> Manik >> >> >> On 14 Sep 2010, at 20:46, Israel Lacerra wrote: >> >>> Hi guys, >>> >>> I'm still thinking about it, but I don't have a really good idea >>> about the lazy iterator yet. The only way (that I see) I could make >>> it more lazily is: >>> >>> - Broadcast the query. >>> - Every node creates a QueryHits instance with the query and keep it >>> in a simple little cache (array, hash, etc) >>> - A "state" of the query is created and every lazyIterator.next() >>> must send a command to a node and get the next hit (the next key). >>> - After a certain time, the instances of queryHits "dies". >>> >>> It seems to me that this is not too efficient. But I don't have any >>> other ideas. >>> >>> Do you have any suggestions about it? >>> >>> thanks! >>> Israel >>> -- Navin Surtani Intern Infinispan _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Also sending the query in broadcast is nice for a first implementation, but this means you can scale the number of searched items but we can't scale the number of queries, this should be designed in such a way to make it possible in a future improvement to send the query only to a subset o the nodes.

2010/9/15 Navin Surtani <nsurtani(a)redhat.com&gt;: > >> >> Also sending the query in broadcast is nice for a first >> implementation, but this means you can scale the number of searched >> items but we can't scale the number of queries, this should be >> designed in such a way to make it possible in a future improvement to >> send the query only to a subset o the nodes. >> > > Well I don't know of any other way to run the query to be honest. If you > imagine that you have several nodes running ISPN in DIST mode, and each > node has it's own local index - we don't necessarily know where all of > the objects are. Each of them has got a share of all the objects and > possibly a share of the indexes, depending on config. So I don't see how > we can optimise where the query is run. > > Naturally, things get easier when your index is a central one and all > nodes have access to it. That just works simpler because you can just > run the query on one node and then once you have the QueryHits you can > then call a Cache.get() on all the nodes. I think :S. I didn't understand how it's assumed the indexes are spread across the nodes; I assume we all agree implicitly that indexes should be split in several pieces, which are then replicated many times across the cluster, so in a similar way to DIST. So in case we use sharding many nodes have the "shard A" and many others have the "shard B", so when you perform a query you don't have to broadcast it but ensure you get an answer from any single node having a copy of A and also from any single node having a copy of B - so you don't get duplicates and don't involve necessarily all nodes. On the opposite way if each node is owning a local-only index of all information it happens to be storing, then the content is not deterministic and also all nodes should re-index everything when rehashing: doesn't seem the way to go. But I think we can do much better; just a set of ideas right now, hopefully they could work; The design of a segment in Lucene is quite similar to a shard. This means we could make good use of the optimisation and segment merging features of Lucene, and finally if each node has a full segment you might even be able to predict where to send the queries as it's similar to a balanced tree. Keep in mind that the current Infinispan Lucene Directory will chunk each segment in pieces when they grow larger than a user defined threshold, this is suboptimal as Lucene isn't aware of it but you can also reconfigure Lucene's LogByteSizeMergePolicy to set a threshold to avoid segments larger than this same value, so to really avoid chunks and have full-segments distribution; so this chunking system is just meant to ease the setup and be compatible to existing software using Lucene, as a drop-in replacement, but not necessarily the way to go for an efficient Query implementation. I think we should think about implementing a distributed Query on top of a non-chunked distributed segments index, as you can predict which segments you're going to hit, and even in which order, so you can send the Query to the appropriate nodes only, and also collect results lazily and avoiding out of memory for sorting issues. I assume it's possible in Infinispan to know which nodes own a specific key, and send queries in round-robin to these; then each node can open it's segment using the segment reader and fullfill the request, as this segment is likely local (always if not during a rehash, but then again still working). Cheers, Sanne > > > -- > Navin Surtani > Intern Infinispan > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

2010/9/17 Manik Surtani <manik(a)jboss.org&gt;: > I think the whole point of ISPN-200 and being able to distribute the query itself is that we don't need to distribute the indexes. Each node maintains indexes only for the data stored on that node locally. that depends on the point of view :) As I personally see it you are still distributing the indexes, as each node has a piece of all information stored in the grid, and to obtain a result it's mandatory to merge them all - so this is clearly "a part of the index".

The problem is that it's undefined which part it is, so you can't take any advantage from it and have to browse all indexes to find matches: the term "index" doesn't seem appropriate any more. Using this structure as said you need de-duplication and broadcast to all nodes, which is suboptimal as it seems that it's possible to avoid it. I don't know how an optimal solution would look like, but I'd take an advantage at least by avoiding these two issues as it seems possible taking advantage from the segments structure - you would really be able to scale. I'm not against solving ISPN-200 by using the proposed approach; at least we will have some code able to send a query command returning a streaming result, so we can later improve on it trying out better approaches; but I hope it could be made in such a way we can later reshape it to avoid broadcasts.

Cheers, Sanne > > Agreed that reindexing during a rehash could be a challenge though. The goal is to have an efficient query right? > > On 16 Sep 2010, at 18:21, Sanne Grinovero wrote: > >> 2010/9/15 Navin Surtani <nsurtani(a)redhat.com&gt;: >>> >>>> >>>> Also sending the query in broadcast is nice for a first >>>> implementation, but this means you can scale the number of searched >>>> items but we can't scale the number of queries, this should be >>>> designed in such a way to make it possible in a future improvement to >>>> send the query only to a subset o the nodes. >>>> >>> >>> Well I don't know of any other way to run the query to be honest. If you >>> imagine that you have several nodes running ISPN in DIST mode, and each >>> node has it's own local index - we don't necessarily know where all of >>> the objects are. Each of them has got a share of all the objects and >>> possibly a share of the indexes, depending on config. So I don't see how >>> we can optimise where the query is run. >>> >>> Naturally, things get easier when your index is a central one and all >>> nodes have access to it. That just works simpler because you can just >>> run the query on one node and then once you have the QueryHits you can >>> then call a Cache.get() on all the nodes. I think :S. >> >> I didn't understand how it's assumed the indexes are spread across the >> nodes; I assume we all agree implicitly that indexes should be split >> in several pieces, which are then replicated many times across the >> cluster, so in a similar way to DIST. >> So in case we use sharding many nodes have the "shard A" and many >> others have the "shard B", so when you perform a query you don't have >> to broadcast it but ensure you get an answer from any single node >> having a copy of A and also from any single node having a copy of B - >> so you don't get duplicates and don't involve necessarily all nodes. >> >> On the opposite way if each node is owning a local-only index of all >> information it happens to be storing, then the content is not >> deterministic and also all nodes should re-index everything when >> rehashing: doesn't seem the way to go. >> >> But I think we can do much better; just a set of ideas right now, >> hopefully they could work; >> The design of a segment in Lucene is quite similar to a shard. This >> means we could make good use of the optimisation and segment merging >> features of Lucene, and finally if each node has a full segment you >> might even be able to predict where to send the queries as it's >> similar to a balanced tree. >> >> Keep in mind that the current Infinispan Lucene Directory will chunk >> each segment in pieces when they grow larger than a user defined >> threshold, this is suboptimal as Lucene isn't aware of it but you can >> also reconfigure Lucene's LogByteSizeMergePolicy to set a threshold to >> avoid segments larger than this same value, so to really avoid chunks >> and have full-segments distribution; so this chunking system is just >> meant to ease the setup and be compatible to existing software using >> Lucene, as a drop-in replacement, but not necessarily the way to go >> for an efficient Query implementation. >> I think we should think about implementing a distributed Query on top >> of a non-chunked distributed segments index, as you can predict which >> segments you're going to hit, and even in which order, so you can send >> the Query to the appropriate nodes only, and also collect results >> lazily and avoiding out of memory for sorting issues. >> I assume it's possible in Infinispan to know which nodes own a >> specific key, and send queries in round-robin to these; then each node >> can open it's segment using the segment reader and fullfill the >> request, as this segment is likely local (always if not during a >> rehash, but then again still working). >> >> Cheers, >> Sanne >> >>> >>> >>> -- >>> Navin Surtani >>> Intern Infinispan >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Hi Israel Any updates on ISPN-200? How are you getting on? Cheers Manik On 15 Sep 2010, at 12:56, Israel Lacerra wrote: Thanks Manik!! I just did not understand the last item.. > - on an implementation level, the GetHItsCommand (or something like that) could return with a single hit, or N hits, with a flag of whether more hits are available or not. Israel On Wed, Sep 15, 2010 at 7:16 AM, Manik Surtani <manik(a)jboss.org&gt; wrote: > How about something like: > > - Broadcast the query > - Every node creates the QueryHits inst, runs the query and collects > results. Starts streaming the results back immediately. > - The lazy iterator returns to the user immediately, and maintains an > internal cache of results coming in from N remote QueryHits instances > - iterator.next() blocks until this cache has available entries to return. > - on an implementation level, the GetHItsCommand (or something like that) > could return with a single hit, or N hits, with a flag of whether more hits > are available or not. > > Does that help? > > Manik > > > On 14 Sep 2010, at 20:46, Israel Lacerra wrote: > > Hi guys, > > I'm still thinking about it, but I don't have a really good idea about the > lazy iterator yet. The only way (that I see) I could make it more lazily is: > > - Broadcast the query. > - Every node creates a QueryHits instance with the query and keep it in a > simple little cache (array, hash, etc) > - A "state" of the query is created and every lazyIterator.next() must > send a command to a node and get the next hit (the next key). > - After a certain time, the instances of queryHits "dies". > > It seems to me that this is not too efficient. But I don't have any other > ideas. > > Do you have any suggestions about it? > > thanks! > Israel > > AnyoneOn Tue, Aug 31, 2010 at 9:14 AM, Israel Lacerra <israeldl(a)gmail.com > > wrote: > >> Hi Navin, >> >> I'm trying to do ISPN-200 (https://jira.jboss.org/browse/ISPN-200) and I >> don't know how to implement a good lazy iterator in a "distributed way". >> >> (Sorry... my english is not soo good. If you don't understand again, >> please just ask again! :) >> >> >> Israel >> >> >> On Tue, Aug 31, 2010 at 6:27 AM, Navin Surtani <nsurtani(a)redhat.com&gt;wrote: >> >>> Hi Israel, >>> >>> Just a quick question to your issue. What do you mean by you do not have >>> the LazyIterator? I suppose I'm not really understanding what your issue >>> is. Is it just that you don't want the lazy version or you don't know >>> how to use it? :S >>> >>> >>> On 25/08/10 14:39, Israel Lacerra wrote: >>> > Ok, Manik! >>> > >>> > I've already have a code working, but not too "lazy" (and without the >>> > "sort" part). I get the keys on all nodes, and then I use them. So I >>> > have a EagerIterator, but not the LazyIterator. :/ >>> > >>> > I'll think about it... >>> > >>> > thanks >>> > >>> > On Wed, Aug 25, 2010 at 10:23 AM, Manik Surtani <manik(a)jboss.org >>> > <mailto:manik@jboss.org>> wrote: >>> > >>> > >>> > On 24 Aug 2010, at 17:20, Israel Lacerra wrote: >>> > >>> > > Manik, >>> > > >>> > > What you mean by: >>> > > " * The calling node returns a CacheQuery impl that lazily >>> fetches >>> > > and collates results from the cluster." (JIRA) >>> > > >>> > > Is enough if each node returns a list of keys and then, we >>> lazily >>> > get the values using the keys? Or the process has to be more lazy >>> yet? >>> > >>> > I think it can be "more lazy" as you said. :) >>> > -- >>> > Manik Surtani >>> > manik(a)jboss.org <mailto:manik@jboss.org> >>> > Lead, Infinispan >>> > Lead, JBoss Cache >>> > http://www.infinispan.org >>> > http://www.jbosscache.org >>> > >>> > >>> > >>> > >>> > >>> > _______________________________________________ >>> > infinispan-dev mailing list >>> > infinispan-dev(a)lists.jboss.org <mailto: >>> infinispan-dev(a)lists.jboss.org&gt; >>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> > >>> > >>> > >>> > >>> > _______________________________________________ >>> > infinispan-dev mailing list >>> > infinispan-dev(a)lists.jboss.org >>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> >>> -- >>> Navin Surtani >>> Intern Infinispan >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >> >> > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org twitter.com/maniksurtani Lead, Infinispan http://www.infinispan.org

On 16 Dec 2010, at 11:15, Israel Lacerra wrote: Hi Manik, I've finished the LazyIterator. Now I need to: - Code some good tests (now I'm testing using InfinispanDemo with some changes) - Implement all methods of iterators (the most simple are not ready yet) - Migrate to git I will send the code to you in the end of the next week (I think git will help with this) and probably you will show a lot of things to improve! No worries. I'll only be able to look at it in the first week of Jan, but I'm sure others on this list will be able to comment in the meanwhile. :-) Cheers Israel On Wed, Dec 15, 2010 at 5:23 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: > Hi Israel > > Any updates on ISPN-200? How are you getting on? > > Cheers > Manik > > On 15 Sep 2010, at 12:56, Israel Lacerra wrote: > > Thanks Manik!! I just did not understand the last item.. > > > - on an implementation level, the GetHItsCommand (or something like > that) could return with a single hit, or N hits, with a flag of whether more > hits are available or not. > > > Israel > > On Wed, Sep 15, 2010 at 7:16 AM, Manik Surtani <manik(a)jboss.org&gt; wrote: > >> How about something like: >> >> - Broadcast the query >> - Every node creates the QueryHits inst, runs the query and collects >> results. Starts streaming the results back immediately. >> - The lazy iterator returns to the user immediately, and maintains an >> internal cache of results coming in from N remote QueryHits instances >> - iterator.next() blocks until this cache has available entries to >> return. >> - on an implementation level, the GetHItsCommand (or something like that) >> could return with a single hit, or N hits, with a flag of whether more hits >> are available or not. >> >> Does that help? >> >> Manik >> >> >> On 14 Sep 2010, at 20:46, Israel Lacerra wrote: >> >> Hi guys, >> >> I'm still thinking about it, but I don't have a really good idea about >> the lazy iterator yet. The only way (that I see) I could make it more lazily >> is: >> >> - Broadcast the query. >> - Every node creates a QueryHits instance with the query and keep it in a >> simple little cache (array, hash, etc) >> - A "state" of the query is created and every lazyIterator.next() must >> send a command to a node and get the next hit (the next key). >> - After a certain time, the instances of queryHits "dies". >> >> It seems to me that this is not too efficient. But I don't have any other >> ideas. >> >> Do you have any suggestions about it? >> >> thanks! >> Israel >> >> AnyoneOn Tue, Aug 31, 2010 at 9:14 AM, Israel Lacerra < >> israeldl(a)gmail.com&gt; wrote: >> >>> Hi Navin, >>> >>> I'm trying to do ISPN-200 (https://jira.jboss.org/browse/ISPN-200) and >>> I don't know how to implement a good lazy iterator in a "distributed way". >>> >>> (Sorry... my english is not soo good. If you don't understand again, >>> please just ask again! :) >>> >>> >>> Israel >>> >>> >>> On Tue, Aug 31, 2010 at 6:27 AM, Navin Surtani <nsurtani(a)redhat.com&gt;wrote: >>> >>>> Hi Israel, >>>> >>>> Just a quick question to your issue. What do you mean by you do not >>>> have >>>> the LazyIterator? I suppose I'm not really understanding what your >>>> issue >>>> is. Is it just that you don't want the lazy version or you don't know >>>> how to use it? :S >>>> >>>> >>>> On 25/08/10 14:39, Israel Lacerra wrote: >>>> > Ok, Manik! >>>> > >>>> > I've already have a code working, but not too "lazy" (and without the >>>> > "sort" part). I get the keys on all nodes, and then I use them. So I >>>> > have a EagerIterator, but not the LazyIterator. :/ >>>> > >>>> > I'll think about it... >>>> > >>>> > thanks >>>> > >>>> > On Wed, Aug 25, 2010 at 10:23 AM, Manik Surtani <manik(a)jboss.org >>>> > <mailto:manik@jboss.org>> wrote: >>>> > >>>> > >>>> > On 24 Aug 2010, at 17:20, Israel Lacerra wrote: >>>> > >>>> > > Manik, >>>> > > >>>> > > What you mean by: >>>> > > " * The calling node returns a CacheQuery impl that lazily >>>> fetches >>>> > > and collates results from the cluster." (JIRA) >>>> > > >>>> > > Is enough if each node returns a list of keys and then, we >>>> lazily >>>> > get the values using the keys? Or the process has to be more lazy >>>> yet? >>>> > >>>> > I think it can be "more lazy" as you said. :) >>>> > -- >>>> > Manik Surtani >>>> > manik(a)jboss.org <mailto:manik@jboss.org> >>>> > Lead, Infinispan >>>> > Lead, JBoss Cache >>>> > http://www.infinispan.org >>>> > http://www.jbosscache.org >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > _______________________________________________ >>>> > infinispan-dev mailing list >>>> > infinispan-dev(a)lists.jboss.org <mailto: >>>> infinispan-dev(a)lists.jboss.org&gt; >>>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> > >>>> > >>>> > >>>> > >>>> > _______________________________________________ >>>> > infinispan-dev mailing list >>>> > infinispan-dev(a)lists.jboss.org >>>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>>> >>>> -- >>>> Navin Surtani >>>> Intern Infinispan >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>> >>> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> -- >> Manik Surtani >> manik(a)jboss.org >> Lead, Infinispan >> Lead, JBoss Cache >> http://www.infinispan.org >> http://www.jbosscache.org >> >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Manik Surtani > manik(a)jboss.org > twitter.com/maniksurtani > > Lead, Infinispan > http://www.infinispan.org > > > > -- Manik Surtani manik(a)jboss.org twitter.com/maniksurtani Lead, Infinispan http://www.infinispan.org

On 30 Dec 2010, at 18:34, Israel Lacerra wrote: > Manik, > > I'm gonna need a few more days. I'll probably send you the code on the middle of the next week (about jan/6). > > Guys, > > About the way I'll send the code, I think the best is to make a fork on github and then I push my changes on this fork and send you the url of the fork. Right? Yes. Pls post the link to this mail list so others can take a look as well. Cheers Manik -- Manik Surtani manik(a)jboss.org twitter.com/maniksurtani Lead, Infinispan http://www.infinispan.org

wrote:

wrote:

On 13 Jan 2011, at 17:05, Israel Lacerra wrote: Are you using Eclipse or another IDE? If you using Eclipse, are you using another GUI (outside Eclipse) to synchronize and see all changes? In general you are probably best off using the cmd line to work with git. See http://community.jboss.org/wiki/InfinispanandGitHub for tips, etc. -- Manik Surtani manik(a)jboss.org twitter.com/maniksurtani Lead, Infinispan http://www.infinispan.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev