[infinispan-dev] BulkGet in hotrod (ISPN-516)

Wed Jul 14 11:39:29 EDT 2010

On 14 Jul 2010, at 16:37, Mircea Markus wrote:

> 
> On 14 Jul 2010, at 15:56, Mircea Markus wrote:
> 
>> 
>> On 14 Jul 2010, at 15:29, Galder Zamarreño wrote:
>> 
>>> 
>>> On Jul 14, 2010, at 3:08 PM, Manik Surtani wrote:
>>> 
>>>> 
>>>> On 14 Jul 2010, at 13:23, Mircea Markus wrote:
>>>> 
>>>>> thanks, see bellow.
>>>>> On 14 Jul 2010, at 11:49, Galder Zamarreño wrote:
>>>>> 
>>>>>> Hi Mircea,
>>>>>> 
>>>>>> You should make it explicit in the title that this is a design document to avoid confusion.
>>>>> done
>>>>>> I'm not sure I understand what the filter and exclusion parameters are about.
>>>>> Filtering is about excluding from the set of entries returned by the server to the client of those filtering keys specified in the request (filter size] [filter key size 1] [filter key 1] [filter key size 2] [filter key 2]...).
>>>>>> A multi get, as I see is about giving a list of keys and returning them. So, why use filtering?
>>>>> We need filtering because of RemoteCacheStore: it needs to implement  CacheLoader.loadAllKeys(Set<Object> keysToExclude)
>>>> 
>>>> Like I said (on the wiki) - this may not be needed.  See my reasons there.
>>> 
>>> In which case, I only see an immediate need for a getAll operation where all keys are retrieved. 
>>> 
>>> I'm still not convinced for the need of a multi get op. Multi gets on a subset could be done by the client parallelizing/pipelining individual get requests.
>> The hotrod client would need to know all the keys in order to do this, which is not possible afaik.
>>> You could also do multi-gets doing async or non-blocking gets. IOW, if you want to get 10 keys, do 10 non-blocking gets and then wait for the results of all of them and it's here where netty based client could be more performant. This way you don't have to wait for each response before moving on and this is simpler than paralelizing. Very interesting stuff to read in http://code.google.com/p/memcached/wiki/FAQ#Batch_your_requests_with_get_multi
>> Future API on client has the same result: requests are being directed to multiple servers, according the specified RequestBalancingStrategy (or key's hash code, if DIST).  
> Me wrong. We don't have an asyncGet, but this totally looks like a cool feature?
that wasn't supposed to be a question :) "this totally looks like a cool feature!"
> Wdyt?
>> 
>>> and http://lists.danga.com/pipermail/memcached/2007-July/004528.html
>> 
>>> 
>>>> 
>>>>> 
>>>>>> And what type of filtering would it be? REGEX based filtering?...etc. Are you trying to use filtering as a way for clients to instruct to load all keys for example? Like get(*) ?
>>>>> Just explicit filtering: specify the keys you want to be filtered, an the way you want to filter them (inclusive or exclusive filtering)
>>>>> 
>>>>>> 
>>>>>> What does it mean that they're inclusive? Only keys that fulfill all filters will be returned? If so, what's the use case for this?
>>>>> it means that only these keys are being returned form the client, if they exist. The use case is adding an RemoteCache.get(Set) method (as stated in the document). 
>>>> 
>>>> How do you currently handle CacheStore.get(Set keys)?
>>>> 
>>>>>> I'm also worried if this might be complicating the protocol when RCS can simply start a bunch of threads and get all the keys in paralell individually. The only thing that would be really be missing here is how to get the list of all keys stored in the server, for loadAll() calls for example.
>>>>> that should be possible by specifying a "key size = -1" in the request - I've updated the document as that bit was missing.  
>>>> 
>>>> If you are using a variable length int as a field for num_keys, you won't be able to set it to -1.  vInts are unsigned.
>>>> 
>>>>>> One thing to note is that Memcached binary protocol removed support for multi-get as such. IOW, in the txt protocol, u could do: get <k1>, <k2>...etc but not any more in the binary one.
>>>>>> 
>>>>>> For the binary protocol, they went for a different approach based on get/getq where they pipeline requests, see "4.2.  Get, Get Quietly, Get Key, Get Key Quietly" section on: http://code.google.com/p/memcached/wiki/MemcacheBinaryProtocol
>>>>>> 
>>>>>> I think it's interesting to find out more about, particularly about this sentence cos I'm not sure I understand it: "You can also do the naive thing and send n pipelined get/getks, but then you could potentially get back a lot of "NOT_FOUND" error code packets. "
>>>>>> 
>>>>> Interesting!  looking into this right now
>>>>>> Cheers,
>>>>>> 
>>>>>> On Jul 14, 2010, at 9:50 AM, Mircea Markus wrote:
>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I've added a wiki to summarise some thoughts around how this should work: http://community.jboss.org/wiki/HotRodBulkGet
>>>>>>> Can you please take a look and comment?
>>>>>>> 
>>>>>>> Cheers,
>>>>>>> Mircea
>>>>>>> _______________________________________________
>>>>>>> infinispan-dev mailing list
>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>> 
>>>>>> --
>>>>>> Galder Zamarreño
>>>>>> Sr. Software Engineer
>>>>>> Infinispan, JBoss Cache
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> infinispan-dev mailing list
>>>>>> infinispan-dev at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>> 
>>>>> _______________________________________________
>>>>> infinispan-dev mailing list
>>>>> infinispan-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>> 
>>>> --
>>>> Manik Surtani
>>>> manik at jboss.org
>>>> Lead, Infinispan
>>>> Lead, JBoss Cache
>>>> http://www.infinispan.org
>>>> http://www.jbosscache.org
>>>> 
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> 
>>> --
>>> Galder Zamarreño
>>> Sr. Software Engineer
>>> Infinispan, JBoss Cache
>>> 
>>> 
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> 
>> 
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev