[infinispan-dev] About size()

Tue Oct 7 08:21:02 EDT 2014

On Tue, Oct 7, 2014 at 7:32 AM, Radim Vansa <rvansa at redhat.com> wrote:
> If you have one local and one shared cache store, how should the command
> behave?
>
> a) distexec/MR sum of cache.withFlags(SKIP_REMOTE_LOOKUP,
> SKIP_BACKUP_ENTRIES).size() from all nodes? (note that there's no
> SKIP_BACKUP_ENTRIES flag right now), where this method returns
> localStore.size() for first non-shared cache store + passivation ?
> dataContainer.size(SKIP_BACKUP_ENTRIES) : 0)

Calling the size method in either distexec or MR will give you
inflated numbers as you need to pay attention to the numOwners to get
a proper count.

> b) distexec/MR sum of sharedStore.size() + passivation ? sum of
> dataContainer.size(SKIP_BACKUP_ENTRIES) : 0

Calling the size on a shared cache actually should work somewhat well
(assuming all entries are stored in the shared cache).  The problem is
if passivation is enabled as you point out because you also have to
check the data container which means you can also have an issue with
concurrent activations and passivations (which you can't verify
properly in either case without knowing the keys).

> c) MR that would count the entries

This is the only reliable way to do this with MR.  And unfortunately
if a rehash occurs I am not sure if you would get inconsistent numbers
or an Exception.  In the latter at least you should be able to make
sure that you have the proper number when it does return without
exception.  I can't say how it works with multiple loaders though, my
guess is that it may process the entry more than once so it depends on
if your mapper is smart enough to realize it.

> d) wrapper on distributed entry iteration with converters set to return
> 0-sized entries

Entry iterator can't return 0 sized entries (just the values).  The
keys are required to make sure that the count is correct and also to
ensure that if a rehash happens in the middle it can properly continue
to operate without having to start over.  Entry iterator should work
properly irrespective of the number of stores/loaders that are
configured, since it keep track of already seen keys (so duplicates
are ignored).

>
> And what about nodes with different configuration?

Hard to know without knowing what the differences are.

>
> Radim
>
> On 10/06/2014 01:57 PM, Sanne Grinovero wrote:
>> On 6 October 2014 12:44, Tristan Tarrant <ttarrant at redhat.com> wrote:
>>> I think we should provide correct implementations of size() (and others)
>>> and provide shortcut implementations using our usual Flag API (e.g.
>>> SKIP_REMOTE_LOOKUP).
>> Right that would be very nice. Same for CacheStore interaction: all
>> cachestores should be included unless skipped explicitly.
>>
>> Sanne
>>
>>> Tristan
>>>
>>> On 06/10/14 12:57, Sanne Grinovero wrote:
>>>> On 3 October 2014 18:38, Dennis Reed <dereed at redhat.com> wrote:
>>>>> Since size() is defined by the ConcurrentMap interface, it already has a
>>>>> precisely defined meaning.  The only "correct" implementation is E.
>>>> +1

This is one of the things I have been wanting to do is actually
implement the other Map methods across the entire cache.  However to
do a lot of these in a memory conscious way they would need to be ran
ignoring any ongoing transactions.  Actually having this requirement
allows these methods to be implemented quite easily especially in
conjunction with the EntryIterator.  I almost made a PR for it a while
back, but it seemed a little zealous to do at the same time and it
didn't seem that people were pushing for it very hard (maybe that was
a wrong assumption).  Also I wasn't quite sure the transactional part
not being functional anymore would be a deterrent.

>>>>
>>>>> The current non-correct implementation was just because it's expensive
>>>>> to calculate correctly.  I'm not sure the current impl is really that
>>>>> useful for anything.
>>>> +1
>>>>
>>>> And not just size() but many others from ConcurrentMap.
>>>> The question is if we should drop the interface and all the methods
>>>> which aren't efficiently implementable, or fix all those methods.
>>>>
>>>> In the past I loved that I could inject "Infinispan superpowers" into
>>>> an application making extensive use of Map and ConcurrentMap without
>>>> changes, but that has been deceiving and required great care such as
>>>> verifying that these features would not be used anywhere in the code.
>>>> I ended up wrapping the Cache implementation in a custom adapter which
>>>> would also implement ConcurrentMap but would throw a RuntimeException
>>>> if any of the "unallowed" methods was called, at least I would detect
>>>> violations safely.
>>>>
>>>> I still think that for the time being - until a better solution is
>>>> planned - we should throw exceptions.. alas that's an old conversation
>>>> and it was never done.
>>>>
>>>> Sanne
>>>>
>>>>
>>>>> -Dennis
>>>>>
>>>>> On 10/03/2014 03:30 AM, Radim Vansa wrote:
>>>>>> Hi,
>>>>>>
>>>>>> recently we had a discussion about what size() returns, but I've
>>>>>> realized there are more things that users would like to know. My
>>>>>> question is whether you think that they would really appreciate it, or
>>>>>> whether it's just my QA point of view where I sometimes compute the
>>>>>> 'checksums' of cache to see if I didn't lost anything.
>>>>>>
>>>>>> There are those sizes:
>>>>>> A) number of owned entries
>>>>>> B) number of entries stored locally in memory
>>>>>> C) number of entries stored in each local cache store
>>>>>> D) number of entries stored in each shared cache store
>>>>>> E) total number of entries in cache
>>>>>>
>>>>>> So far, we can get
>>>>>> B via withFlags(SKIP_CACHE_LOAD).size()
>>>>>> (passivation ? B : 0) + firstNonZero(C, D) via size()
>>>>>> E via distributed iterators / MR
>>>>>> A via data container iteration + distribution manager query, but only
>>>>>> without cache store
>>>>>> C or D through
>>>>>> getComponentRegistry().getLocalComponent(PersistenceManager.class).getStores()
>>>>>>
>>>>>> I think that it would go along with users' expectations if size()
>>>>>> returned E and for the rest we should have special methods on
>>>>>> AdvancedCache. That would of course change the meaning of size(), but
>>>>>> I'd say that finally to something that has firm meaning.
>>>>>>
>>>>>> WDYT?
>>>>>>
>>>>>> Radim
>>>>>>
>>>>> _______________________________________________
>>>>> infinispan-dev mailing list
>>>>> infinispan-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>
>>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Radim Vansa <rvansa at redhat.com>
> JBoss DataGrid QA
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev