Re: [infinispan-dev] Cache.size() on distributed caches?

From: "William Burns" <mudokonman(a)gmail.com&gt; To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; Sent: Monday, July 28, 2014 7:51:27 PM Subject: Re: [infinispan-dev] Cache.size() on distributed caches? I am assuming that you were using a shared cache loader without passivation?  In that case the size method will return all the entries in the cache properly (albeit using a large amount of memory)

To be honest I can't think of a way to get an accurate count of what is in the cache unless you are using a shared loader without passivation in previous versions.  The only way would be if you had no gets/writes(passivation issues) or state transfer while you did the map/reduce task and had it based on if passivation was enabled or not (a bit messy but doable). The distributed iterator should work irrespective of configuration or concurrent operations though. On Fri, Jul 25, 2014 at 10:47 AM, Alan Field <afield(a)redhat.com&gt; wrote: > OK, thanks Mircea! > > Alan > > ----- Original Message ----- >> From: "Mircea Markus" <mmarkus(a)redhat.com&gt; >> To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; >> Sent: Friday, July 25, 2014 4:42:50 PM >> Subject: Re: [infinispan-dev] Cache.size() on distributed caches? >> >> That has changed in ISPN 6: https://issues.jboss.org/browse/ISPN-761 >> >> >> On Jul 25, 2014, at 15:40, Radim Vansa <rvansa(a)redhat.com&gt; wrote: >> >> > I don't think RadarGun is much related to that, besides the fact that it >> > can be buggy :) >> > >> > The question is whether the cache.size() behaviour has changed in a way >> > that i would report full cache size, not just the size of container/the >> > size of cache store? >> > >> > Radim >> > >> > On 07/25/2014 03:54 PM, Alan Field wrote: >> >> Hey, >> >> >> >> I have been looking at adding the ability to get the total size of a >> >> cache >> >> in RadarGun. The first implementation I coded used the distributed >> >> iterators in Infinispan 7.[1] I then realized that implementing >> >> getTotalSize() method using a distributed executor would allow the code >> >> in versions back to Infinispan 5.2. I have the code written, and I have >> >> been running some Jenkins jobs with Infinispan 6.0.1 Final to verify >> >> that >> >> the results are correct.[2] I use the RandomData stage to put data in >> >> the >> >> cache. Here is what it writes in the log: >> >> >> >> 04:11:59,573 INFO  [org.radargun.stages.cache.RandomDataStage] (main) >> >> Received responses from all 4 slaves. Durations [0 = 17.04 minutes, 1 = >> >> 18.36 minutes, 2 = 18.44 minutes, 3 = 18.58 minutes] >> >> 04:11:59,574 INFO  [org.radargun.stages.cache.RandomDataStage] (main) >> >> -------------------- >> >> 04:11:59,574 INFO  [org.radargun.stages.cache.RandomDataStage] (main) >> >> Filled cache with String objects totaling 25% of the Java heap >> >> 04:11:59,574 INFO  [org.radargun.stages.cache.RandomDataStage] (main) >> >> Slave 0 wrote 479352 values to the cache with a total size of 958,704 >> >> kb; >> >> targetMemoryUse = 1,022,368 kb; countOfWordsInData = 44900952 >> >> 04:11:59,575 INFO  [org.radargun.stages.cache.RandomDataStage] (main) >> >> Slave 1 wrote 479352 values to the cache with a total size of 958,704 >> >> kb; >> >> targetMemoryUse = 1,022,368 kb; countOfWordsInData = 44914319 >> >> 04:11:59,575 INFO  [org.radargun.stages.cache.RandomDataStage] (main) >> >> Slave 2 wrote 479352 values to the cache with a total size of 958,704 >> >> kb; >> >> targetMemoryUse = 1,022,368 kb; countOfWordsInData = 44906729 >> >> 04:11:59,575 INFO  [org.radargun.stages.cache.RandomDataStage] (main) >> >> Slave 3 wrote 479352 values to the cache with a total size of 958,704 >> >> kb; >> >> targetMemoryUse = 1,022,368 kb; countOfWordsInData = 44908687 >> >> 04:11:59,576 INFO  [org.radargun.stages.cache.RandomDataStage] (main) >> >> The >> >> cache contains 1917408 values with a total size of 3,834,816 kb >> >> 04:11:59,576 INFO  [org.radargun.stages.cache.RandomDataStage] (main) >> >> 100 >> >> words were generated with a maximum length of 20 characters >> >> 04:11:59,576 INFO  [org.radargun.stages.cache.RandomDataStage] (main) >> >> -------------------- >> >> >> >> These are the outputs from my getTotalSize() code: >> >> >> >> 04:11:59,591 INFO  [org.radargun.service.Infinispan53CacheInfo] (main) >> >> org.radargun.service.Infinispan53CacheInfo$Cache.getTotalSize() for >> >> cache >> >> testCache >> >> 04:12:12,094 INFO  [org.radargun.service.Infinispan53CacheInfo] (main) >> >> cache.size() = 1917408 >> >> 04:12:26,283 INFO  [org.radargun.service.Infinispan53CacheInfo] (main) >> >> cache.getAdvancedCache().size() = 1917408 >> >> 04:12:26,283 INFO  [org.radargun.service.Infinispan53CacheInfo] (main) >> >> cache.getAdvancedCache().getCacheConfiguration().clustering().hash().numOwners() >> >> = 2 >> >> 04:12:26,283 INFO  [org.radargun.service.Infinispan53CacheInfo] (main) >> >> cache.getCacheManager().getMembers().size() = 4 >> >> 04:12:41,955 INFO  [org.radargun.stages.cache.ClearCacheStage] (main) >> >> Cache size = 3834800 >> >> >> >> The "Cache size =" message is from the results of my distributed >> >> executor, >> >> and the other messages are informational. These outputs show that >> >> calling >> >> cache size on a distributed cache returns the size of the entire cache >> >> including any passivated entries, not just the size of the cache on the >> >> local node. This breaks the code of my distributed executor, but mostly >> >> makes it unnecessary if I can just call cache.size(). >> >> >> >> Is this an expected change in behavior? >> >> >> >> Thanks, >> >> Alan >> >> >> >> [1] >> >> https://github.com/radargun/radargun/blob/master/plugins/infinispan70/src... >> >> [2] >> >> https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jdg-radargun-gettota... >> >> >> > >> > >> > -- >> > Radim Vansa <rvansa(a)redhat.com&gt; >> > JBoss DataGrid QA >> > >> > _______________________________________________ >> > infinispan-dev mailing list >> > infinispan-dev(a)lists.jboss.org >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> Cheers, >> -- >> Mircea Markus >> Infinispan lead (www.infinispan.org) >> >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev