8:54 a.m.
Hey,
I have been looking at adding the ability to get the total size of a cache in RadarGun.
The first implementation I coded used the distributed iterators in Infinispan 7.[1] I then
realized that implementing getTotalSize() method using a distributed executor would allow
the code in versions back to Infinispan 5.2. I have the code written, and I have been
running some Jenkins jobs with Infinispan 6.0.1 Final to verify that the results are
correct.[2] I use the RandomData stage to put data in the cache. Here is what it writes in
the log:
04:11:59,573 INFO [org.radargun.stages.cache.RandomDataStage] (main) Received responses
from all 4 slaves. Durations [0 = 17.04 minutes, 1 = 18.36 minutes, 2 = 18.44 minutes, 3 =
18.58 minutes]
04:11:59,574 INFO [org.radargun.stages.cache.RandomDataStage] (main)
--------------------
04:11:59,574 INFO [org.radargun.stages.cache.RandomDataStage] (main) Filled cache with
String objects totaling 25% of the Java heap
04:11:59,574 INFO [org.radargun.stages.cache.RandomDataStage] (main) Slave 0 wrote 479352
values to the cache with a total size of 958,704 kb; targetMemoryUse = 1,022,368 kb;
countOfWordsInData = 44900952
04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage] (main) Slave 1 wrote 479352
values to the cache with a total size of 958,704 kb; targetMemoryUse = 1,022,368 kb;
countOfWordsInData = 44914319
04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage] (main) Slave 2 wrote 479352
values to the cache with a total size of 958,704 kb; targetMemoryUse = 1,022,368 kb;
countOfWordsInData = 44906729
04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage] (main) Slave 3 wrote 479352
values to the cache with a total size of 958,704 kb; targetMemoryUse = 1,022,368 kb;
countOfWordsInData = 44908687
04:11:59,576 INFO [org.radargun.stages.cache.RandomDataStage] (main) The cache contains
1917408 values with a total size of 3,834,816 kb
04:11:59,576 INFO [org.radargun.stages.cache.RandomDataStage] (main) 100 words were
generated with a maximum length of 20 characters
04:11:59,576 INFO [org.radargun.stages.cache.RandomDataStage] (main)
--------------------
These are the outputs from my getTotalSize() code:
04:11:59,591 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
org.radargun.service.Infinispan53CacheInfo$Cache.getTotalSize() for cache testCache
04:12:12,094 INFO [org.radargun.service.Infinispan53CacheInfo] (main) cache.size() =
1917408
04:12:26,283 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
cache.getAdvancedCache().size() = 1917408
04:12:26,283 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
cache.getAdvancedCache().getCacheConfiguration().clustering().hash().numOwners() = 2
04:12:26,283 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
cache.getCacheManager().getMembers().size() = 4
04:12:41,955 INFO [org.radargun.stages.cache.ClearCacheStage] (main) Cache size =
3834800
The "Cache size =" message is from the results of my distributed executor, and
the other messages are informational. These outputs show that calling cache size on a
distributed cache returns the size of the entire cache including any passivated entries,
not just the size of the cache on the local node. This breaks the code of my distributed
executor, but mostly makes it unnecessary if I can just call cache.size().
Is this an expected change in behavior?
Thanks,
Alan
[1]
https://github.com/radargun/radargun/blob/master/plugins/infinispan70/src...
[2]
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jdg-radargun-gettota...
--
Alan Field
Principal Quality Engineer - JBoss Data Grid
T: (919) 890-8932 | Ext. 8148932
9:40 a.m.
I don't think RadarGun is much related to that, besides the fact that it
can be buggy :)
The question is whether the cache.size() behaviour has changed in a way
that i would report full cache size, not just the size of container/the
size of cache store?
Radim
On 07/25/2014 03:54 PM, Alan Field wrote:
Hey,
I have been looking at adding the ability to get the total size of a cache in RadarGun.
The first implementation I coded used the distributed iterators in Infinispan 7.[1] I then
realized that implementing getTotalSize() method using a distributed executor would allow
the code in versions back to Infinispan 5.2. I have the code written, and I have been
running some Jenkins jobs with Infinispan 6.0.1 Final to verify that the results are
correct.[2] I use the RandomData stage to put data in the cache. Here is what it writes in
the log:
04:11:59,573 INFO [org.radargun.stages.cache.RandomDataStage] (main) Received responses
from all 4 slaves. Durations [0 = 17.04 minutes, 1 = 18.36 minutes, 2 = 18.44 minutes, 3 =
18.58 minutes]
04:11:59,574 INFO [org.radargun.stages.cache.RandomDataStage] (main)
--------------------
04:11:59,574 INFO [org.radargun.stages.cache.RandomDataStage] (main) Filled cache with
String objects totaling 25% of the Java heap
04:11:59,574 INFO [org.radargun.stages.cache.RandomDataStage] (main) Slave 0 wrote
479352 values to the cache with a total size of 958,704 kb; targetMemoryUse = 1,022,368
kb; countOfWordsInData = 44900952
04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage] (main) Slave 1 wrote
479352 values to the cache with a total size of 958,704 kb; targetMemoryUse = 1,022,368
kb; countOfWordsInData = 44914319
04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage] (main) Slave 2 wrote
479352 values to the cache with a total size of 958,704 kb; targetMemoryUse = 1,022,368
kb; countOfWordsInData = 44906729
04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage] (main) Slave 3 wrote
479352 values to the cache with a total size of 958,704 kb; targetMemoryUse = 1,022,368
kb; countOfWordsInData = 44908687
04:11:59,576 INFO [org.radargun.stages.cache.RandomDataStage] (main) The cache contains
1917408 values with a total size of 3,834,816 kb
04:11:59,576 INFO [org.radargun.stages.cache.RandomDataStage] (main) 100 words were
generated with a maximum length of 20 characters
04:11:59,576 INFO [org.radargun.stages.cache.RandomDataStage] (main)
--------------------
These are the outputs from my getTotalSize() code:
04:11:59,591 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
org.radargun.service.Infinispan53CacheInfo$Cache.getTotalSize() for cache testCache
04:12:12,094 INFO [org.radargun.service.Infinispan53CacheInfo] (main) cache.size() =
1917408
04:12:26,283 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
cache.getAdvancedCache().size() = 1917408
04:12:26,283 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
cache.getAdvancedCache().getCacheConfiguration().clustering().hash().numOwners() = 2
04:12:26,283 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
cache.getCacheManager().getMembers().size() = 4
04:12:41,955 INFO [org.radargun.stages.cache.ClearCacheStage] (main) Cache size =
3834800
The "Cache size =" message is from the results of my distributed executor, and
the other messages are informational. These outputs show that calling cache size on a
distributed cache returns the size of the entire cache including any passivated entries,
not just the size of the cache on the local node. This breaks the code of my distributed
executor, but mostly makes it unnecessary if I can just call cache.size().
Is this an expected change in behavior?
Thanks,
Alan
[1]
https://github.com/radargun/radargun/blob/master/plugins/infinispan70/src...
[2]
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jdg-radargun-gettota...
--
Radim Vansa <rvansa(a)redhat.com>
JBoss DataGrid QA
9:42 a.m.
That has changed in ISPN 6: https://issues.jboss.org/browse/ISPN-761
On Jul 25, 2014, at 15:40, Radim Vansa <rvansa(a)redhat.com> wrote:
I don't think RadarGun is much related to that, besides the fact
that it
can be buggy :)
The question is whether the cache.size() behaviour has changed in a way
that i would report full cache size, not just the size of container/the
size of cache store?
Radim
On 07/25/2014 03:54 PM, Alan Field wrote:
> Hey,
>
> I have been looking at adding the ability to get the total size of a cache in
RadarGun. The first implementation I coded used the distributed iterators in Infinispan
7.[1] I then realized that implementing getTotalSize() method using a distributed executor
would allow the code in versions back to Infinispan 5.2. I have the code written, and I
have been running some Jenkins jobs with Infinispan 6.0.1 Final to verify that the results
are correct.[2] I use the RandomData stage to put data in the cache. Here is what it
writes in the log:
>
> 04:11:59,573 INFO [org.radargun.stages.cache.RandomDataStage] (main) Received
responses from all 4 slaves. Durations [0 = 17.04 minutes, 1 = 18.36 minutes, 2 = 18.44
minutes, 3 = 18.58 minutes]
> 04:11:59,574 INFO [org.radargun.stages.cache.RandomDataStage] (main)
--------------------
> 04:11:59,574 INFO [org.radargun.stages.cache.RandomDataStage] (main) Filled cache
with String objects totaling 25% of the Java heap
> 04:11:59,574 INFO [org.radargun.stages.cache.RandomDataStage] (main) Slave 0 wrote
479352 values to the cache with a total size of 958,704 kb; targetMemoryUse = 1,022,368
kb; countOfWordsInData = 44900952
> 04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage] (main) Slave 1 wrote
479352 values to the cache with a total size of 958,704 kb; targetMemoryUse = 1,022,368
kb; countOfWordsInData = 44914319
> 04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage] (main) Slave 2 wrote
479352 values to the cache with a total size of 958,704 kb; targetMemoryUse = 1,022,368
kb; countOfWordsInData = 44906729
> 04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage] (main) Slave 3 wrote
479352 values to the cache with a total size of 958,704 kb; targetMemoryUse = 1,022,368
kb; countOfWordsInData = 44908687
> 04:11:59,576 INFO [org.radargun.stages.cache.RandomDataStage] (main) The cache
contains 1917408 values with a total size of 3,834,816 kb
> 04:11:59,576 INFO [org.radargun.stages.cache.RandomDataStage] (main) 100 words were
generated with a maximum length of 20 characters
> 04:11:59,576 INFO [org.radargun.stages.cache.RandomDataStage] (main)
--------------------
>
> These are the outputs from my getTotalSize() code:
>
> 04:11:59,591 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
org.radargun.service.Infinispan53CacheInfo$Cache.getTotalSize() for cache testCache
> 04:12:12,094 INFO [org.radargun.service.Infinispan53CacheInfo] (main) cache.size() =
1917408
> 04:12:26,283 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
cache.getAdvancedCache().size() = 1917408
> 04:12:26,283 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
cache.getAdvancedCache().getCacheConfiguration().clustering().hash().numOwners() = 2
> 04:12:26,283 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
cache.getCacheManager().getMembers().size() = 4
> 04:12:41,955 INFO [org.radargun.stages.cache.ClearCacheStage] (main) Cache size =
3834800
>
> The "Cache size =" message is from the results of my distributed executor,
and the other messages are informational. These outputs show that calling cache size on a
distributed cache returns the size of the entire cache including any passivated entries,
not just the size of the cache on the local node. This breaks the code of my distributed
executor, but mostly makes it unnecessary if I can just call cache.size().
>
> Is this an expected change in behavior?
>
> Thanks,
> Alan
>
> [1]
https://github.com/radargun/radargun/blob/master/plugins/infinispan70/src...
> [2]
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jdg-radargun-gettota...
>
--
Radim Vansa <rvansa(a)redhat.com>
JBoss DataGrid QA
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
9:47 a.m.
OK, thanks Mircea!
Alan
----- Original Message -----
From: "Mircea Markus" <mmarkus(a)redhat.com>
To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>
Sent: Friday, July 25, 2014 4:42:50 PM
Subject: Re: [infinispan-dev] Cache.size() on distributed caches?
That has changed in ISPN 6: https://issues.jboss.org/browse/ISPN-761
On Jul 25, 2014, at 15:40, Radim Vansa <rvansa(a)redhat.com> wrote:
> I don't think RadarGun is much related to that, besides the fact that it
> can be buggy :)
>
> The question is whether the cache.size() behaviour has changed in a way
> that i would report full cache size, not just the size of container/the
> size of cache store?
>
> Radim
>
> On 07/25/2014 03:54 PM, Alan Field wrote:
>> Hey,
>>
>> I have been looking at adding the ability to get the total size of a cache
>> in RadarGun. The first implementation I coded used the distributed
>> iterators in Infinispan 7.[1] I then realized that implementing
>> getTotalSize() method using a distributed executor would allow the code
>> in versions back to Infinispan 5.2. I have the code written, and I have
>> been running some Jenkins jobs with Infinispan 6.0.1 Final to verify that
>> the results are correct.[2] I use the RandomData stage to put data in the
>> cache. Here is what it writes in the log:
>>
>> 04:11:59,573 INFO [org.radargun.stages.cache.RandomDataStage] (main)
>> Received responses from all 4 slaves. Durations [0 = 17.04 minutes, 1 =
>> 18.36 minutes, 2 = 18.44 minutes, 3 = 18.58 minutes]
>> 04:11:59,574 INFO [org.radargun.stages.cache.RandomDataStage] (main)
>> --------------------
>> 04:11:59,574 INFO [org.radargun.stages.cache.RandomDataStage] (main)
>> Filled cache with String objects totaling 25% of the Java heap
>> 04:11:59,574 INFO [org.radargun.stages.cache.RandomDataStage] (main)
>> Slave 0 wrote 479352 values to the cache with a total size of 958,704 kb;
>> targetMemoryUse = 1,022,368 kb; countOfWordsInData = 44900952
>> 04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage] (main)
>> Slave 1 wrote 479352 values to the cache with a total size of 958,704 kb;
>> targetMemoryUse = 1,022,368 kb; countOfWordsInData = 44914319
>> 04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage] (main)
>> Slave 2 wrote 479352 values to the cache with a total size of 958,704 kb;
>> targetMemoryUse = 1,022,368 kb; countOfWordsInData = 44906729
>> 04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage] (main)
>> Slave 3 wrote 479352 values to the cache with a total size of 958,704 kb;
>> targetMemoryUse = 1,022,368 kb; countOfWordsInData = 44908687
>> 04:11:59,576 INFO [org.radargun.stages.cache.RandomDataStage] (main) The
>> cache contains 1917408 values with a total size of 3,834,816 kb
>> 04:11:59,576 INFO [org.radargun.stages.cache.RandomDataStage] (main) 100
>> words were generated with a maximum length of 20 characters
>> 04:11:59,576 INFO [org.radargun.stages.cache.RandomDataStage] (main)
>> --------------------
>>
>> These are the outputs from my getTotalSize() code:
>>
>> 04:11:59,591 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
>> org.radargun.service.Infinispan53CacheInfo$Cache.getTotalSize() for cache
>> testCache
>> 04:12:12,094 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
>> cache.size() = 1917408
>> 04:12:26,283 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
>> cache.getAdvancedCache().size() = 1917408
>> 04:12:26,283 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
>>
cache.getAdvancedCache().getCacheConfiguration().clustering().hash().numOwners()
>> = 2
>> 04:12:26,283 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
>> cache.getCacheManager().getMembers().size() = 4
>> 04:12:41,955 INFO [org.radargun.stages.cache.ClearCacheStage] (main)
>> Cache size = 3834800
>>
>> The "Cache size =" message is from the results of my distributed
executor,
>> and the other messages are informational. These outputs show that calling
>> cache size on a distributed cache returns the size of the entire cache
>> including any passivated entries, not just the size of the cache on the
>> local node. This breaks the code of my distributed executor, but mostly
>> makes it unnecessary if I can just call cache.size().
>>
>> Is this an expected change in behavior?
>>
>> Thanks,
>> Alan
>>
>> [1]
>>
https://github.com/radargun/radargun/blob/master/plugins/infinispan70/src...
>> [2]
>>
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jdg-radargun-gettota...
>>
>
>
> --
> Radim Vansa <rvansa(a)redhat.com>
> JBoss DataGrid QA
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
12:51 p.m.
I am assuming that you were using a shared cache loader without
passivation? In that case the size method will return all the entries
in the cache properly (albeit using a large amount of memory)
To be honest I can't think of a way to get an accurate count of what
is in the cache unless you are using a shared loader without
passivation in previous versions. The only way would be if you had no
gets/writes(passivation issues) or state transfer while you did the
map/reduce task and had it based on if passivation was enabled or not
(a bit messy but doable).
The distributed iterator should work irrespective of configuration or
concurrent operations though.
On Fri, Jul 25, 2014 at 10:47 AM, Alan Field <afield(a)redhat.com> wrote:
OK, thanks Mircea!
Alan
----- Original Message -----
> From: "Mircea Markus" <mmarkus(a)redhat.com>
> To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>
> Sent: Friday, July 25, 2014 4:42:50 PM
> Subject: Re: [infinispan-dev] Cache.size() on distributed caches?
>
> That has changed in ISPN 6: https://issues.jboss.org/browse/ISPN-761
>
>
> On Jul 25, 2014, at 15:40, Radim Vansa <rvansa(a)redhat.com> wrote:
>
> > I don't think RadarGun is much related to that, besides the fact that it
> > can be buggy :)
> >
> > The question is whether the cache.size() behaviour has changed in a way
> > that i would report full cache size, not just the size of container/the
> > size of cache store?
> >
> > Radim
> >
> > On 07/25/2014 03:54 PM, Alan Field wrote:
> >> Hey,
> >>
> >> I have been looking at adding the ability to get the total size of a cache
> >> in RadarGun. The first implementation I coded used the distributed
> >> iterators in Infinispan 7.[1] I then realized that implementing
> >> getTotalSize() method using a distributed executor would allow the code
> >> in versions back to Infinispan 5.2. I have the code written, and I have
> >> been running some Jenkins jobs with Infinispan 6.0.1 Final to verify that
> >> the results are correct.[2] I use the RandomData stage to put data in the
> >> cache. Here is what it writes in the log:
> >>
> >> 04:11:59,573 INFO [org.radargun.stages.cache.RandomDataStage] (main)
> >> Received responses from all 4 slaves. Durations [0 = 17.04 minutes, 1 =
> >> 18.36 minutes, 2 = 18.44 minutes, 3 = 18.58 minutes]
> >> 04:11:59,574 INFO [org.radargun.stages.cache.RandomDataStage] (main)
> >> --------------------
> >> 04:11:59,574 INFO [org.radargun.stages.cache.RandomDataStage] (main)
> >> Filled cache with String objects totaling 25% of the Java heap
> >> 04:11:59,574 INFO [org.radargun.stages.cache.RandomDataStage] (main)
> >> Slave 0 wrote 479352 values to the cache with a total size of 958,704 kb;
> >> targetMemoryUse = 1,022,368 kb; countOfWordsInData = 44900952
> >> 04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage] (main)
> >> Slave 1 wrote 479352 values to the cache with a total size of 958,704 kb;
> >> targetMemoryUse = 1,022,368 kb; countOfWordsInData = 44914319
> >> 04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage] (main)
> >> Slave 2 wrote 479352 values to the cache with a total size of 958,704 kb;
> >> targetMemoryUse = 1,022,368 kb; countOfWordsInData = 44906729
> >> 04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage] (main)
> >> Slave 3 wrote 479352 values to the cache with a total size of 958,704 kb;
> >> targetMemoryUse = 1,022,368 kb; countOfWordsInData = 44908687
> >> 04:11:59,576 INFO [org.radargun.stages.cache.RandomDataStage] (main) The
> >> cache contains 1917408 values with a total size of 3,834,816 kb
> >> 04:11:59,576 INFO [org.radargun.stages.cache.RandomDataStage] (main) 100
> >> words were generated with a maximum length of 20 characters
> >> 04:11:59,576 INFO [org.radargun.stages.cache.RandomDataStage] (main)
> >> --------------------
> >>
> >> These are the outputs from my getTotalSize() code:
> >>
> >> 04:11:59,591 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
> >> org.radargun.service.Infinispan53CacheInfo$Cache.getTotalSize() for cache
> >> testCache
> >> 04:12:12,094 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
> >> cache.size() = 1917408
> >> 04:12:26,283 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
> >> cache.getAdvancedCache().size() = 1917408
> >> 04:12:26,283 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
> >>
cache.getAdvancedCache().getCacheConfiguration().clustering().hash().numOwners()
> >> = 2
> >> 04:12:26,283 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
> >> cache.getCacheManager().getMembers().size() = 4
> >> 04:12:41,955 INFO [org.radargun.stages.cache.ClearCacheStage] (main)
> >> Cache size = 3834800
> >>
> >> The "Cache size =" message is from the results of my distributed
executor,
> >> and the other messages are informational. These outputs show that calling
> >> cache size on a distributed cache returns the size of the entire cache
> >> including any passivated entries, not just the size of the cache on the
> >> local node. This breaks the code of my distributed executor, but mostly
> >> makes it unnecessary if I can just call cache.size().
> >>
> >> Is this an expected change in behavior?
> >>
> >> Thanks,
> >> Alan
> >>
> >> [1]
> >>
https://github.com/radargun/radargun/blob/master/plugins/infinispan70/src...
> >> [2]
> >>
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jdg-radargun-gettota...
> >>
> >
> >
> > --
> > Radim Vansa <rvansa(a)redhat.com>
> > JBoss DataGrid QA
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
2:53 a.m.
Hey Will,
----- Original Message -----
From: "William Burns" <mudokonman(a)gmail.com>
To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>
Sent: Monday, July 28, 2014 7:51:27 PM
Subject: Re: [infinispan-dev] Cache.size() on distributed caches?
I am assuming that you were using a shared cache loader without
passivation? In that case the size method will return all the entries
in the cache properly (albeit using a large amount of memory)
This is the cache store configuration I had for the cache in question:
<eviction maxEntries="50" />
<persistence passivation="true" >
<leveldbStore xmlns="urn:infinispan:config:store:leveldb:6.0"
implementationType="JAVA"
location="/tmp/ispn-leveldb-jni/data"
expiredLocation="/tmp/ispn-leveldb-jni/expired"
purgeOnStartup="true"
preload="false"
/>
</persistence>
Shared is disabled by default, but passivation was enabled. However, I am also seeing this
behavior with these lines commented out in the cache configuration.
ISPN-761 talks about handling passivated entries in a cache store, but also says that
size() will only show the number of entries on the local node and won't check with
other nodes in the cluster. That is not what I am seeing. I have four different machines
writing entries to the cache:
04:11:59,574 INFO [org.radargun.stages.cache.RandomDataStage] (main) Filled cache with
String objects totaling 25% of the Java heap
04:11:59,574 INFO [org.radargun.stages.cache.RandomDataStage] (main) Slave 0 wrote 479352
values to the cache with a total size of 958,704 kb; targetMemoryUse = 1,022,368 kb;
countOfWordsInData = 44900952
04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage] (main) Slave 1 wrote 479352
values to the cache with a total size of 958,704 kb; targetMemoryUse = 1,022,368 kb;
countOfWordsInData = 44914319
04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage] (main) Slave 2 wrote 479352
values to the cache with a total size of 958,704 kb; targetMemoryUse = 1,022,368 kb;
countOfWordsInData = 44906729
04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage] (main) Slave 3 wrote 479352
values to the cache with a total size of 958,704 kb; targetMemoryUse = 1,022,368 kb;
countOfWordsInData = 44908687
04:11:59,576 INFO [org.radargun.stages.cache.RandomDataStage] (main) The cache contains
1917408 values with a total size of 3,834,816 kb
And when I call Cache.size(), I am getting 1917408 which is the total size of the cache
across all 4 nodes:
04:12:12,094 INFO [org.radargun.service.Infinispan53CacheInfo] (main) cache.size() =
1917408
I am not expecting this to happen, but that is what I am seeing with or without a cache
store.
Thanks,
Alan
To be honest I can't think of a way to get an accurate count of what
is in the cache unless you are using a shared loader without
passivation in previous versions. The only way would be if you had no
gets/writes(passivation issues) or state transfer while you did the
map/reduce task and had it based on if passivation was enabled or not
(a bit messy but doable).
The distributed iterator should work irrespective of configuration or
concurrent operations though.
On Fri, Jul 25, 2014 at 10:47 AM, Alan Field <afield(a)redhat.com> wrote:
> OK, thanks Mircea!
>
> Alan
>
> ----- Original Message -----
>> From: "Mircea Markus" <mmarkus(a)redhat.com>
>> To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>
>> Sent: Friday, July 25, 2014 4:42:50 PM
>> Subject: Re: [infinispan-dev] Cache.size() on distributed caches?
>>
>> That has changed in ISPN 6: https://issues.jboss.org/browse/ISPN-761
>>
>>
>> On Jul 25, 2014, at 15:40, Radim Vansa <rvansa(a)redhat.com> wrote:
>>
>> > I don't think RadarGun is much related to that, besides the fact that
it
>> > can be buggy :)
>> >
>> > The question is whether the cache.size() behaviour has changed in a way
>> > that i would report full cache size, not just the size of container/the
>> > size of cache store?
>> >
>> > Radim
>> >
>> > On 07/25/2014 03:54 PM, Alan Field wrote:
>> >> Hey,
>> >>
>> >> I have been looking at adding the ability to get the total size of a
>> >> cache
>> >> in RadarGun. The first implementation I coded used the distributed
>> >> iterators in Infinispan 7.[1] I then realized that implementing
>> >> getTotalSize() method using a distributed executor would allow the
code
>> >> in versions back to Infinispan 5.2. I have the code written, and I
have
>> >> been running some Jenkins jobs with Infinispan 6.0.1 Final to verify
>> >> that
>> >> the results are correct.[2] I use the RandomData stage to put data in
>> >> the
>> >> cache. Here is what it writes in the log:
>> >>
>> >> 04:11:59,573 INFO [org.radargun.stages.cache.RandomDataStage] (main)
>> >> Received responses from all 4 slaves. Durations [0 = 17.04 minutes, 1
=
>> >> 18.36 minutes, 2 = 18.44 minutes, 3 = 18.58 minutes]
>> >> 04:11:59,574 INFO [org.radargun.stages.cache.RandomDataStage] (main)
>> >> --------------------
>> >> 04:11:59,574 INFO [org.radargun.stages.cache.RandomDataStage] (main)
>> >> Filled cache with String objects totaling 25% of the Java heap
>> >> 04:11:59,574 INFO [org.radargun.stages.cache.RandomDataStage] (main)
>> >> Slave 0 wrote 479352 values to the cache with a total size of 958,704
>> >> kb;
>> >> targetMemoryUse = 1,022,368 kb; countOfWordsInData = 44900952
>> >> 04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage] (main)
>> >> Slave 1 wrote 479352 values to the cache with a total size of 958,704
>> >> kb;
>> >> targetMemoryUse = 1,022,368 kb; countOfWordsInData = 44914319
>> >> 04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage] (main)
>> >> Slave 2 wrote 479352 values to the cache with a total size of 958,704
>> >> kb;
>> >> targetMemoryUse = 1,022,368 kb; countOfWordsInData = 44906729
>> >> 04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage] (main)
>> >> Slave 3 wrote 479352 values to the cache with a total size of 958,704
>> >> kb;
>> >> targetMemoryUse = 1,022,368 kb; countOfWordsInData = 44908687
>> >> 04:11:59,576 INFO [org.radargun.stages.cache.RandomDataStage] (main)
>> >> The
>> >> cache contains 1917408 values with a total size of 3,834,816 kb
>> >> 04:11:59,576 INFO [org.radargun.stages.cache.RandomDataStage] (main)
>> >> 100
>> >> words were generated with a maximum length of 20 characters
>> >> 04:11:59,576 INFO [org.radargun.stages.cache.RandomDataStage] (main)
>> >> --------------------
>> >>
>> >> These are the outputs from my getTotalSize() code:
>> >>
>> >> 04:11:59,591 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
>> >> org.radargun.service.Infinispan53CacheInfo$Cache.getTotalSize() for
>> >> cache
>> >> testCache
>> >> 04:12:12,094 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
>> >> cache.size() = 1917408
>> >> 04:12:26,283 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
>> >> cache.getAdvancedCache().size() = 1917408
>> >> 04:12:26,283 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
>> >>
cache.getAdvancedCache().getCacheConfiguration().clustering().hash().numOwners()
>> >> = 2
>> >> 04:12:26,283 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
>> >> cache.getCacheManager().getMembers().size() = 4
>> >> 04:12:41,955 INFO [org.radargun.stages.cache.ClearCacheStage] (main)
>> >> Cache size = 3834800
>> >>
>> >> The "Cache size =" message is from the results of my
distributed
>> >> executor,
>> >> and the other messages are informational. These outputs show that
>> >> calling
>> >> cache size on a distributed cache returns the size of the entire cache
>> >> including any passivated entries, not just the size of the cache on
the
>> >> local node. This breaks the code of my distributed executor, but
mostly
>> >> makes it unnecessary if I can just call cache.size().
>> >>
>> >> Is this an expected change in behavior?
>> >>
>> >> Thanks,
>> >> Alan
>> >>
>> >> [1]
>> >>
https://github.com/radargun/radargun/blob/master/plugins/infinispan70/src...
>> >> [2]
>> >>
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jdg-radargun-gettota...
>> >>
>> >
>> >
>> > --
>> > Radim Vansa <rvansa(a)redhat.com>
>> > JBoss DataGrid QA
>> >
>> > _______________________________________________
>> > infinispan-dev mailing list
>> > infinispan-dev(a)lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> Cheers,
>> --
>> Mircea Markus
>> Infinispan lead (www.infinispan.org)
>>
>>
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
7:42 a.m.
Will helped me figure this out. The cache was in replicated mode. I thought the cache was
in distributed mode because
Cache.getAdvancedCache().getCacheConfiguration().clustering().hash().numOwners() was
returning 2. I expected it to return the correct number of owners in replicated mode, but
that's not the case.
Sorry for the noise, and for my confusion.
Thanks,
Alan
----- Original Message -----
From: "Alan Field" <afield(a)redhat.com>
To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>
Sent: Tuesday, July 29, 2014 9:53:42 AM
Subject: Re: [infinispan-dev] Cache.size() on distributed caches?
Hey Will,
----- Original Message -----
> From: "William Burns" <mudokonman(a)gmail.com>
> To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>
> Sent: Monday, July 28, 2014 7:51:27 PM
> Subject: Re: [infinispan-dev] Cache.size() on distributed caches?
>
> I am assuming that you were using a shared cache loader without
> passivation? In that case the size method will return all the entries
> in the cache properly (albeit using a large amount of memory)
This is the cache store configuration I had for the cache in question:
<eviction maxEntries="50" />
<persistence passivation="true" >
<leveldbStore xmlns="urn:infinispan:config:store:leveldb:6.0"
implementationType="JAVA"
location="/tmp/ispn-leveldb-jni/data"
expiredLocation="/tmp/ispn-leveldb-jni/expired"
purgeOnStartup="true"
preload="false"
/>
</persistence>
Shared is disabled by default, but passivation was enabled. However, I am
also seeing this behavior with these lines commented out in the cache
configuration.
ISPN-761 talks about handling passivated entries in a cache store, but also
says that size() will only show the number of entries on the local node and
won't check with other nodes in the cluster. That is not what I am seeing. I
have four different machines writing entries to the cache:
04:11:59,574 INFO [org.radargun.stages.cache.RandomDataStage] (main) Filled
cache with String objects totaling 25% of the Java heap
04:11:59,574 INFO [org.radargun.stages.cache.RandomDataStage] (main) Slave 0
wrote 479352 values to the cache with a total size of 958,704 kb;
targetMemoryUse = 1,022,368 kb; countOfWordsInData = 44900952
04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage] (main) Slave 1
wrote 479352 values to the cache with a total size of 958,704 kb;
targetMemoryUse = 1,022,368 kb; countOfWordsInData = 44914319
04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage] (main) Slave 2
wrote 479352 values to the cache with a total size of 958,704 kb;
targetMemoryUse = 1,022,368 kb; countOfWordsInData = 44906729
04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage] (main) Slave 3
wrote 479352 values to the cache with a total size of 958,704 kb;
targetMemoryUse = 1,022,368 kb; countOfWordsInData = 44908687
04:11:59,576 INFO [org.radargun.stages.cache.RandomDataStage] (main) The
cache contains 1917408 values with a total size of 3,834,816 kb
And when I call Cache.size(), I am getting 1917408 which is the total size of
the cache across all 4 nodes:
04:12:12,094 INFO [org.radargun.service.Infinispan53CacheInfo] (main)
cache.size() = 1917408
I am not expecting this to happen, but that is what I am seeing with or
without a cache store.
Thanks,
Alan
>
> To be honest I can't think of a way to get an accurate count of what
> is in the cache unless you are using a shared loader without
> passivation in previous versions. The only way would be if you had no
> gets/writes(passivation issues) or state transfer while you did the
> map/reduce task and had it based on if passivation was enabled or not
> (a bit messy but doable).
>
> The distributed iterator should work irrespective of configuration or
> concurrent operations though.
>
> On Fri, Jul 25, 2014 at 10:47 AM, Alan Field <afield(a)redhat.com> wrote:
> > OK, thanks Mircea!
> >
> > Alan
> >
> > ----- Original Message -----
> >> From: "Mircea Markus" <mmarkus(a)redhat.com>
> >> To: "infinispan -Dev List"
<infinispan-dev(a)lists.jboss.org>
> >> Sent: Friday, July 25, 2014 4:42:50 PM
> >> Subject: Re: [infinispan-dev] Cache.size() on distributed caches?
> >>
> >> That has changed in ISPN 6: https://issues.jboss.org/browse/ISPN-761
> >>
> >>
> >> On Jul 25, 2014, at 15:40, Radim Vansa <rvansa(a)redhat.com> wrote:
> >>
> >> > I don't think RadarGun is much related to that, besides the fact
that
> >> > it
> >> > can be buggy :)
> >> >
> >> > The question is whether the cache.size() behaviour has changed in a
> >> > way
> >> > that i would report full cache size, not just the size of
> >> > container/the
> >> > size of cache store?
> >> >
> >> > Radim
> >> >
> >> > On 07/25/2014 03:54 PM, Alan Field wrote:
> >> >> Hey,
> >> >>
> >> >> I have been looking at adding the ability to get the total size of
a
> >> >> cache
> >> >> in RadarGun. The first implementation I coded used the
distributed
> >> >> iterators in Infinispan 7.[1] I then realized that implementing
> >> >> getTotalSize() method using a distributed executor would allow
the
> >> >> code
> >> >> in versions back to Infinispan 5.2. I have the code written, and
I
> >> >> have
> >> >> been running some Jenkins jobs with Infinispan 6.0.1 Final to
verify
> >> >> that
> >> >> the results are correct.[2] I use the RandomData stage to put data
in
> >> >> the
> >> >> cache. Here is what it writes in the log:
> >> >>
> >> >> 04:11:59,573 INFO [org.radargun.stages.cache.RandomDataStage]
(main)
> >> >> Received responses from all 4 slaves. Durations [0 = 17.04
minutes, 1
> >> >> =
> >> >> 18.36 minutes, 2 = 18.44 minutes, 3 = 18.58 minutes]
> >> >> 04:11:59,574 INFO [org.radargun.stages.cache.RandomDataStage]
(main)
> >> >> --------------------
> >> >> 04:11:59,574 INFO [org.radargun.stages.cache.RandomDataStage]
(main)
> >> >> Filled cache with String objects totaling 25% of the Java heap
> >> >> 04:11:59,574 INFO [org.radargun.stages.cache.RandomDataStage]
(main)
> >> >> Slave 0 wrote 479352 values to the cache with a total size of
958,704
> >> >> kb;
> >> >> targetMemoryUse = 1,022,368 kb; countOfWordsInData = 44900952
> >> >> 04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage]
(main)
> >> >> Slave 1 wrote 479352 values to the cache with a total size of
958,704
> >> >> kb;
> >> >> targetMemoryUse = 1,022,368 kb; countOfWordsInData = 44914319
> >> >> 04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage]
(main)
> >> >> Slave 2 wrote 479352 values to the cache with a total size of
958,704
> >> >> kb;
> >> >> targetMemoryUse = 1,022,368 kb; countOfWordsInData = 44906729
> >> >> 04:11:59,575 INFO [org.radargun.stages.cache.RandomDataStage]
(main)
> >> >> Slave 3 wrote 479352 values to the cache with a total size of
958,704
> >> >> kb;
> >> >> targetMemoryUse = 1,022,368 kb; countOfWordsInData = 44908687
> >> >> 04:11:59,576 INFO [org.radargun.stages.cache.RandomDataStage]
(main)
> >> >> The
> >> >> cache contains 1917408 values with a total size of 3,834,816 kb
> >> >> 04:11:59,576 INFO [org.radargun.stages.cache.RandomDataStage]
(main)
> >> >> 100
> >> >> words were generated with a maximum length of 20 characters
> >> >> 04:11:59,576 INFO [org.radargun.stages.cache.RandomDataStage]
(main)
> >> >> --------------------
> >> >>
> >> >> These are the outputs from my getTotalSize() code:
> >> >>
> >> >> 04:11:59,591 INFO [org.radargun.service.Infinispan53CacheInfo]
> >> >> (main)
> >> >> org.radargun.service.Infinispan53CacheInfo$Cache.getTotalSize()
for
> >> >> cache
> >> >> testCache
> >> >> 04:12:12,094 INFO [org.radargun.service.Infinispan53CacheInfo]
> >> >> (main)
> >> >> cache.size() = 1917408
> >> >> 04:12:26,283 INFO [org.radargun.service.Infinispan53CacheInfo]
> >> >> (main)
> >> >> cache.getAdvancedCache().size() = 1917408
> >> >> 04:12:26,283 INFO [org.radargun.service.Infinispan53CacheInfo]
> >> >> (main)
> >> >>
cache.getAdvancedCache().getCacheConfiguration().clustering().hash().numOwners()
> >> >> = 2
> >> >> 04:12:26,283 INFO [org.radargun.service.Infinispan53CacheInfo]
> >> >> (main)
> >> >> cache.getCacheManager().getMembers().size() = 4
> >> >> 04:12:41,955 INFO [org.radargun.stages.cache.ClearCacheStage]
(main)
> >> >> Cache size = 3834800
> >> >>
> >> >> The "Cache size =" message is from the results of my
distributed
> >> >> executor,
> >> >> and the other messages are informational. These outputs show that
> >> >> calling
> >> >> cache size on a distributed cache returns the size of the entire
> >> >> cache
> >> >> including any passivated entries, not just the size of the cache
on
> >> >> the
> >> >> local node. This breaks the code of my distributed executor, but
> >> >> mostly
> >> >> makes it unnecessary if I can just call cache.size().
> >> >>
> >> >> Is this an expected change in behavior?
> >> >>
> >> >> Thanks,
> >> >> Alan
> >> >>
> >> >> [1]
> >> >>
https://github.com/radargun/radargun/blob/master/plugins/infinispan70/src...
> >> >> [2]
> >> >>
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jdg-radargun-gettota...
> >> >>
> >> >
> >> >
> >> > --
> >> > Radim Vansa <rvansa(a)redhat.com>
> >> > JBoss DataGrid QA
> >> >
> >> > _______________________________________________
> >> > infinispan-dev mailing list
> >> > infinispan-dev(a)lists.jboss.org
> >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> >> Cheers,
> >> --
> >> Mircea Markus
> >> Infinispan lead (www.infinispan.org)
> >>
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev(a)lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
4123
days inactive
4127
days old
infinispan-dev@lists.jboss.org
6 comments
4 participants
participants (4)
-
Alan Field -
Mircea Markus -
Radim Vansa -
William Burns