Hi Martin, The server memory consumption should improve now that we have ISPN-2281 integrated. would it be possible to run the memory consumption tests to see where we are? Begin forwarded message: >> I've just integrated the implementation for ISPN-2281 - great stuff! >> >> >> As this is a very intrusive change it is important to know to measure the gain in memory consumption. i.e. would be great to see some figures on memory usage before this optimisation and after it in your blog post. > ^ QE guys have some memory tests already, why don't we get them to re-run those tests? :) Cheers,

Hey, so I have some good news. I've re-ran my tests for memory consumption where I store 100MB of data and here's the results for in-VM mode and client-server with HotRod: Previously, the memory overhead for HotRod was (can be found at http://infinispan.blogspot.cz/2013/01/infinispan-memory-overhead.html): entry size -> overhead per entry 512B -> ~174B 1kB -> ~178B 10kB -> ~176B 1MB -> ~245B Now it is: 512B -> ~159 (~ -9%) 1kB -> ~159 (~ -11%) 10kB -> ~154 (this is perhaps affected by the low number of entries stored in mem.) 1MB -> ~184 (the same applies here) For in-VM mode it was: 512B -> ~151B 1kB -> ~151B 10kB -> ~155B 1MB -> ~143B Now it is: 512B -> ~150 1kB -> ~150 10kB -> ~149 (this is perhaps affected by the low number of entries stored in mem.) 1MB -> ~143 (the same applies here) Summary: Almost no change for in-VM mode but a nice improvement for the HotRod client.

I can make a blog post once we have this for Memcached and REST. I guess it is not ready yet.

On 7 May 2013, at 15:39, Martin Gencur <mgencur(a)redhat.com&gt; wrote: > I can make a blog post once we have this for Memcached and REST. I guess it is not ready yet. Yes please. Nice work. :) -- Manik Surtani manik(a)jboss.org twitter.com/maniksurtani Platform Architect, JBoss Data Grid http://red.ht/data-grid _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

> You can blog, but bear in mind that the numbers might change. To be precise, the changes that went into Beta1 are: By all means, wait for Galder to *finish* his work before blogging about it. ;)

You can blog, but bear in mind that the numbers might change. To be precise, the changes that went into Beta1 are: - Change from ByteArrayKey (wrapper) to byte[] keys - Change from CacheValue (wrapper) to byte[] value - Version stored directly in Versioned*Entry classes So, this means that we loose two wrapper objects. However, in next release, there will be a wrapper class in the value side (Metadata), but stored internally, in a Metadata*Entry class (all Versioned* classes are heading out...). So, we will gain one wrapper class. But it's worth calculating the memory consumption then, and maybe think if we can optimise this away for Hot Rod.

REST and Memcached will do need to store the entire Metadata object. REST adds content type and Memcached adds both version and flags. Anyway, this is something we can tweak further, but I'm very pleased we have a baseline for beta1, because we can see what the effects of supporting the other endpoint compatibility. Cheers, On May 7, 2013, at 4:55 PM, Manik Surtani <msurtani(a)redhat.com&gt; wrote: > > On 7 May 2013, at 15:39, Martin Gencur <mgencur(a)redhat.com&gt; wrote: > >> I can make a blog post once we have this for Memcached and REST. I guess it is not ready yet. > > Yes please. Nice work. :) > > -- > Manik Surtani > manik(a)jboss.org > twitter.com/maniksurtani > > Platform Architect, JBoss Data Grid > http://red.ht/data-grid > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño galder(a)redhat.com twitter.com/galderz Project Lead, Escalante http://escalante.io Engineer, Infinispan http://infinispan.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Hi Martin, Galder has finalised the remaining bits of ISPN-2281. Is it possible for you to re-run the test to see where we are with the memory consumption? On 13 May 2013, at 10:44, Galder Zamarreño <galder(a)redhat.com&gt; wrote: >> On May 7, 2013, at 4:55 PM, Manik Surtani <msurtani(a)redhat.com&gt; wrote: >> >>> On 7 May 2013, at 15:39, Martin Gencur <mgencur(a)redhat.com&gt; wrote: >>> >>>> I can make a blog post once we have this for Memcached and REST. I guess it is not ready yet. >>> Yes please. Nice work. :) Cheers,

Hi, so I gave it another try with latest Infinispan and Infinispan-server snapshots (HEAD: a901168, resp. bc432fa) . In short, the results are still the same for inVM mode but worse for client-server mode. Of course, I haven't changed anything in the test since last time. This time, I left out the results for 1MB entries because there's high distortion due to the low number of entries stored in a cache (storing 100MB of data). Previously, the results looked better for HotRod compared to the first round of tests I did for ISPN 5.2. Now the results for HotRod are worst of the three measurements, inVM mode remains the same: HotRod (ISPN 5.2): entry size -> overhead per entry 512B -> 174B 1kB -> 178B 10kB -> 176B HotRod (ISPN 5.3-SNAPSHOT, a few weeks ago) 512B -> 159 (~ -9%) 1kB -> 159 (~ -11%) 10kB -> 154 (this is perhaps affected by the low number of entries stored in mem.) HotRod (ISPN 5.3-SNAPSHOT, now) 512B -> 191 1kB -> 191 (measured twice) 10kB -> 186 (looks a bit distorted already) ------------------------ Memcached (ISPN 5.2) 512B -> 184 1kB -> 181 10kB -> 182 Memcached (ISPN 5.3-SNAPSHOT) 512 -> 228 1kB -> 227 10kB -> 235 -------------------------------- REST (ISPN 5.2) 512B -> 208 1kB -> 205 10kB -> 206 REST (ISPN 5.3-SNAPSHOT) 512 -> 247 1kB -> 247 10kB -> 251 ------------------------------------ inVM (ISPN 5.2) 512B -> 151 1kB -> 151 10kB -> 155 inVM (ISPN 5.3-SNAPSHOT) 512 -> 150 1kB -> 150 10kB -> 150 Martin Dne 23.5.2013 18:13, Mircea Markus napsal(a): > Hi Martin, > > Galder has finalised the remaining bits of ISPN-2281. Is it possible for you to re-run the test to see where we are with the memory consumption? > > On 13 May 2013, at 10:44, Galder Zamarreño > <galder(a)redhat.com&gt; > wrote: > > >>> On May 7, 2013, at 4:55 PM, Manik Surtani <msurtani(a)redhat.com&gt; >>> wrote: >>> >>> >>>> On 7 May 2013, at 15:39, Martin Gencur <mgencur(a)redhat.com&gt; >>>> wrote: >>>> >>>> >>>>> I can make a blog post once we have this for Memcached and REST. I guess it is not ready yet. >>>>> >>>> Yes please. Nice work. :) >>>> > Cheers, > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

All, I've managed to replicate this locally and I'm now looking at the heap dumps. Cheers, On May 27, 2013, at 5:34 PM, Galder Zamarreño <galder(a)redhat.com&gt; wrote: > Hmmmm, not the expected results. There's no way we should be consuming more memory per entry. It should definitely be less, particularly for Hot Rod and Memcached and about the same for REST. The fact that all of them grow seems to be there's an issue in the core impl. > > @Martin, can you generate a heap dump, say for Hot Rod (with 512b entries) at the end of the test (when the cache is still populated)? > > Cheers, > > On May 24, 2013, at 2:50 PM, Martin Gencur <mgencur(a)redhat.com&gt; wrote: > >> Hi, >> so I gave it another try with latest Infinispan and Infinispan-server snapshots (HEAD: a901168, resp. bc432fa) . In short, the results are still the same for inVM mode but worse for client-server mode. Of course, I haven't changed anything in the test since last time. >> >> This time, I left out the results for 1MB entries because there's high distortion due to the low number of entries stored in a cache (storing 100MB of data). Previously, the results looked better for HotRod compared to the first round of tests I did for ISPN 5.2. Now the results for HotRod are worst of the three measurements, inVM mode remains the same: >> >> HotRod (ISPN 5.2): >> entry size -> overhead per entry >> 512B -> 174B >> 1kB -> 178B >> 10kB -> 176B >> >> HotRod (ISPN 5.3-SNAPSHOT, a few weeks ago) >> 512B -> 159 (~ -9%) >> 1kB -> 159 (~ -11%) >> 10kB -> 154 (this is perhaps affected by the low number of entries stored in mem.) >> >> HotRod (ISPN 5.3-SNAPSHOT, now) >> 512B -> 191 >> 1kB -> 191 (measured twice) >> 10kB -> 186 (looks a bit distorted already) >> >> ------------------------ >> >> Memcached (ISPN 5.2) >> 512B -> 184 >> 1kB -> 181 >> 10kB -> 182 >> >> Memcached (ISPN 5.3-SNAPSHOT) >> 512 -> 228 >> 1kB -> 227 >> 10kB -> 235 >> >> -------------------------------- >> >> REST (ISPN 5.2) >> 512B -> 208 >> 1kB -> 205 >> 10kB -> 206 >> >> REST (ISPN 5.3-SNAPSHOT) >> 512 -> 247 >> 1kB -> 247 >> 10kB -> 251 >> >> ------------------------------------ >> >> inVM (ISPN 5.2) >> 512B -> 151 >> 1kB -> 151 >> 10kB -> 155 >> >> inVM (ISPN 5.3-SNAPSHOT) >> 512 -> 150 >> 1kB -> 150 >> 10kB -> 150 >> >> >> Martin >> >> >> Dne 23.5.2013 18:13, Mircea Markus napsal(a): >>> Hi Martin, >>> >>> Galder has finalised the remaining bits of ISPN-2281. Is it possible for you to re-run the test to see where we are with the memory consumption? >>> >>> On 13 May 2013, at 10:44, Galder Zamarreño >>> <galder(a)redhat.com&gt; >>> wrote: >>> >>> >>>>> On May 7, 2013, at 4:55 PM, Manik Surtani <msurtani(a)redhat.com&gt; >>>>> wrote: >>>>> >>>>> >>>>>> On 7 May 2013, at 15:39, Martin Gencur <mgencur(a)redhat.com&gt; >>>>>> wrote: >>>>>> >>>>>> >>>>>>> I can make a blog post once we have this for Memcached and REST. I guess it is not ready yet. >>>>>>> >>>>>> Yes please. Nice work. :) >>>>>> >>> Cheers, >>> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Galder Zamarreño > galder(a)redhat.com > twitter.com/galderz > > Project Lead, Escalante > http://escalante.io > > Engineer, Infinispan > http://infinispan.org > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño galder(a)redhat.com twitter.com/galderz Project Lead, Escalante http://escalante.io Engineer, Infinispan http://infinispan.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Cheers Dan On Tue, May 28, 2013 at 4:20 PM, Galder Zamarreño <galder(a)redhat.com&gt; wrote: Basically, the difference between Hot Rod then and now is: 32 bytes, which is: new metadata object, a reference to it, and 2 unusued longs for immortal entries. With embedded metadata (~191 bytes per entry): https://dl.dropboxusercontent.com/u/6148072/with-embedded.png With versioned entries (~159 bytes per entry): https://dl.dropboxusercontent.com/u/6148072/with-versioned.png And there's more: we have internal cache entries that have a reference to internal cache values, per entry. This is useful for some cases (cache stores…etc), but this extra reference to the value, plus the value object itself, is 16 bytes (how useful is it really to have a separate value instance to keep just the value? Needs reassessing its usefulness...). +1 to remove InternalCacheValue (I thought about storing InternalCacheValues directly in the DataContainer, but it would probably create too much garbage to create InternalCacheEntries all the time).

So really, I think the minimum Hot Rod overhead we should aim for is ~143 bytes. If each server module could define what the ICE class (well, classes to cover all immortal, mortal…etc cases) to use, which is purely designed for their servers (i.e. hot rod just needs: value + version; memcached needs: value + version + flags), we could get to this level… You mean embed the metadata fields directly in the ICE? Wouldn't that get us back to what we had before introducing the Metadata interface?

You still want the metadata to be passed from the client, but for those specialised use cases in Infinispan, we could have a mapping between the metadata type and the type of ICEs created… If HotRod knows it always needs a ServerEntryVersion, it could implement EntryVersion directly in HotRodMetadata implementation that implements EntryVersion directly (version() returns this). This could even be combined with your idea, so we could have a HotRodInternalCacheEntry that implements InternalCacheEntry, Metadata, and EntryVersion :)

If anyone has any other ideas, shout now! :) Cheers, On May 28, 2013, at 3:07 PM, Galder Zamarreño <galder(a)redhat.com&gt; wrote: > All, I've managed to replicate this locally and I'm now looking at the heap dumps. > > Cheers, > > On May 27, 2013, at 5:34 PM, Galder Zamarreño <galder(a)redhat.com&gt; wrote: > >> Hmmmm, not the expected results. There's no way we should be consuming more memory per entry. It should definitely be less, particularly for Hot Rod and Memcached and about the same for REST. The fact that all of them grow seems to be there's an issue in the core impl. >> >> @Martin, can you generate a heap dump, say for Hot Rod (with 512b entries) at the end of the test (when the cache is still populated)? >> >> Cheers, >> >> On May 24, 2013, at 2:50 PM, Martin Gencur <mgencur(a)redhat.com&gt; wrote: >> >>> Hi, >>> so I gave it another try with latest Infinispan and Infinispan-server snapshots (HEAD: a901168, resp. bc432fa) . In short, the results are still the same for inVM mode but worse for client-server mode. Of course, I haven't changed anything in the test since last time. >>> >>> This time, I left out the results for 1MB entries because there's high distortion due to the low number of entries stored in a cache (storing 100MB of data). Previously, the results looked better for HotRod compared to the first round of tests I did for ISPN 5.2. Now the results for HotRod are worst of the three measurements, inVM mode remains the same: >>> >>> HotRod (ISPN 5.2): >>> entry size -> overhead per entry >>> 512B -> 174B >>> 1kB -> 178B >>> 10kB -> 176B >>> >>> HotRod (ISPN 5.3-SNAPSHOT, a few weeks ago) >>> 512B -> 159 (~ -9%) >>> 1kB -> 159 (~ -11%) >>> 10kB -> 154 (this is perhaps affected by the low number of entries stored in mem.) >>> >>> HotRod (ISPN 5.3-SNAPSHOT, now) >>> 512B -> 191 >>> 1kB -> 191 (measured twice) >>> 10kB -> 186 (looks a bit distorted already) >>> >>> ------------------------ >>> >>> Memcached (ISPN 5.2) >>> 512B -> 184 >>> 1kB -> 181 >>> 10kB -> 182 >>> >>> Memcached (ISPN 5.3-SNAPSHOT) >>> 512 -> 228 >>> 1kB -> 227 >>> 10kB -> 235 >>> >>> -------------------------------- >>> >>> REST (ISPN 5.2) >>> 512B -> 208 >>> 1kB -> 205 >>> 10kB -> 206 >>> >>> REST (ISPN 5.3-SNAPSHOT) >>> 512 -> 247 >>> 1kB -> 247 >>> 10kB -> 251 >>> >>> ------------------------------------ >>> >>> inVM (ISPN 5.2) >>> 512B -> 151 >>> 1kB -> 151 >>> 10kB -> 155 >>> >>> inVM (ISPN 5.3-SNAPSHOT) >>> 512 -> 150 >>> 1kB -> 150 >>> 10kB -> 150 >>> >>> >>> Martin >>> >>> >>> Dne 23.5.2013 18:13, Mircea Markus napsal(a): >>>> Hi Martin, >>>> >>>> Galder has finalised the remaining bits of ISPN-2281. Is it possible for you to re-run the test to see where we are with the memory consumption? >>>> >>>> On 13 May 2013, at 10:44, Galder Zamarreño >>>> <galder(a)redhat.com&gt; >>>> wrote: >>>> >>>> >>>>>> On May 7, 2013, at 4:55 PM, Manik Surtani <msurtani(a)redhat.com&gt; >>>>>> wrote: >>>>>> >>>>>> >>>>>>> On 7 May 2013, at 15:39, Martin Gencur <mgencur(a)redhat.com&gt; >>>>>>> wrote: >>>>>>> >>>>>>> >>>>>>>> I can make a blog post once we have this for Memcached and REST. I guess it is not ready yet. >>>>>>>> >>>>>>> Yes please. Nice work. :) >>>>>>> >>>> Cheers, >>>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> -- >> Galder Zamarreño >> galder(a)redhat.com >> twitter.com/galderz >> >> Project Lead, Escalante >> http://escalante.io >> >> Engineer, Infinispan >> http://infinispan.org >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Galder Zamarreño > galder(a)redhat.com > twitter.com/galderz > > Project Lead, Escalante > http://escalante.io > > Engineer, Infinispan > http://infinispan.org > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño galder(a)redhat.com twitter.com/galderz Project Lead, Escalante http://escalante.io Engineer, Infinispan http://infinispan.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev