On 04/18/2013 09:35 PM, Manik Surtani wrote: > Guys, > > Based on some recent micro benchmarks I've been doing, I've seen: > > MapStressTest configuration: capacity 100000, test running time 60 seconds > Testing mixed read/write performance with capacity 100,000, keys 300,000, concurrency level 32, threads 12, read:write ratio 0:1 > Container CHM Ops/s 21,165,771.67 Gets/s 0.00 Puts/s 21,165,771.67 HitRatio 100.00 Size 262,682 stdDev 77,540.73 > Container CHMV8 Ops/s 33,513,807.09 Gets/s 0.00 Puts/s 33,513,807.09 HitRatio 100.00 Size 262,682 stdDev 77,540.73 > > So under high concurrency (12 threads, on my workstation with 12 hardware threads - so all threads are always working), we see that Infinispan's CHMv8 implementation is 50% faster than JDK6's CHM implementation when doing puts. > > We use a fair number of CHMs all over Infinispan's codebase. By default, these are all JDK-provided CHMs. But we have the option to switch to our CHMv8 implementation by passing in -Dinfinispan.unsafe.allow_jdk8_chm=true. > > The question is, should this be the default? Thoughts, opinions? The JDK's concurrency code - especially CHM - changes all the time. You'd be very well-served, in my opinion, to go with something like CHMv8 just because you could be so much more sure that you'll have more consistent (and possibly better, but definitely more consistent) performance across all JVMs, instead of being at the mercy of whatever particular implementation happens to run on whatever JVM. -- - DML _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Why not. Only doubt I'd have is that other usages of the CHM are - I guess - services registry and similar configuration tools, for which write performance is irrelevant: your test measured puts, are there drawbacks on gets or memory usage? Recently you changed all (most?) CHM creations to use a consistent factory, maybe we could improve on that by actually using a couple of factories which differentiate on the intended usage of the CHM: for example some maps who change very infrequently - mostly during boot or reconfiguration, maybe even topology change - could be better served by a non concurrent structure using copy-on-wrtite. Sanne On 19 Apr 2013 08:48, "Dan Berindei" <dan.berindei(a)gmail.com&gt; wrote: > +1 to make CHMv8 the default on JDK6 and JDK7 > > But I'm not convinced we should make it the default for JDK8 - even > though we don't know exactly what we're getting with the JDK's > implementation. > > > On Fri, Apr 19, 2013 at 5:39 AM, David M. Lloyd <david.lloyd(a)redhat.com&gt;wrote: > >> On 04/18/2013 09:35 PM, Manik Surtani wrote: >> > Guys, >> > >> > Based on some recent micro benchmarks I've been doing, I've seen: >> > >> > MapStressTest configuration: capacity 100000, test running time 60 >> seconds >> > Testing mixed read/write performance with capacity 100,000, keys >> 300,000, concurrency level 32, threads 12, read:write ratio 0:1 >> > Container CHM Ops/s 21,165,771.67 Gets/s 0.00 Puts/s >> 21,165,771.67 HitRatio 100.00 Size 262,682 stdDev 77,540.73 >> > Container CHMV8 Ops/s 33,513,807.09 Gets/s 0.00 Puts/s >> 33,513,807.09 HitRatio 100.00 Size 262,682 stdDev 77,540.73 >> > >> > So under high concurrency (12 threads, on my workstation with 12 >> hardware threads - so all threads are always working), we see that >> Infinispan's CHMv8 implementation is 50% faster than JDK6's CHM >> implementation when doing puts. >> > >> > We use a fair number of CHMs all over Infinispan's codebase. By >> default, these are all JDK-provided CHMs. But we have the option to switch >> to our CHMv8 implementation by passing in >> -Dinfinispan.unsafe.allow_jdk8_chm=true. >> > >> > The question is, should this be the default? Thoughts, opinions? >> >> The JDK's concurrency code - especially CHM - changes all the time. >> You'd be very well-served, in my opinion, to go with something like >> CHMv8 just because you could be so much more sure that you'll have more >> consistent (and possibly better, but definitely more consistent) >> performance across all JVMs, instead of being at the mercy of whatever >> particular implementation happens to run on whatever JVM. >> >> >> -- >> - DML >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 19 April 2013 10:37, Dan Berindei <dan.berindei(a)gmail.com&gt; wrote: > Testing mixed read/write performance with capacity 100000, keys 300000, > concurrency level 32, threads 12, read:write ratio 99:1 > Container CHM Ops/s 5178894.77 Gets/s 5127105.82 Puts/s > 51788.95 HitRatio 86.23 Size 177848 stdDev 60896.42 > Container CHMV8 Ops/s 5768824.37 Gets/s 5711136.13 Puts/s > 57688.24 HitRatio 84.72 Size 171964 stdDev 60249.99 Nice, thanks. > > The test is probably limited by the 1% writes, but I think it does show that > reads in CHMV8 are not slower than reads in OpenJDK7's CHM. > I haven't measured it, but the memory footprint should also be better, > because it doesn't use segments any more. > > AFAIK the memoryCHMV8 also uses copy-on-write at the bucket level, but we > could definitely do a pure read test with a HashMap to see how big the > performance difference is. By copy-on-write I didn't mean on the single elements, but on the whole map instance: private volatile HashMap configuration; synchronized addConfigurationProperty(String, String) { HashMap newcopy = new HashMap( configuration ): newcopy.put(..); configuration = newcopy; } Of course that is never going to scale for writes, but if writes stop at runtime after all services are started I would expect that the simplicity of the non-threadsafe HashMap should have some benefit over CHM{whatever}, or it would have been removed already?

> > > > > On Fri, Apr 19, 2013 at 11:07 AM, Sanne Grinovero <sanne(a)infinispan.org&gt; > wrote: >> >> Why not. Only doubt I'd have is that other usages of the CHM are - I guess >> - services registry and similar configuration tools, for which write >> performance is irrelevant: your test measured puts, are there drawbacks on >> gets or memory usage? >> >> Recently you changed all (most?) CHM creations to use a consistent >> factory, maybe we could improve on that by actually using a couple of >> factories which differentiate on the intended usage of the CHM: for example >> some maps who change very infrequently - mostly during boot or >> reconfiguration, maybe even topology change - could be better served by a >> non concurrent structure using copy-on-wrtite. >> >> Sanne >> >> On 19 Apr 2013 08:48, "Dan Berindei" <dan.berindei(a)gmail.com&gt; wrote: >>> >>> +1 to make CHMv8 the default on JDK6 and JDK7 >>> >>> But I'm not convinced we should make it the default for JDK8 - even >>> though we don't know exactly what we're getting with the JDK's >>> implementation. >>> >>> >>> On Fri, Apr 19, 2013 at 5:39 AM, David M. Lloyd < david.lloyd(a)redhat.com&gt; >>> wrote: >>>> >>>> On 04/18/2013 09:35 PM, Manik Surtani wrote: >>>> > Guys, >>>> > >>>> > Based on some recent micro benchmarks I've been doing, I've seen: >>>> > >>>> > MapStressTest configuration: capacity 100000, test running time 60 >>>> > seconds >>>> > Testing mixed read/write performance with capacity 100,000, keys >>>> > 300,000, concurrency level 32, threads 12, read:write ratio 0:1 >>>> > Container CHM Ops/s 21,165,771.67 Gets/s 0.00 Puts/s >>>> > 21,165,771.67 HitRatio 100.00 Size 262,682 stdDev 77,540.73 >>>> > Container CHMV8 Ops/s 33,513,807.09 Gets/s 0.00 Puts/s >>>> > 33,513,807.09 HitRatio 100.00 Size 262,682 stdDev 77,540.73 >>>> > >>>> > So under high concurrency (12 threads, on my workstation with 12 >>>> > hardware threads - so all threads are always working), we see that >>>> > Infinispan's CHMv8 implementation is 50% faster than JDK6's CHM >>>> > implementation when doing puts. >>>> > >>>> > We use a fair number of CHMs all over Infinispan's codebase. By >>>> > default, these are all JDK-provided CHMs. But we have the option to switch >>>> > to our CHMv8 implementation by passing in >>>> > -Dinfinispan.unsafe.allow_jdk8_chm=true. >>>> > >>>> > The question is, should this be the default? Thoughts, opinions? >>>> >>>> The JDK's concurrency code - especially CHM - changes all the time. >>>> You'd be very well-served, in my opinion, to go with something like >>>> CHMv8 just because you could be so much more sure that you'll have more >>>> consistent (and possibly better, but definitely more consistent) >>>> performance across all JVMs, instead of being at the mercy of whatever >>>> particular implementation happens to run on whatever JVM. >>>> >>>> >>>> -- >>>> - DML >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 19 April 2013 11:10, Dan Berindei <dan.berindei(a)gmail.com&gt; wrote: > > > > On Fri, Apr 19, 2013 at 12:58 PM, Sanne Grinovero <sanne(a)infinispan.org&gt; > wrote: >> >> On 19 April 2013 10:37, Dan Berindei <dan.berindei(a)gmail.com&gt; wrote: >>> Testing mixed read/write performance with capacity 100000, keys 300000, >>> concurrency level 32, threads 12, read:write ratio 99:1 >>> Container CHM Ops/s 5178894.77 Gets/s 5127105.82 Puts/s >>> 51788.95 HitRatio 86.23 Size 177848 stdDev 60896.42 >>> Container CHMV8 Ops/s 5768824.37 Gets/s 5711136.13 Puts/s >>> 57688.24 HitRatio 84.72 Size 171964 stdDev 60249.99 >> >> Nice, thanks. >>> >>> The test is probably limited by the 1% writes, but I think it does show >>> that >>> reads in CHMV8 are not slower than reads in OpenJDK7's CHM. >>> I haven't measured it, but the memory footprint should also be better, >>> because it doesn't use segments any more. >>> >>> AFAIK the memoryCHMV8 also uses copy-on-write at the bucket level, but >>> we >>> could definitely do a pure read test with a HashMap to see how big the >>> performance difference is. >> >> By copy-on-write I didn't mean on the single elements, but on the >> whole map instance: >> >> private volatile HashMap configuration; >> >> synchronized addConfigurationProperty(String, String) { >> HashMap newcopy = new HashMap( configuration ): >> newcopy.put(..); >> configuration = newcopy; >> } >> >> Of course that is never going to scale for writes, but if writes stop >> at runtime after all services are started I would expect that the >> simplicity of the non-threadsafe HashMap should have some benefit over >> CHM{whatever}, or it would have been removed already? >> > > Right, we should be able to tell whether that's worth doing with a pure read > test with a CHMV8 and a HashMap :) IFF you find out CHMV8 is as good as HashMap for read only, you have two options: - ask the JDK team to drop the HashMap code as it's no longer needed - fix your benchmark :-P In other words, I'd consider it highly surprising and suspicious (still interesting though!)

On 19 April 2013 13:52, David M. Lloyd <david.lloyd(a)redhat.com&gt; wrote: > On 04/19/2013 05:17 AM, Sanne Grinovero wrote: >> On 19 April 2013 11:10, Dan Berindei <dan.berindei(a)gmail.com&gt; wrote: >>> >>> >>> >>> On Fri, Apr 19, 2013 at 12:58 PM, Sanne Grinovero <sanne(a)infinispan.org&gt; >>> wrote: >>>> >>>> On 19 April 2013 10:37, Dan Berindei <dan.berindei(a)gmail.com&gt; wrote: >>>>> Testing mixed read/write performance with capacity 100000, keys 300000, >>>>> concurrency level 32, threads 12, read:write ratio 99:1 >>>>> Container CHM Ops/s 5178894.77 Gets/s 5127105.82 Puts/s >>>>> 51788.95 HitRatio 86.23 Size 177848 stdDev 60896.42 >>>>> Container CHMV8 Ops/s 5768824.37 Gets/s 5711136.13 Puts/s >>>>> 57688.24 HitRatio 84.72 Size 171964 stdDev 60249.99 >>>> >>>> Nice, thanks. >>>>> >>>>> The test is probably limited by the 1% writes, but I think it does show >>>>> that >>>>> reads in CHMV8 are not slower than reads in OpenJDK7's CHM. >>>>> I haven't measured it, but the memory footprint should also be better, >>>>> because it doesn't use segments any more. >>>>> >>>>> AFAIK the memoryCHMV8 also uses copy-on-write at the bucket level, but >>>>> we >>>>> could definitely do a pure read test with a HashMap to see how big the >>>>> performance difference is. >>>> >>>> By copy-on-write I didn't mean on the single elements, but on the >>>> whole map instance: >>>> >>>> private volatile HashMap configuration; >>>> >>>> synchronized addConfigurationProperty(String, String) { >>>> HashMap newcopy = new HashMap( configuration ): >>>> newcopy.put(..); >>>> configuration = newcopy; >>>> } >>>> >>>> Of course that is never going to scale for writes, but if writes stop >>>> at runtime after all services are started I would expect that the >>>> simplicity of the non-threadsafe HashMap should have some benefit over >>>> CHM{whatever}, or it would have been removed already? >>>> >>> >>> Right, we should be able to tell whether that's worth doing with a pure read >>> test with a CHMV8 and a HashMap :) >> >> IFF you find out CHMV8 is as good as HashMap for read only, you have >> two options: >> - ask the JDK team to drop the HashMap code as it's no longer needed >> - fix your benchmark :-P >> >> In other words, I'd consider it highly surprising and suspicious >> (still interesting though!) > > It's not as surprising as you think. On x86, volatile reads are the > same as regular reads (not counting some possible reordering magic). So > if a CHM read is a hash, an array access, and a list traversal, and so > is HM (and I believe this is true though I'd have to review the code > again to be sure), I'd expect very similar execution performance on > read. I think some of the anti-collision features in V8 might come into > play under some circumstances though which might affect performance in a > negative way (wrt the constant big-O component) but overall in a > positive way (by turning the linear big-O component into a logarithmic one). Thanks David. I know about the cost of a volatile read, what I'm referring to is that I would expect the non-concurrent Maps to generally contain some simpler code than a conccurrent one. If this was not the case, why would any JDK team maintain two different implementations? That's why I would consider it surprising if it turned out that the CHMV8 was superior over a regular one on all fronts: there certainly is some scenario in which the regular one would be a more appropriate choice, which directly proofs that blindly replacing all usages in a large project is not optimal. Of course, it might be close to optimal..

Right. If we have anywhere a map that's initialized from a single thread and then accessed only for reading from many threads, it probably makes sense to use a HashMap and wrap it in an UnmodifiableMap. But if it can be written from multiple threads as well, I think we should use a CHMV8. BTW, the HashMap implementation in OpenJDK 1.7 seems to have some anti-collision features (a VM-dependent hash code generator for Strings), but our version of CHMV8 doesn't. Perhaps we need to upgrade to the latest CHMV8 version? On Fri, Apr 19, 2013 at 4:32 PM, David M. Lloyd <david.lloyd(a)redhat.com&gt; wrote: > > On 04/19/2013 08:22 AM, Sanne Grinovero wrote: > > On 19 April 2013 13:52, David M. Lloyd <david.lloyd(a)redhat.com&gt; wrote: > >> On 04/19/2013 05:17 AM, Sanne Grinovero wrote: > >>> On 19 April 2013 11:10, Dan Berindei <dan.berindei(a)gmail.com&gt; wrote: > >>>> > >>>> > >>>> > >>>> On Fri, Apr 19, 2013 at 12:58 PM, Sanne Grinovero > >>>> <sanne(a)infinispan.org&gt; > >>>> wrote: > >>>>> > >>>>> On 19 April 2013 10:37, Dan Berindei <dan.berindei(a)gmail.com&gt; wrote: > >>>>>> Testing mixed read/write performance with capacity 100000, keys > >>>>>> 300000, > >>>>>> concurrency level 32, threads 12, read:write ratio 99:1 > >>>>>> Container CHM Ops/s 5178894.77 Gets/s 5127105.82 Puts/s > >>>>>> 51788.95 HitRatio 86.23 Size 177848 stdDev 60896.42 > >>>>>> Container CHMV8 Ops/s 5768824.37 Gets/s 5711136.13 Puts/s > >>>>>> 57688.24 HitRatio 84.72 Size 171964 stdDev 60249.99 > >>>>> > >>>>> Nice, thanks. > >>>>>> > >>>>>> The test is probably limited by the 1% writes, but I think it does > >>>>>> show > >>>>>> that > >>>>>> reads in CHMV8 are not slower than reads in OpenJDK7's CHM. > >>>>>> I haven't measured it, but the memory footprint should also be > >>>>>> better, > >>>>>> because it doesn't use segments any more. > >>>>>> > >>>>>> AFAIK the memoryCHMV8 also uses copy-on-write at the bucket level, > >>>>>> but > >>>>>> we > >>>>>> could definitely do a pure read test with a HashMap to see how big > >>>>>> the > >>>>>> performance difference is. > >>>>> > >>>>> By copy-on-write I didn't mean on the single elements, but on the > >>>>> whole map instance: > >>>>> > >>>>> private volatile HashMap configuration; > >>>>> > >>>>> synchronized addConfigurationProperty(String, String) { > >>>>> HashMap newcopy = new HashMap( configuration ): > >>>>> newcopy.put(..); > >>>>> configuration = newcopy; > >>>>> } > >>>>> > >>>>> Of course that is never going to scale for writes, but if writes > >>>>> stop > >>>>> at runtime after all services are started I would expect that the > >>>>> simplicity of the non-threadsafe HashMap should have some benefit > >>>>> over > >>>>> CHM{whatever}, or it would have been removed already? > >>>>> > >>>> > >>>> Right, we should be able to tell whether that's worth doing with a > >>>> pure read > >>>> test with a CHMV8 and a HashMap :) > >>> > >>> IFF you find out CHMV8 is as good as HashMap for read only, you have > >>> two options: > >>> - ask the JDK team to drop the HashMap code as it's no longer > >>> needed > >>> - fix your benchmark :-P > >>> > >>> In other words, I'd consider it highly surprising and suspicious > >>> (still interesting though!) > >> > >> It's not as surprising as you think. On x86, volatile reads are the > >> same as regular reads (not counting some possible reordering magic). > >> So > >> if a CHM read is a hash, an array access, and a list traversal, and so > >> is HM (and I believe this is true though I'd have to review the code > >> again to be sure), I'd expect very similar execution performance on > >> read. I think some of the anti-collision features in V8 might come > >> into > >> play under some circumstances though which might affect performance in > >> a > >> negative way (wrt the constant big-O component) but overall in a > >> positive way (by turning the linear big-O component into a logarithmic > >> one). > > > > Thanks David. I know about the cost of a volatile read, what I'm > > referring to > > is that I would expect the non-concurrent Maps to generally contain some > > simpler code than a conccurrent one. If this was not the case, > > why would any JDK team maintain two different implementations? > > That's why I would consider it surprising if it turned out that the > > CHMV8 was > > superior over a regular one on all fronts: there certainly is some > > scenario in which the regular one would be a more appropriate choice, > > which directly proofs that blindly replacing all usages in a large > > project > > is not optimal. Of course, it might be close to optimal.. > > You are right, it is not superior on all fronts. It is definitely > similar in terms of read, but writes will have a substantially higher > cost, involving (at the very least) multiple volatile writes which are > orders of magnitude more expensive than normal writes (on Intel they > have the costly impact of memory fence instructions). So I don't think > anyone will want to drop HashMap any time soon. :-) > > -- > - DML > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

We also have been toying with the idea to hash each key only once, instead of both with the consistent hash (to assign the node owner) and once in the CHM backing the datacontainer.

I doubt we need the datacontainer to implement Map at all,

but at least if we go this way we don't want the hash to be affected by the VM instance or different nodes won't agree on the expected owner ;-) Also there where reports of it having a very bad impact on performance, I'm not sure if they where resolved yet, or are going to be resolved at all as it was important for security reasons. On 22 April 2013 12:19, Dan Berindei <dan.berindei(a)gmail.com&gt; wrote: > Right. If we have anywhere a map that's initialized from a single thread and > then accessed only for reading from many threads, it probably makes sense to > use a HashMap and wrap it in an UnmodifiableMap. But if it can be written > from multiple threads as well, I think we should use a CHMV8. > > BTW, the HashMap implementation in OpenJDK 1.7 seems to have some > anti-collision features (a VM-dependent hash code generator for Strings), > but our version of CHMV8 doesn't. Perhaps we need to upgrade to the latest > CHMV8 version? > > > > On Fri, Apr 19, 2013 at 4:32 PM, David M. Lloyd <david.lloyd(a)redhat.com&gt; > wrote: >> >> On 04/19/2013 08:22 AM, Sanne Grinovero wrote: >>> On 19 April 2013 13:52, David M. Lloyd <david.lloyd(a)redhat.com&gt; wrote: >>>> On 04/19/2013 05:17 AM, Sanne Grinovero wrote: >>>>> On 19 April 2013 11:10, Dan Berindei <dan.berindei(a)gmail.com&gt; wrote: >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Apr 19, 2013 at 12:58 PM, Sanne Grinovero >>>>>> <sanne(a)infinispan.org&gt; >>>>>> wrote: >>>>>>> >>>>>>> On 19 April 2013 10:37, Dan Berindei <dan.berindei(a)gmail.com&gt; wrote: >>>>>>>> Testing mixed read/write performance with capacity 100000, keys >>>>>>>> 300000, >>>>>>>> concurrency level 32, threads 12, read:write ratio 99:1 >>>>>>>> Container CHM Ops/s 5178894.77 Gets/s 5127105.82 Puts/s >>>>>>>> 51788.95 HitRatio 86.23 Size 177848 stdDev 60896.42 >>>>>>>> Container CHMV8 Ops/s 5768824.37 Gets/s 5711136.13 Puts/s >>>>>>>> 57688.24 HitRatio 84.72 Size 171964 stdDev 60249.99 >>>>>>> >>>>>>> Nice, thanks. >>>>>>>> >>>>>>>> The test is probably limited by the 1% writes, but I think it does >>>>>>>> show >>>>>>>> that >>>>>>>> reads in CHMV8 are not slower than reads in OpenJDK7's CHM. >>>>>>>> I haven't measured it, but the memory footprint should also be >>>>>>>> better, >>>>>>>> because it doesn't use segments any more. >>>>>>>> >>>>>>>> AFAIK the memoryCHMV8 also uses copy-on-write at the bucket level, >>>>>>>> but >>>>>>>> we >>>>>>>> could definitely do a pure read test with a HashMap to see how big >>>>>>>> the >>>>>>>> performance difference is. >>>>>>> >>>>>>> By copy-on-write I didn't mean on the single elements, but on the >>>>>>> whole map instance: >>>>>>> >>>>>>> private volatile HashMap configuration; >>>>>>> >>>>>>> synchronized addConfigurationProperty(String, String) { >>>>>>> HashMap newcopy = new HashMap( configuration ): >>>>>>> newcopy.put(..); >>>>>>> configuration = newcopy; >>>>>>> } >>>>>>> >>>>>>> Of course that is never going to scale for writes, but if writes >>>>>>> stop >>>>>>> at runtime after all services are started I would expect that the >>>>>>> simplicity of the non-threadsafe HashMap should have some benefit >>>>>>> over >>>>>>> CHM{whatever}, or it would have been removed already? >>>>>>> >>>>>> >>>>>> Right, we should be able to tell whether that's worth doing with a >>>>>> pure read >>>>>> test with a CHMV8 and a HashMap :) >>>>> >>>>> IFF you find out CHMV8 is as good as HashMap for read only, you have >>>>> two options: >>>>> - ask the JDK team to drop the HashMap code as it's no longer >>>>> needed >>>>> - fix your benchmark :-P >>>>> >>>>> In other words, I'd consider it highly surprising and suspicious >>>>> (still interesting though!) >>>> >>>> It's not as surprising as you think. On x86, volatile reads are the >>>> same as regular reads (not counting some possible reordering magic). >>>> So >>>> if a CHM read is a hash, an array access, and a list traversal, and so >>>> is HM (and I believe this is true though I'd have to review the code >>>> again to be sure), I'd expect very similar execution performance on >>>> read. I think some of the anti-collision features in V8 might come >>>> into >>>> play under some circumstances though which might affect performance in >>>> a >>>> negative way (wrt the constant big-O component) but overall in a >>>> positive way (by turning the linear big-O component into a logarithmic >>>> one). >>> >>> Thanks David. I know about the cost of a volatile read, what I'm >>> referring to >>> is that I would expect the non-concurrent Maps to generally contain some >>> simpler code than a conccurrent one. If this was not the case, >>> why would any JDK team maintain two different implementations? >>> That's why I would consider it surprising if it turned out that the >>> CHMV8 was >>> superior over a regular one on all fronts: there certainly is some >>> scenario in which the regular one would be a more appropriate choice, >>> which directly proofs that blindly replacing all usages in a large >>> project >>> is not optimal. Of course, it might be close to optimal.. >> >> You are right, it is not superior on all fronts. It is definitely >> similar in terms of read, but writes will have a substantially higher >> cost, involving (at the very least) multiple volatile writes which are >> orders of magnitude more expensive than normal writes (on Intel they >> have the costly impact of memory fence instructions). So I don't think >> anyone will want to drop HashMap any time soon. :-) >> >> -- >> - DML >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On Mon, Apr 22, 2013 at 2:37 PM, Sanne Grinovero <sanne(a)infinispan.org&gt; wrote: We also have been toying with the idea to hash each key only once, instead of both with the consistent hash (to assign the node owner) and once in the CHM backing the datacontainer. I doubt we need the datacontainer to implement Map at all, but at least if we go this way we don't want the hash to be affected by the VM instance or different nodes won't agree on the expected owner ;-) For consistent hashing it would probably be better to cache the hash after applying MurmurHash to it anyway. So we could in theory hack our CHMV8 to use a cached hash code computed with MurmurHash and a cluster-specific salt.

Also there where reports of it having a very bad impact on performance, I'm not sure if they where resolved yet, or are going to be resolved at all as it was important for security reasons. I suspect most of the performance impact came from no longer using the cached hash code in the String class. And since String.hashCode() isn't allowed to change, that isn't going to change any time soon. On 22 April 2013 12:19, Dan Berindei <dan.berindei(a)gmail.com&gt; wrote: > Right. If we have anywhere a map that's initialized from a single thread and > then accessed only for reading from many threads, it probably makes sense to > use a HashMap and wrap it in an UnmodifiableMap. But if it can be written > from multiple threads as well, I think we should use a CHMV8. > > BTW, the HashMap implementation in OpenJDK 1.7 seems to have some > anti-collision features (a VM-dependent hash code generator for Strings), > but our version of CHMV8 doesn't. Perhaps we need to upgrade to the latest > CHMV8 version? > > > > On Fri, Apr 19, 2013 at 4:32 PM, David M. Lloyd <david.lloyd(a)redhat.com&gt; > wrote: >> >> On 04/19/2013 08:22 AM, Sanne Grinovero wrote: >> > On 19 April 2013 13:52, David M. Lloyd <david.lloyd(a)redhat.com&gt; wrote: >> >> On 04/19/2013 05:17 AM, Sanne Grinovero wrote: >> >>> On 19 April 2013 11:10, Dan Berindei <dan.berindei(a)gmail.com&gt; wrote: >> >>>> >> >>>> >> >>>> >> >>>> On Fri, Apr 19, 2013 at 12:58 PM, Sanne Grinovero >> >>>> <sanne(a)infinispan.org&gt; >> >>>> wrote: >> >>>>> >> >>>>> On 19 April 2013 10:37, Dan Berindei <dan.berindei(a)gmail.com&gt; wrote: >> >>>>>> Testing mixed read/write performance with capacity 100000, keys >> >>>>>> 300000, >> >>>>>> concurrency level 32, threads 12, read:write ratio 99:1 >> >>>>>> Container CHM Ops/s 5178894.77 Gets/s 5127105.82 Puts/s >> >>>>>> 51788.95 HitRatio 86.23 Size 177848 stdDev 60896.42 >> >>>>>> Container CHMV8 Ops/s 5768824.37 Gets/s 5711136.13 Puts/s >> >>>>>> 57688.24 HitRatio 84.72 Size 171964 stdDev 60249.99 >> >>>>> >> >>>>> Nice, thanks. >> >>>>>> >> >>>>>> The test is probably limited by the 1% writes, but I think it does >> >>>>>> show >> >>>>>> that >> >>>>>> reads in CHMV8 are not slower than reads in OpenJDK7's CHM. >> >>>>>> I haven't measured it, but the memory footprint should also be >> >>>>>> better, >> >>>>>> because it doesn't use segments any more. >> >>>>>> >> >>>>>> AFAIK the memoryCHMV8 also uses copy-on-write at the bucket level, >> >>>>>> but >> >>>>>> we >> >>>>>> could definitely do a pure read test with a HashMap to see how big >> >>>>>> the >> >>>>>> performance difference is. >> >>>>> >> >>>>> By copy-on-write I didn't mean on the single elements, but on the >> >>>>> whole map instance: >> >>>>> >> >>>>> private volatile HashMap configuration; >> >>>>> >> >>>>> synchronized addConfigurationProperty(String, String) { >> >>>>> HashMap newcopy = new HashMap( configuration ): >> >>>>> newcopy.put(..); >> >>>>> configuration = newcopy; >> >>>>> } >> >>>>> >> >>>>> Of course that is never going to scale for writes, but if writes >> >>>>> stop >> >>>>> at runtime after all services are started I would expect that the >> >>>>> simplicity of the non-threadsafe HashMap should have some benefit >> >>>>> over >> >>>>> CHM{whatever}, or it would have been removed already? >> >>>>> >> >>>> >> >>>> Right, we should be able to tell whether that's worth doing with a >> >>>> pure read >> >>>> test with a CHMV8 and a HashMap :) >> >>> >> >>> IFF you find out CHMV8 is as good as HashMap for read only, you have >> >>> two options: >> >>> - ask the JDK team to drop the HashMap code as it's no longer >> >>> needed >> >>> - fix your benchmark :-P >> >>> >> >>> In other words, I'd consider it highly surprising and suspicious >> >>> (still interesting though!) >> >> >> >> It's not as surprising as you think. On x86, volatile reads are the >> >> same as regular reads (not counting some possible reordering magic). >> >> So >> >> if a CHM read is a hash, an array access, and a list traversal, and so >> >> is HM (and I believe this is true though I'd have to review the code >> >> again to be sure), I'd expect very similar execution performance on >> >> read. I think some of the anti-collision features in V8 might come >> >> into >> >> play under some circumstances though which might affect performance in >> >> a >> >> negative way (wrt the constant big-O component) but overall in a >> >> positive way (by turning the linear big-O component into a logarithmic >> >> one). >> > >> > Thanks David. I know about the cost of a volatile read, what I'm >> > referring to >> > is that I would expect the non-concurrent Maps to generally contain some >> > simpler code than a conccurrent one. If this was not the case, >> > why would any JDK team maintain two different implementations? >> > That's why I would consider it surprising if it turned out that the >> > CHMV8 was >> > superior over a regular one on all fronts: there certainly is some >> > scenario in which the regular one would be a more appropriate choice, >> > which directly proofs that blindly replacing all usages in a large >> > project >> > is not optimal. Of course, it might be close to optimal.. >> >> You are right, it is not superior on all fronts. It is definitely >> similar in terms of read, but writes will have a substantially higher >> cost, involving (at the very least) multiple volatile writes which are >> orders of magnitude more expensive than normal writes (on Intel they >> have the costly impact of memory fence instructions). So I don't think >> anyone will want to drop HashMap any time soon. :-) >> >> -- >> - DML >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 25 April 2013 11:51, Galder Zamarreño <galder(a)redhat.com&gt; wrote: > > On Apr 22, 2013, at 3:09 PM, Dan Berindei <dan.berindei(a)gmail.com&gt; wrote: > >> >> >> >> On Mon, Apr 22, 2013 at 2:37 PM, Sanne Grinovero <sanne(a)infinispan.org&gt; wrote: >> We also have been toying with the idea to hash each key only once, >> instead of both with the consistent hash (to assign the node owner) >> and once in the CHM backing the datacontainer. >> I doubt we need the datacontainer to implement Map at all, but at >> least if we go this way we don't want the hash to be affected by the >> VM instance or different nodes won't agree on the expected owner ;-) >> >> >> For consistent hashing it would probably be better to cache the hash after applying MurmurHash to it anyway. So we could in theory hack our CHMV8 to use a cached hash code computed with MurmurHash and a cluster-specific salt. > > ^ Rather than hacking CHMv8, better to provide an Equivalence function (which CHMv8 will have an instance variable of) for the keys which keeps the cache of hashes or something… once that work is committed, we can discuss further :) +1 to use your cool implementation. Don't like too much the sound of "cache the hashes or something", I didn't actually look at the code, but from gut feeling I would hope the Equivalence function to be stateless? We might be able to pass the to-be-reused hash as a parameter in primitive form. Sanne _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

We also have been toying with the idea to hash each key only once, instead of both with the consistent hash (to assign the node owner) and once in the CHM backing the datacontainer.

I doubt we need the datacontainer to implement Map at all, but at least if we go this way we don't want the hash to be affected by the VM instance or different nodes won't agree on the expected owner ;-) Also there where reports of it having a very bad impact on performance, I'm not sure if they where resolved yet, or are going to be resolved at all as it was important for security reasons. On 22 April 2013 12:19, Dan Berindei <dan.berindei(a)gmail.com&gt; wrote: > Right. If we have anywhere a map that's initialized from a single thread and > then accessed only for reading from many threads, it probably makes sense to > use a HashMap and wrap it in an UnmodifiableMap. But if it can be written > from multiple threads as well, I think we should use a CHMV8. > > BTW, the HashMap implementation in OpenJDK 1.7 seems to have some > anti-collision features (a VM-dependent hash code generator for Strings), > but our version of CHMV8 doesn't. Perhaps we need to upgrade to the latest > CHMV8 version? > > > > On Fri, Apr 19, 2013 at 4:32 PM, David M. Lloyd <david.lloyd(a)redhat.com&gt; > wrote: >> >> On 04/19/2013 08:22 AM, Sanne Grinovero wrote: >>> On 19 April 2013 13:52, David M. Lloyd <david.lloyd(a)redhat.com&gt; wrote: >>>> On 04/19/2013 05:17 AM, Sanne Grinovero wrote: >>>>> On 19 April 2013 11:10, Dan Berindei <dan.berindei(a)gmail.com&gt; wrote: >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Apr 19, 2013 at 12:58 PM, Sanne Grinovero >>>>>> <sanne(a)infinispan.org&gt; >>>>>> wrote: >>>>>>> >>>>>>> On 19 April 2013 10:37, Dan Berindei <dan.berindei(a)gmail.com&gt; wrote: >>>>>>>> Testing mixed read/write performance with capacity 100000, keys >>>>>>>> 300000, >>>>>>>> concurrency level 32, threads 12, read:write ratio 99:1 >>>>>>>> Container CHM Ops/s 5178894.77 Gets/s 5127105.82 Puts/s >>>>>>>> 51788.95 HitRatio 86.23 Size 177848 stdDev 60896.42 >>>>>>>> Container CHMV8 Ops/s 5768824.37 Gets/s 5711136.13 Puts/s >>>>>>>> 57688.24 HitRatio 84.72 Size 171964 stdDev 60249.99 >>>>>>> >>>>>>> Nice, thanks. >>>>>>>> >>>>>>>> The test is probably limited by the 1% writes, but I think it does >>>>>>>> show >>>>>>>> that >>>>>>>> reads in CHMV8 are not slower than reads in OpenJDK7's CHM. >>>>>>>> I haven't measured it, but the memory footprint should also be >>>>>>>> better, >>>>>>>> because it doesn't use segments any more. >>>>>>>> >>>>>>>> AFAIK the memoryCHMV8 also uses copy-on-write at the bucket level, >>>>>>>> but >>>>>>>> we >>>>>>>> could definitely do a pure read test with a HashMap to see how big >>>>>>>> the >>>>>>>> performance difference is. >>>>>>> >>>>>>> By copy-on-write I didn't mean on the single elements, but on the >>>>>>> whole map instance: >>>>>>> >>>>>>> private volatile HashMap configuration; >>>>>>> >>>>>>> synchronized addConfigurationProperty(String, String) { >>>>>>> HashMap newcopy = new HashMap( configuration ): >>>>>>> newcopy.put(..); >>>>>>> configuration = newcopy; >>>>>>> } >>>>>>> >>>>>>> Of course that is never going to scale for writes, but if writes >>>>>>> stop >>>>>>> at runtime after all services are started I would expect that the >>>>>>> simplicity of the non-threadsafe HashMap should have some benefit >>>>>>> over >>>>>>> CHM{whatever}, or it would have been removed already? >>>>>>> >>>>>> >>>>>> Right, we should be able to tell whether that's worth doing with a >>>>>> pure read >>>>>> test with a CHMV8 and a HashMap :) >>>>> >>>>> IFF you find out CHMV8 is as good as HashMap for read only, you have >>>>> two options: >>>>> - ask the JDK team to drop the HashMap code as it's no longer >>>>> needed >>>>> - fix your benchmark :-P >>>>> >>>>> In other words, I'd consider it highly surprising and suspicious >>>>> (still interesting though!) >>>> >>>> It's not as surprising as you think. On x86, volatile reads are the >>>> same as regular reads (not counting some possible reordering magic). >>>> So >>>> if a CHM read is a hash, an array access, and a list traversal, and so >>>> is HM (and I believe this is true though I'd have to review the code >>>> again to be sure), I'd expect very similar execution performance on >>>> read. I think some of the anti-collision features in V8 might come >>>> into >>>> play under some circumstances though which might affect performance in >>>> a >>>> negative way (wrt the constant big-O component) but overall in a >>>> positive way (by turning the linear big-O component into a logarithmic >>>> one). >>> >>> Thanks David. I know about the cost of a volatile read, what I'm >>> referring to >>> is that I would expect the non-concurrent Maps to generally contain some >>> simpler code than a conccurrent one. If this was not the case, >>> why would any JDK team maintain two different implementations? >>> That's why I would consider it surprising if it turned out that the >>> CHMV8 was >>> superior over a regular one on all fronts: there certainly is some >>> scenario in which the regular one would be a more appropriate choice, >>> which directly proofs that blindly replacing all usages in a large >>> project >>> is not optimal. Of course, it might be close to optimal.. >> >> You are right, it is not superior on all fronts. It is definitely >> similar in terms of read, but writes will have a substantially higher >> cost, involving (at the very least) multiple volatile writes which are >> orders of magnitude more expensive than normal writes (on Intel they >> have the costly impact of memory fence instructions). So I don't think >> anyone will want to drop HashMap any time soon. :-) >> >> -- >> - DML >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Right. If we have anywhere a map that's initialized from a single thread and then accessed only for reading from many threads, it probably makes sense to use a HashMap and wrap it in an UnmodifiableMap. But if it can be written from multiple threads as well, I think we should use a CHMV8. BTW, the HashMap implementation in OpenJDK 1.7 seems to have some anti-collision features (a VM-dependent hash code generator for Strings), but our version of CHMV8 doesn't. Perhaps we need to upgrade to the latest CHMV8 version? On Fri, Apr 19, 2013 at 4:32 PM, David M. Lloyd <david.lloyd(a)redhat.com <mailto:david.lloyd@redhat.com>> wrote: On 04/19/2013 08:22 AM, Sanne Grinovero wrote: > On 19 April 2013 13:52, David M. Lloyd <david.lloyd(a)redhat.com <mailto:david.lloyd@redhat.com>> wrote: >> On 04/19/2013 05:17 AM, Sanne Grinovero wrote: >>> On 19 April 2013 11:10, Dan Berindei <dan.berindei(a)gmail.com <mailto:dan.berindei@gmail.com>> wrote: >>>> >>>> >>>> >>>> On Fri, Apr 19, 2013 at 12:58 PM, Sanne Grinovero <sanne(a)infinispan.org <mailto:sanne@infinispan.org>> >>>> wrote: >>>>> >>>>> On 19 April 2013 10:37, Dan Berindei <dan.berindei(a)gmail.com <mailto:dan.berindei@gmail.com>> wrote: >>>>>> Testing mixed read/write performance with capacity 100000, keys 300000, >>>>>> concurrency level 32, threads 12, read:write ratio 99:1 >>>>>> Container CHM Ops/s 5178894.77 Gets/s 5127105.82 Puts/s >>>>>> 51788.95 HitRatio 86.23 Size 177848 stdDev 60896.42 >>>>>> Container CHMV8 Ops/s 5768824.37 Gets/s 5711136.13 Puts/s >>>>>> 57688.24 HitRatio 84.72 Size 171964 stdDev 60249.99 >>>>> >>>>> Nice, thanks. >>>>>> >>>>>> The test is probably limited by the 1% writes, but I think it does show >>>>>> that >>>>>> reads in CHMV8 are not slower than reads in OpenJDK7's CHM. >>>>>> I haven't measured it, but the memory footprint should also be better, >>>>>> because it doesn't use segments any more. >>>>>> >>>>>> AFAIK the memoryCHMV8 also uses copy-on-write at the bucket level, but >>>>>> we >>>>>> could definitely do a pure read test with a HashMap to see how big the >>>>>> performance difference is. >>>>> >>>>> By copy-on-write I didn't mean on the single elements, but on the >>>>> whole map instance: >>>>> >>>>> private volatile HashMap configuration; >>>>> >>>>> synchronized addConfigurationProperty(String, String) { >>>>> HashMap newcopy = new HashMap( configuration ): >>>>> newcopy.put(..); >>>>> configuration = newcopy; >>>>> } >>>>> >>>>> Of course that is never going to scale for writes, but if writes stop >>>>> at runtime after all services are started I would expect that the >>>>> simplicity of the non-threadsafe HashMap should have some benefit over >>>>> CHM{whatever}, or it would have been removed already? >>>>> >>>> >>>> Right, we should be able to tell whether that's worth doing with a pure read >>>> test with a CHMV8 and a HashMap :) >>> >>> IFF you find out CHMV8 is as good as HashMap for read only, you have >>> two options: >>> - ask the JDK team to drop the HashMap code as it's no longer needed >>> - fix your benchmark :-P >>> >>> In other words, I'd consider it highly surprising and suspicious >>> (still interesting though!) >> >> It's not as surprising as you think. On x86, volatile reads are the >> same as regular reads (not counting some possible reordering magic). So >> if a CHM read is a hash, an array access, and a list traversal, and so >> is HM (and I believe this is true though I'd have to review the code >> again to be sure), I'd expect very similar execution performance on >> read. I think some of the anti-collision features in V8 might come into >> play under some circumstances though which might affect performance in a >> negative way (wrt the constant big-O component) but overall in a >> positive way (by turning the linear big-O component into a logarithmic one). > > Thanks David. I know about the cost of a volatile read, what I'm referring to > is that I would expect the non-concurrent Maps to generally contain some > simpler code than a conccurrent one. If this was not the case, > why would any JDK team maintain two different implementations? > That's why I would consider it surprising if it turned out that the CHMV8 was > superior over a regular one on all fronts: there certainly is some > scenario in which the regular one would be a more appropriate choice, > which directly proofs that blindly replacing all usages in a large project > is not optimal. Of course, it might be close to optimal.. You are right, it is not superior on all fronts. It is definitely similar in terms of read, but writes will have a substantially higher cost, involving (at the very least) multiple volatile writes which are orders of magnitude more expensive than normal writes (on Intel they have the costly impact of memory fence instructions). So I don't think anyone will want to drop HashMap any time soon. :-) -- - DML _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org <mailto:infinispan-dev@lists.jboss.org> https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Why not. Only doubt I'd have is that other usages of the CHM are - I guess - services registry and similar configuration tools, for which write performance is irrelevant: your test measured puts, are there drawbacks on gets or memory usage?

https://issues.jboss.org/browse/ISPN-3022 In fact, we should also re-implement the BoundedConcurrentHashMap - the basis for LIRS and LRU eviction algorithms - to use techniques similar to CHMv8 rather than the old segment-based approach. https://issues.jboss.org/browse/ISPN-3023 Anyone has a few spare cycles - and is really good with complex concurrency code - want to take this on? ;)

So, in case anyone is itching to start on 3022 and 3023, hold on for a few days while I finish the work to include EquivalentCHMv8.

+1 to make CHMv8 the default on JDK6 and JDK7

Do you have the numbers for JDK 7 as well? Radim ----- Original Message ----- | From: "Manik Surtani" <msurtani(a)redhat.com&gt; | To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; | Sent: Friday, April 19, 2013 4:35:53 AM | Subject: [infinispan-dev] CHM or CHMv8? | | Guys, | | Based on some recent micro benchmarks I've been doing, I've seen: | | MapStressTest configuration: capacity 100000, test running time 60 seconds | Testing mixed read/write performance with capacity 100,000, keys 300,000, | concurrency level 32, threads 12, read:write ratio 0:1 | Container CHM Ops/s 21,165,771.67 Gets/s 0.00 Puts/s | 21,165,771.67 HitRatio 100.00 Size 262,682 stdDev 77,540.73 | Container CHMV8 Ops/s 33,513,807.09 Gets/s 0.00 Puts/s | 33,513,807.09 HitRatio 100.00 Size 262,682 stdDev 77,540.73 | | So under high concurrency (12 threads, on my workstation with 12 hardware | threads - so all threads are always working), we see that Infinispan's CHMv8 | implementation is 50% faster than JDK6's CHM implementation when doing puts. | | We use a fair number of CHMs all over Infinispan's codebase. By default, | these are all JDK-provided CHMs. But we have the option to switch to our | CHMv8 implementation by passing in -Dinfinispan.unsafe.allow_jdk8_chm=true. | | The question is, should this be the default? Thoughts, opinions? | | - M | | -- | Manik Surtani | manik(a)jboss.org | twitter.com/maniksurtani | | Platform Architect, JBoss Data Grid | http://red.ht/data-grid | | | _______________________________________________ | infinispan-dev mailing list | infinispan-dev(a)lists.jboss.org | https://lists.jboss.org/mailman/listinfo/infinispan-dev | _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev