This may have been stated somewhere but will the Hash function be configurable? At least at the deployment level. Not sure about at the cache level. I could see where different deployments might want different hash attributes. E.g. spread vs. speed or CPU neutral behavior vs. spread, etc. Bryan -----Original Message----- From: infinispan-dev-bounces(a)lists.jboss.org [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Manik Surtani Sent: Tuesday, March 23, 2010 10:44 AM To: infinispan -Dev List Subject: [infinispan-dev] Hash functions Some very early results, http://pastie.org/883111 Only testing MurmurHash2 (endian-neutral variant which is 2x as slow as the original algo) and SuperFastHash. I haven't implemented FNV-1 as yet. Some notes: * Test was run over 100k random keys * Max size for a String and byte[] key set to 16. Actual size is a random number between 1 and MaxSize. * Functions mainly implemented to handle byte[]'s. * Functions handle Strings by calling String.getBytes(). The bulk of the time spent in String keys is therefore attributed to String.getBytes(). * Functions handle Object hashcodes by taking the int hashcode and creating a 4-element byte[] out of it. Again, the bulk of the time spent here is in this conversion. * Keys generated before any measurements taken, a full cycle run to warm up the hotspot compiler as well. Looks like MurmurHash2, despite using the slower version to accommodate CPU endian neutral behaviour, is winning in terms of distribution. And by a fair way too. For those interested in the test and the hash impls, have a look at this (dependency on Apache commons-math): http://pastie.org/883135 Cheers -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev --------------------------------------------------------------------- This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful. _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Some very early results, http://pastie.org/883111 Only testing MurmurHash2 (endian-neutral variant which is 2x as slow as the original algo) and SuperFastHash. I haven't implemented FNV-1 as yet. Some notes: * Test was run over 100k random keys * Max size for a String and byte[] key set to 16. Actual size is a random number between 1 and MaxSize. * Functions mainly implemented to handle byte[]'s. * Functions handle Strings by calling String.getBytes(). The bulk of the time spent in String keys is therefore attributed to String.getBytes(). * Functions handle Object hashcodes by taking the int hashcode and creating a 4-element byte[] out of it. Again, the bulk of the time spent here is in this conversion. * Keys generated before any measurements taken, a full cycle run to warm up the hotspot compiler as well. Looks like MurmurHash2, despite using the slower version to accommodate CPU endian neutral behaviour, is winning in terms of distribution. And by a fair way too. For those interested in the test and the hash impls, have a look at this (dependency on Apache commons-math): http://pastie.org/883135 Cheers -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 29 Mar 2010, at 10:30, Galder Zamarreno wrote: > Manik, > > If String's are dealt with by calling String.getBytes() which in turns > gets processed as a byte[], why is there such a big difference > SuperFastHash String speed and MurmurHash2 String speed (162 ms vs 56 ms) > but there's hardly any difference between the byte[] tests (29ms vs 40ms) ? No idea. Microbenchmarks are notoriously fickle though, I will try and run this again over a larger sample size to smooth things out a bit.

Jason T. Greene wrote: > Manik Surtani wrote: >> On 29 Mar 2010, at 10:30, Galder Zamarreno wrote: >> >>> Manik, >>> >>> If String's are dealt with by calling String.getBytes() which in >>> turns gets processed as a byte[], why is there such a big >>> difference SuperFastHash String speed and MurmurHash2 String speed >>> (162 ms vs 56 ms) but there's hardly any difference between the >>> byte[] tests (29ms vs 40ms) ? >> >> No idea. Microbenchmarks are notoriously fickle though, I will try >> and run this again over a larger sample size to smooth things out a bit. >> > > UTF-8 encoding. It has a cost. > ^ measurable