[infinispan-dev] MurmurHash fixes

[Sanne Grinovero]

Very nice.
wouldn't it make sense to be able to run Infinispan in a "migration
mode", to cover all possible future cases in which a hash function is
upgraded, or for those cases people want to replace their function?
We could provide a feature to use a secondary function to try fetching
what was not found using the primary function, and when this triggers
it should also take care of moving data around the stores.
You could see this operation as an extension to the rehashing process,
so even if it has a performance hit it could reuse some code and
benefit of all optimizations you do during rehashing.

Thinking it as a rehashing operation, means we could potentially offer
means to change the hash algorithm on the fly; quite nice to
experiment on big datasets with different implementations.

Sanne

2011/1/14 Manik Surtani <manik at jboss.org>:
> We do support rolling upgrade.  And further, we also support the following scenario:
>
> 1) Run a cluster, using a cache store that is *not* shared.  Entries get hashed to specific nodes.
> 2) Shut down the cluster, upgrade Infinispan.
> 3) Should still be able to locate your entries, as they would be loaded from the cache stores.
>
> If the hash function changes between upgrades, then requests for keys would now be mapped to different nodes, and the keys won't be found in those nodes' cache stores.
>
>
> On 14 Jan 2011, at 09:45, Sanne Grinovero wrote:
>
>> I'm wondering why you're considering backwards compatibility of the
>> hashing distribution. Is Infinispan supposed to support multiple
>> different versions to be connected in a cluster, providing some form
>> of rolling upgrade?
>>
>> Cheers,
>> Sanne
>>
>> 2011/1/14 Emmanuel Bernard <emmanuel at hibernate.org>:
>>> Or release a 4.3.
>>>
>>> On 14 janv. 2011, at 09:32, Bela Ban wrote:
>>>
>>>> I would fix the existing MurmurHash; IMO backward compat doesn't extend
>>>> to consistent hashing...
>>>>
>>>> Does leaving the 1 bit out create extreme anomalies wrt hashing
>>>> distribution ? I wouldn't think so...
>>>>
>>>> I'm fine with the other solution, too.
>>>>
>>>>
>>>>
>>>> On 1/13/11 8:35 PM, Manik Surtani wrote:
>>>>> Guys,
>>>>>
>>>>> The MurmurHash2 impl we have in 4.2.0 is buggy in that my translation from the original C source was faulty and it effectively hashes over just 31 bits instead of 32.  It means the distribution result isn't as good as it could be.
>>>>>
>>>>> Now it isn't that easy for me to just *fix* this in 4.2.1, since it means keys mapped to nodes using 4.2.0 may not map to the same node in 4.2.1.
>>>>>
>>>>> So here is what I propose:
>>>>>
>>>>> 1) Fix it in 4.2.x as MurmurHash2A
>>>>> 2) Use MurmurHash2A by default, *unless* a config flag is provided that forces the use of MurmurHash2.  (e.g.,<hash function="MurmurHash2">)
>>>>>
>>>>> This will even give us the ability to use MurmurHash3 in 5.0 when we have it.
>>>>>
>>>>> WDYT?
>>>>
>>>>
>>>> --
>>>> Bela Ban
>>>> Lead JGroups / Clustering Team
>>>> JBoss
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> --
> Manik Surtani
> manik at jboss.org
> twitter.com/maniksurtani
>
> Lead, Infinispan
> http://www.infinispan.org
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>