[infinispan-dev] Recommended Cluster Size for Replication Mode

Radim Vansa rvansa at redhat.com
Fri Aug 16 06:56:56 EDT 2013 

On 08/16/2013 12:27 PM, Faseela K wrote:
> Hi Radim,
>
>     I tried my performance test again with Infinispan - 5.3.0.
>     Now, "replication writes" are taking almost double the time as in 5.2.3.
You may try transactional one-phase commits for the 5.2.x behaviour 
(what accounts to messages), although there may be other overhead 
slowing that down.
>     Could you please clarify the issues with replication mode in 5.2.3?
Let's consider two concurrent writes for key K, values are V1 and V2. 
There are 3 nodes A, B, C, the write is executed from A, B is the 
primary owner of this key.
5.2.x:
A locally writes V1 and sends broadcast message to B and C to update K 
to V1, waiting for acks
A locally writes V2 and sends broadcast message to B and C to update K 
to V2, waiting for acks
B receives and applies write V1, sends ack to A
B receives and applies write V2, sends ack to A
C receives and applies write V2, sends ack to A
C receives and applies write V1, sends ack to A
A receives acks for V1, returns
A receives acks for V2, returns
result is A[K] = V2, B[K] = V2, C[K] = V1 => inconsistency
5.3.x:
A sends to B as the primary owner with write V1
A sends to B as the primary owner with write V2
B locks key K, locally writes V1 and sends message to A and C, waiting 
for acks
B tries to lock key K in order to write V2, but waits for lock
A and C apply write V1 and send acks
B unlocks K and sends ack to A
A receives acks for V1, returns
B locks key K, locally writes V2 and sends message to A and C, waiting 
for acks
A and C apply write V2 and send acks
B unlocks K and sends ack to A
A receives acks for V2, returns
result is A[K] = V2, B[K] = V2, C[K] = V2 => cluster is consistent


>     I guess, to overcome those issues, some more messages were added to replication mode in 5.3.0, because of which writes are
>     taking more time in 5.3.0.
>     Please note that, with a 3 node cluster, now distribution and replication are taking almost the same time for "writes".
That's because these modes do not differ anymore in implementation.

Radim

> -----Original Message-----
> From: infinispan-dev-bounces at lists.jboss.org [mailto:infinispan-dev-bounces at lists.jboss.org] On Behalf Of Radim Vansa
> Sent: Tuesday, August 13, 2013 5:34 PM
> To: infinispan-dev at lists.jboss.org
> Subject: Re: [infinispan-dev] Recommended Cluster Size for Replication Mode
>
> On 08/13/2013 11:58 AM, Faseela K wrote:
>> Hi Radim,
>>
>>     Thanks for the information.
>>     A few more doubts :
>>
>>            -  Is it possible to have different named caches with different clustering configuration?
>>               Eg : I need a cache's data to be replicated asynchronously, and one other cache's data to be distributed synchronously.
> Yes, this is expected configuration.
>
>>            -  What will be the impact of putAsync() call on clustered performance?
> The putAsync differs from regular put call just by executing the command in different thread (from threadpool) and returning future instead of the previous value. Besides this the processing of the command inside Infinispan is identical.
> Btw., if you drive for highest performance, you could consider using the IGNORE_RETURN_VALUE flag.
>>            -  Is there any performace numbers already avaialble for various clustering modes?
>>               Are there any benchmarks already documented?
> Such results are not disclosed. For benchmarking, you may consider using the RadarGun tool [1] (use snapshot version, 1.0 is prehistorical release).
>
> Radim
>
> [1] https://github.com/radargun/radargun
>
>>     I will try my performance test once again with 5.3.0 version.
>>     I will also analyse impacts of moving my clustering mode to asynchronous replication/distribution.
>>
>> Thanks,
>> Faseela
>>
>> -----Original Message-----
>> From: infinispan-dev-bounces at lists.jboss.org
>> [mailto:infinispan-dev-bounces at lists.jboss.org] On Behalf Of Radim
>> Vansa
>> Sent: Tuesday, August 13, 2013 2:08 PM
>> To: infinispan-dev at lists.jboss.org
>> Subject: Re: [infinispan-dev] Recommended Cluster Size for Replication
>> Mode
>>
>> Yes, with 5.2.3 it makes sense - in replication mode there were less messages used for the write but key may have different value on different nodes after concurrent modification. In order to evade this, you can setup 4-node distributed cluster with numOwners=4.
>>
>> All the performance optimizations depend on how consistent the storage has to be. Do you write to the same keys in one moment from multiple locations? Would you be OK with a bit out-of-date reads? Is it a problem if you read different values for the key on different nodes? Is it a problem to a write getting lost once a while?
>> With strong consistency requirements, you can't get top performance, but maybe you don't really need it - or you need it just for some data in your app which can be positioned into cache with different configuration.
>> There is a nice table [1] for asynchronous options guarantees.
>>
>> And by the way - "my application involves both reads and writes heavily"
>> does not say anything - the factor is the ratio (percentage) of reads to writes.
>>
>> Radim
>>
>> [1]
>> https://docs.jboss.org/author/pages/viewpage.action?pageId=63636092
>>
>> On 08/13/2013 06:29 AM, Faseela K wrote:
>>> Hi,
>>>
>>>       I am using infinispan 5.2.3.
>>>       My configuration is non-transactional, synchronous.
>>>       With this configuration, is my replication supposed to perform better than distribution, for both reads and writes?
>>>       My Cluster Size requirement is 4 nodes.
>>>       And my application involves both reads and writes heavily.
>>>       For better performance, are there any suggestions on the clustering modes/configurations?
>>>       All my tests show, replication having better performance than distribution for reads as well as writes, with 4 nodes.
>>>
>>> Thanks,
>>> Faseela
>>>
>>> -----Original Message-----
>>> From: infinispan-dev-bounces at lists.jboss.org
>>> [mailto:infinispan-dev-bounces at lists.jboss.org] On Behalf Of Radim
>>> Vansa
>>> Sent: Monday, August 12, 2013 6:13 PM
>>> To: infinispan-dev at lists.jboss.org
>>> Subject: Re: [infinispan-dev] Recommended Cluster Size for
>>> Replication Mode
>>>
>>> Hi,
>>>
>>> which version exactly do you use, 5.2.x, 5.3.x or 6.0.x? In 5.2 the replication mode was implemented separately from distribution mode and depending on your configuration (is it non-transactional synchronous?) the message control flow could differ. Since 5.3 replication mode is implemented in the same manner and the results should be more comparable.
>>> I may be wrong here, but in 5.2.x concurrent writes to single key in non-transactional mode could result in entries being out of sync on some nodes (the writes could arrive at two nodes in different order). I think this cannot happen in >= 5.3 anymore.
>>>
>>> Radim
>>>
>>> On 08/12/2013 09:04 AM, Faseela K wrote:
>>>> Hi,
>>>>
>>>>       With a 3 node cluster, even for "WRITES" my replication performance is better than distribution.
>>>>       That's why I came across this doubt.
>>>>       Could some body please clarify, why the behaviour is like this?
>>>>
>>>> Thanks,
>>>> Faseela
>>>>
>>>> -----Original Message-----
>>>> From: infinispan-dev-bounces at lists.jboss.org
>>>> [mailto:infinispan-dev-bounces at lists.jboss.org] On Behalf Of Mircea
>>>> Markus
>>>> Sent: Friday, August 09, 2013 7:14 PM
>>>> To: infinispan -Dev List
>>>> Subject: Re: [infinispan-dev] Recommended Cluster Size for
>>>> Replication Mode
>>>>
>>>> On 6 Aug 2013, at 15:19, Faseela K <faseela.k at ericsson.com> wrote:
>>>>
>>>>>        What is the recommended cluster size for Replication Mode?
>>>>>        Given 3 nodes, My replication configuration performs better than my distributed configuration.
>>>>>        Just wanted to know, at what cluster size, distribution will perform better than replication.
>>>> There's no straight answer, it depends on the read/write ratio and the amount of data you store.
>>>> Replication will always perform better for reads as it won't involve a remote call to get the data.
>>>> If you're mostly doing reads and your memory allows (replication is more memory consuming) then you should use replication.
>>>> If the amount of data increases or you're doing more writes, distribution is the way to go.
>>>>
>>>> Cheers,
>>>> --
>>>> Mircea Markus
>>>> Infinispan lead (www.infinispan.org)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

More information about the infinispan-dev mailing list