[infinispan-dev] Using infinispan as quorum-based nosql

[vitalii.tymchyshyn at ubs.com]

Hello.

Thanks for your information. I will subscribe and vote for the issues noted.
In the meantime I've implemented hacky JgroupsTransport that "downgrades" all (but CacheViewControlCommand and StateTransferControlCommand) SYNCHRONOUS invokeRemotely calls to SYNCHRONOUS_IGNORE_LEAVERS and checks if required number of answers was received with a filter (I've tried to use original invokeRemotely return value but it often returns some strange value, like empty map). It seems to do the trick for me. But I am still not sure if this has any side effects.

For now I can see merge problem in my test: different values are picked during merge. I am going to dig a little deeper and follow up. But it's already a little strange for me, since the test algorithm is:
1)Assign "old" value to full cluster (it's REPL_SYNC mode)
2)Block coordinator
3)Writer "new" value to one of two remaining nodes. It's syncrhonized to second remaining node
4)Unblock coordinator
5)Wait (I could not find a good way to wait for state transfer but wait in this case).
6)Check the value on coordinator

And in my test I am randomly getting "old" or "new" in assert. I am now going to check why. May be I will need to "reinitialize" smaller cluster part to ensure data is taken from the quorum part of the cluster.

Best regards, Vitalii Tymchyshyn

-----Original Message-----
From: infinispan-dev-bounces at lists.jboss.org [mailto:infinispan-dev-bounces at lists.jboss.org] On Behalf Of Galder Zamarreno
Sent: Monday, June 03, 2013 9:04 AM
To: infinispan -Dev List
Subject: Re: [infinispan-dev] Using infinispan as quorum-based nosql


On May 30, 2013, at 5:10 PM, vitalii.tymchyshyn at ubs.com wrote:

> Hello.
> 
> We are going to use Infinispan in our project as NoSQL solution. It 
> performs quite well for us, but currently we've faced next problem.
> Note: We are using Infinispan 5.1.6 in SYNC_REPL mode in small cluster.
> The problem is that when any node fails, any running transactions wait 
> for Jgroups to decide if it've really failed or not and rollback 
> because of SuspectException after that. While we can live with a 
> delay, we'd really like to skip rolling back. As for me, I actually 
> don't see a reason for rollback because transactions started after 
> leave will succeed. So, as for me, previously running transactions 
> could do the same.

We're aware of the problem (https://issues.jboss.org/browse/ISPN-2402).

@Dan, has there been any updates on this?

> The question for is if node that left will synchronize it's state 
> after merge (even if merge was done without infinispan restart). As 
> for me, it should or it won't work correctly at all.

This is not in yet: https://issues.jboss.org/browse/ISPN-263

> So, I've found RpcManager's ResponseMode.SYNCHRONOUS_IGNORE_LEAVERS 
> and think on switching to it for RpcManager calls that don't specify 
> ResponseMode explicitly. As for me, it should do the trick. Also, I am 
> going to enforce Quorum number of reponses, but that's another story.
> So, how do you think, would it work?

^ Not sure if that'll work. @Dan?

> P.S. Another Q for me, how does it work now, when SuspectException is 
> thrown from CommitCommand broadcasting. Af far as I can see, commit is 
> still done on some remote nodes (that are still in the cluster), but 
> rolled back on local node because of this exception. Am I correct?

^ How Infinispan reacts in these situations depends a lot on the type of communications (synchronous or asynchronous) and the transaction configuration. Mircea can provide more details on this.

Cheers,

> This
> can cause inconsistencies, but we must leave with something in 
> peer-to-peer world :) The only other option is to switch from 
> write-all, read-local to write-quorum, read-quorum scenario that is 
> too complex move for Infinispan as for me.
> 
> Best regards, Vitalii Tymchyshyn
> 
> Please visit our website at
> http://financialservicesinc.ubs.com/wealth/E-maildisclaimer.html
> for important disclosures and information about our e-mail policies. 
> For your protection, please do not transmit orders or instructions by 
> e-mail or include account numbers, Social Security numbers, credit 
> card numbers, passwords, or other personal information.
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


--
Galder Zamarreño
galder at redhat.com
twitter.com/galderz

Project Lead, Escalante
http://escalante.io

Engineer, Infinispan
http://infinispan.org


_______________________________________________
infinispan-dev mailing list
infinispan-dev at lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Please visit our website at 
http://financialservicesinc.ubs.com/wealth/E-maildisclaimer.html 
for important disclosures and information about our e-mail 
policies. For your protection, please do not transmit orders 
or instructions by e-mail or include account numbers, Social 
Security numbers, credit card numbers, passwords, or other 
personal information.