<div dir="ltr"><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jun 3, 2013 at 4:23 PM, <span dir="ltr"><<a href="mailto:vitalii.tymchyshyn@ubs.com" target="_blank">vitalii.tymchyshyn@ubs.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello.<br>
<br>
Thanks for your information. I will subscribe and vote for the issues noted.<br>
In the meantime I've implemented hacky JgroupsTransport that "downgrades" all (but CacheViewControlCommand and StateTransferControlCommand) SYNCHRONOUS invokeRemotely calls to SYNCHRONOUS_IGNORE_LEAVERS and checks if required number of answers was received with a filter (I've tried to use original invokeRemotely return value but it often returns some strange value, like empty map). It seems to do the trick for me. But I am still not sure if this has any side effects.<br>
<br></blockquote><div><br></div><div>Indeed, I started working on a solution, but I over-engineered it and then I got side-tracked with other stuff. Sorry about that.<br></div><div><br>The problem with using SYNCHRONOUS_IGNORE_LEAVERS everywhere, as I found
out, is that you don't want to ignore the primary owner of a key
leaving during a prepare/lock command (or the coordinator, in REPL mode prior to 5.3.0.CR1/ISPN-2772).
If that happens, you have to retry on the new primary owner, otherwise you can't
know if the prepare command has locked the key or not.<br><br>A similar problem appears in non-transactional caches with supportsConcurrentUpdates=true: there the primary owner can ignore any of the backup owners leaving, but the originator can't ignore the primary owner leaving.<br>
<br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
For now I can see merge problem in my test: different values are picked during merge. I am going to dig a little deeper and follow up. But it's already a little strange for me, since the test algorithm is:<br>
1)Assign "old" value to full cluster (it's REPL_SYNC mode)<br>
2)Block coordinator<br>
3)Writer "new" value to one of two remaining nodes. It's syncrhonized to second remaining node<br>
4)Unblock coordinator<br>
5)Wait (I could not find a good way to wait for state transfer but wait in this case).<br>
6)Check the value on coordinator<br>
<br>
And in my test I am randomly getting "old" or "new" in assert. I am now going to check why. May be I will need to "reinitialize" smaller cluster part to ensure data is taken from the quorum part of the cluster.<br>
<br></blockquote><div><br></div><div>We don't handle merges properly. See <a href="https://issues.jboss.org/browse/ISPN-263" target="_blank">https://issues.jboss.org/browse/ISPN-263</a> and the discussion at <a href="http://markmail.org/message/meyczotzobuva7js">http://markmail.org/message/meyczotzobuva7js</a></div>
<div><br></div><div>What happens right now is that after a merge, all the caches are assumed to have up-to-date data, so there is no state transfer. We had several ideas floating around on how we could force the smaller partition to receive data from the quorum partition, but I think with the public API your best option is to stop all the caches in the smaller partition after the split and start them back up after the merge.<br>
<br></div><div>Cheers<br></div><div>Dan<br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Best regards, Vitalii Tymchyshyn<br>
<div><div><br>
-----Original Message-----<br>
From: <a href="mailto:infinispan-dev-bounces@lists.jboss.org" target="_blank">infinispan-dev-bounces@lists.jboss.org</a> [mailto:<a href="mailto:infinispan-dev-bounces@lists.jboss.org" target="_blank">infinispan-dev-bounces@lists.jboss.org</a>] On Behalf Of Galder Zamarreno<br>
Sent: Monday, June 03, 2013 9:04 AM<br>
To: infinispan -Dev List<br>
Subject: Re: [infinispan-dev] Using infinispan as quorum-based nosql<br>
<br>
<br>
On May 30, 2013, at 5:10 PM, <a href="mailto:vitalii.tymchyshyn@ubs.com" target="_blank">vitalii.tymchyshyn@ubs.com</a> wrote:<br>
<br>
> Hello.<br>
><br>
> We are going to use Infinispan in our project as NoSQL solution. It<br>
> performs quite well for us, but currently we've faced next problem.<br>
> Note: We are using Infinispan 5.1.6 in SYNC_REPL mode in small cluster.<br>
> The problem is that when any node fails, any running transactions wait<br>
> for Jgroups to decide if it've really failed or not and rollback<br>
> because of SuspectException after that. While we can live with a<br>
> delay, we'd really like to skip rolling back. As for me, I actually<br>
> don't see a reason for rollback because transactions started after<br>
> leave will succeed. So, as for me, previously running transactions<br>
> could do the same.<br>
<br>
We're aware of the problem (<a href="https://issues.jboss.org/browse/ISPN-2402" target="_blank">https://issues.jboss.org/browse/ISPN-2402</a>).<br>
<br>
@Dan, has there been any updates on this?<br>
<br>
> The question for is if node that left will synchronize it's state<br>
> after merge (even if merge was done without infinispan restart). As<br>
> for me, it should or it won't work correctly at all.<br>
<br>
This is not in yet: <a href="https://issues.jboss.org/browse/ISPN-263" target="_blank">https://issues.jboss.org/browse/ISPN-263</a><br>
<br>
> So, I've found RpcManager's ResponseMode.SYNCHRONOUS_IGNORE_LEAVERS<br>
> and think on switching to it for RpcManager calls that don't specify<br>
> ResponseMode explicitly. As for me, it should do the trick. Also, I am<br>
> going to enforce Quorum number of reponses, but that's another story.<br>
> So, how do you think, would it work?<br>
<br>
^ Not sure if that'll work. @Dan?<br>
<br>
> P.S. Another Q for me, how does it work now, when SuspectException is<br>
> thrown from CommitCommand broadcasting. Af far as I can see, commit is<br>
> still done on some remote nodes (that are still in the cluster), but<br>
> rolled back on local node because of this exception. Am I correct?<br>
<br>
^ How Infinispan reacts in these situations depends a lot on the type of communications (synchronous or asynchronous) and the transaction configuration. Mircea can provide more details on this.<br>
<br>
Cheers,<br>
<br>
> This<br>
> can cause inconsistencies, but we must leave with something in<br>
> peer-to-peer world :) The only other option is to switch from<br>
> write-all, read-local to write-quorum, read-quorum scenario that is<br>
> too complex move for Infinispan as for me.<br>
><br>
> Best regards, Vitalii Tymchyshyn<br>
><br>
> Please visit our website at<br>
> <a href="http://financialservicesinc.ubs.com/wealth/E-maildisclaimer.html" target="_blank">http://financialservicesinc.ubs.com/wealth/E-maildisclaimer.html</a><br>
> for important disclosures and information about our e-mail policies.<br>
> For your protection, please do not transmit orders or instructions by<br>
> e-mail or include account numbers, Social Security numbers, credit<br>
> card numbers, passwords, or other personal information.<br>
><br>
> _______________________________________________<br>
> infinispan-dev mailing list<br>
> <a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
<br>
<br>
--<br>
Galder Zamarreņo<br>
<a href="mailto:galder@redhat.com" target="_blank">galder@redhat.com</a><br>
<a href="http://twitter.com/galderz" target="_blank">twitter.com/galderz</a><br>
<br>
Project Lead, Escalante<br>
<a href="http://escalante.io" target="_blank">http://escalante.io</a><br>
<br>
Engineer, Infinispan<br>
<a href="http://infinispan.org" target="_blank">http://infinispan.org</a><br>
<br>
<br>
_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
Please visit our website at<br>
<a href="http://financialservicesinc.ubs.com/wealth/E-maildisclaimer.html" target="_blank">http://financialservicesinc.ubs.com/wealth/E-maildisclaimer.html</a><br>
for important disclosures and information about our e-mail<br>
policies. For your protection, please do not transmit orders<br>
or instructions by e-mail or include account numbers, Social<br>
Security numbers, credit card numbers, passwords, or other<br>
personal information.<br>
<br>
_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
</div></div></blockquote></div><br></div></div>