[ https://issues.jboss.org/browse/JGRP-1484?page=com.atlassian.jira.plugin.... ] Bela Ban updated JGRP-1484: --------------------------- Fix Version/s: 3.1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1484?page=com.atlassian.jira.plugin.... ] David Hotham commented on JGRP-1484: ------------------------------------ Hi, Do you know when you hope to be able to take a look at this one? It would be very helpful for me to know this, to help me decide what (if anything) I need to do about it in the meantime. It would be a great first step if we could just come up with a design for how this ought to work. If that design isn't too complicated, I might be able to relieve you of the task of doing the coding... Thanks David -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1484?page=com.atlassian.jira.plugin.... ] David Hotham commented on JGRP-1484: ------------------------------------ OK, thanks for letting me know. Do you have any thoughts on the right fix? Eg does my suggestion (last couple of sentences of original report) look plausible? I'd feel much happier embarking on an attempted fix if you could at least go so far as to say that it didn't sound completely idiotic ;-) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1484?page=com.atlassian.jira.plugin.... ] David Hotham commented on JGRP-1484: ------------------------------------ Hi, I hope you enjoyed your vacation. Are you able to say when you expect to take a look at this and my other open tickets? I believe that I currently have seven open issues, but the good news is: * there's a pull request associated with each of them * with these fixes, my cluster is currently withstanding all of the stress that I can throw at it. ** I think this is the first time since February that I don't know a way of breaking the cluster - which I'm very happy about! What I'd really like, if possible, is to get all of those issues resolved and then have a 3.1.1 delivery containing the fixes. Is there any chance we can make that happen? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1484?page=com.atlassian.jira.plugin.... ] David Hotham commented on JGRP-1484: ------------------------------------ No news? I understand that there's no special reason that my problems should be your priority - but for my own planning purposes, I'd really appreciate an insight into if/when you expect to look at them. Thanks! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1484?page=com.atlassian.jira.plugin.... ] Bela Ban commented on JGRP-1484: -------------------------------- I'm not in favor of changing GMS because of a bug in SEQUENCER. I'll have to look into this more closely, but haven't we addressed this problem before ? I thought the fix preventing A to forward the view broadcast to the old coordinator B was to handle event TMP_VIEW and change the coord (in this case to C). I looked at the unit test you added in pull request #54 (thanks !) and I'll think about how this can be fixed in SEQUENCER, without having to change code in GMS. Cheers, -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1484?page=com.atlassian.jira.plugin.... ] Bela Ban edited comment on JGRP-1484 at 9/3/12 12:10 PM: --------------------------------------------------------- Hmm. I assume the reason you have SEQUENCER *below* GMS is that the order of views and messages is the same everywhere. I've thought about how to keep this property in the case of a merge, but so far haven't come up with a good solution. In your suggested solution, when a merge participant gets a (unicast) INSTALL_MERGE_VIEW, it installs the MergeView directly. However, this destroys the property above, as the order of the view installation with respect to messages is not the same across all members. I experimented with marking the MergeView multicast as NO_TOTAL_ORDER, ie. bypassing SEQUENCER for the MergeView installation: this would allow a merge participant/coordinator to install a new MergeView. However, the consequence of this is the same as with your solution: we'd lose the ordering between MergeViews and messages. The reason why we can't simply multicast the MergeView is that the coordinator might be in a different partition, and we'd discard any multicasts coming from a member not in our (pre-MergeView) view. E.g. {A,B} and {C,D} where B and C and merge coordinators would work, as A and C would multicast the MergeView locally, and then the new coordinator (e.g. C) would be visible by all members. However, the case below wouldn't work: A: {D,A} B,C,D: {C,B,D} If the coordinator in A's partition is D, then the INSTALL_MERGE_VIEW would trigger a multicast to disseminate the new MergeView, and A (the merge participant) would forward it to D. Say the new coordinator of the MergeView is C, then C's VIEW multicast would be discarded by A. was (Author: belaban): Hmm. I assume the reason you have SEQUENCER *below* GMS is that the order of views and messages is the same everywhere. I've thought about how to keep this property in the case of a merge, but so far haven't come up with a good solution. In your suggested solution, when a merge participant gets a (unicast) INSTALL_MERGE_VIEW, it installs the MergeView directly. However, this destroys the property above, as the order of the view installation with respect to messages is not the same across all members. I experimented with marking the MergeView multicast as NO_TOTAL_ORDER, ie. bypassing SEQUENCER for the MergeView installation: this would allow a merge participant/coordinator to install a new MergeView. However, the consequence of this is the same as with your solution: we'd lose the ordering between MergeViews and messages. The reason why we can't simply multicast the MergeView is that the coordinator might be in a different partition, and we'd discard any multicasts coming from a member not in our (pre-MergeView) view. E.g. {A,B} and {C,D} where B and C and merge coordinators would work, as A and C would multicast the MergeView locally, and then the new coordinator (e.g. A) would be visible by all members. However, the case below wouldn't work: A: {D,A} B,C,D: {D,B,C} If the coordinator in A's partition is D, then the INSTALL_MERGE_VIEW would trigger a multicast to disseminate the new MergeView, and A (the merge participant) would forward it to D. Say the new coordinator of the MergeView is C, then C's VIEW multicast would be discarded by A. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1484?page=com.atlassian.jira.plugin.... ] Bela Ban edited comment on JGRP-1484 at 9/3/12 12:20 PM: --------------------------------------------------------- Hmm. I assume the reason you have SEQUENCER *below* GMS is that the order of views and messages is the same everywhere. I've thought about how to keep this property in the case of a merge, but so far haven't come up with a good solution. In your suggested solution, when a merge participant gets a (unicast) INSTALL_MERGE_VIEW, it installs the MergeView directly. However, this destroys the property above, as the order of the view installation with respect to messages is not the same across all members. I experimented with marking the MergeView multicast as NO_TOTAL_ORDER, ie. bypassing SEQUENCER for the MergeView installation: this would allow a merge participant/coordinator to install a new MergeView. However, the consequence of this is the same as with your solution: we'd lose the ordering between MergeViews and messages. The reason why we can't simply multicast the MergeView through SEQUENCER is that the coordinator (as we still see it) might be in a different partition, and it might discard forwarded messages as it is not the new coordinator in that partition. E.g. {A,B} and {C,D} where B and C and merge coordinators would work, as A and C would multicast the MergeView locally, and then the new coordinator (e.g. C) would be visible by all members. However, the case below wouldn't work: A: {D,A} B,C,D: {C,B,D} If the coordinator in A's partition is D, then the INSTALL_MERGE_VIEW would trigger a multicast to disseminate the new MergeView, and A (the merge participant) would forward it to D. Say the new coordinator of the MergeView is C, then D would discard the forwarded message as it is not the coordinator. Even if the new coord was C, and A forwarded the VIEW to C, then C's broadcast of the VIEW would be discarded by A as C is not in A's (old) view. was (Author: belaban): Hmm. I assume the reason you have SEQUENCER *below* GMS is that the order of views and messages is the same everywhere. I've thought about how to keep this property in the case of a merge, but so far haven't come up with a good solution. In your suggested solution, when a merge participant gets a (unicast) INSTALL_MERGE_VIEW, it installs the MergeView directly. However, this destroys the property above, as the order of the view installation with respect to messages is not the same across all members. I experimented with marking the MergeView multicast as NO_TOTAL_ORDER, ie. bypassing SEQUENCER for the MergeView installation: this would allow a merge participant/coordinator to install a new MergeView. However, the consequence of this is the same as with your solution: we'd lose the ordering between MergeViews and messages. The reason why we can't simply multicast the MergeView through SEQUENCER is that the coordinator (as we still see it) might be in a different partition, and it might discard forwarded messages as it is not the new coordinator in that partition. E.g. {A,B} and {C,D} where B and C and merge coordinators would work, as A and C would multicast the MergeView locally, and then the new coordinator (e.g. C) would be visible by all members. However, the case below wouldn't work: A: {D,A} B,C,D: {C,B,D} If the coordinator in A's partition is D, then the INSTALL_MERGE_VIEW would trigger a multicast to disseminate the new MergeView, and A (the merge participant) would forward it to D. Say the new coordinator of the MergeView is C, then D would discard the forwarded message as it is not the coordinator. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1484?page=com.atlassian.jira.plugin.... ] Bela Ban commented on JGRP-1484: -------------------------------- I haven't come up with a good solution to preserve ordering between messages and merge views. Unless we'd run a flush protocol to quiesce the cluster before installing a merge view, we'll have a short period of non-ordering between merge views and messages around a merge view installation. The solution I'll go with is to mark a merge view multicast as NO_TOTAL_ORDER, so it bypasses SEQUENCER and applies the merge view multicast locally. I don't think this is too bad as the application has to reconcile state after a merge anyway, as the state might have diverged in different partitions. WDYT ? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1484?page=com.atlassian.jira.plugin.... ] David Hotham edited comment on JGRP-1484 at 9/4/12 3:26 AM: ------------------------------------------------------------ I must say, I prefer my fix! It took me a while to figure out how my proposal could lead to messages and views being mis-ordered. I think it requires: * a merge participant P * whose coordinator C agrees that he is a coordinator ** (for else its SEQUENCER wouldn't broadcast C's messages at all) * but C fails to announce itself as a coordinator during the merge ** (for else P would not have been a participant) Then messages broadcast from P may be mis-ordered with respect to the MergeView. Do you agree that those are the required circumstances and problematic messages, or do you think the problem is more general than that? In contrast, if you mark the merge view multicast as NO_TOTAL_ORDER then everyone's messages are allowed to be mis-ordered with respect to the MergeView, even in perfectly normal circumstances (ie even when there's no is-C-a-coordinator-or-not? issue). was (Author: dimbleby): I must say, I prefer my fix! It took me a while to figure out how my proposal could lead to messages and views being mis-ordered. I think it requires: * a merge participant P * whose coordinator C agrees that he is a coordinator ** (for else its SEQUENCER wouldn't broadcast C's messages at all) * but who fails to announce itself as a coordinator during the merge ** (for else P would not have been a participant) Then messages broadcast from P may be mis-ordered with respect to the MergeView. Do you agree that those are the required circumstances and problematic messages, or do you think the problem is more general than that? In contrast, if you mark the merge view multicast as NO_TOTAL_ORDER then everyone's messages are allowed to be mis-ordered with respect to the MergeView, even in perfectly normal circumstances (ie even when there's no is-C-a-coordinator-or-not? issue). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1484?page=com.atlassian.jira.plugin.... ] Bela Ban commented on JGRP-1484: -------------------------------- First, we're installing the MergeView via a unicast in the participants and a broadcast in the coordinators' partition. If we're broadcasting regular messages during this, then the ordering between MergeViews and messages is undefined, as there is no ordering between unicasts and multicasts. Therefore, with both your and my solution, ordering between MergeView and messages is going to be temporarily broken. Second, I don't like to unicast MergeView in all participants. Imagine if we have hundreds or thousands of nodes: usually, we have only a few merge coordinators, but many participants. Under your proposal, this would means, we're sending hundreds of INSTALL_MERGE_VIEW unicasts to all participants. I don't like this. OK, I committed by JGRP-1484, why don't you pull my changes and see if your system still works ? Unless you bring more arguments in favor of installing MergeViews in participants directly, I'm going to go with the NO_TOTAL_ORDER solution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1484?page=com.atlassian.jira.plugin.... ] David Hotham commented on JGRP-1484: ------------------------------------ I wonder if we're having terminology confusion. You say "usually, we have only a few merge coordinators, but many participants". But I think that usually we have only a few coordinators, and (if all is well) even fewer participants. To be clear, by participant I mean a member who has reported himself to the merge leader; but for whom no coordinator claims responsibility. Eg if A, B, C report {A,B}, {A, B}, {A, B, C} then C is a participant because his supposed coordinator A does not have C in view. Then I'm proposing that the merge leader sends INSTALL_MERGE_VIEW * to A, who broadcasts it to A and B * to C, who installs it directly (whereas today C will attempt to broadcast this to A,B,C - which is pointless). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1484?page=com.atlassian.jira.plugin.... ] David Hotham edited comment on JGRP-1484 at 9/4/12 7:46 AM: ------------------------------------------------------------ I wonder if we're having terminology confusion. You say "usually, we have only a few merge coordinators, but many participants". But I think that usually we have only a few coordinators, and (if all is well) even fewer participants. To be clear, by participant I mean a member who has reported himself to the merge leader; but for whom no coordinator claims responsibility. Eg if A, B, C report {A,B}, {A, B}, {A, B, C} then C is a participant because his supposed coordinator A does not have C in view. Then I'm proposing that the merge leader sends INSTALL_MERGE_VIEW * to A, who broadcasts the new view to A and B * to C, who installs it directly (whereas today C will attempt to broadcast the new view to A,B,C - which I claim is pointless). was (Author: dimbleby): I wonder if we're having terminology confusion. You say "usually, we have only a few merge coordinators, but many participants". But I think that usually we have only a few coordinators, and (if all is well) even fewer participants. To be clear, by participant I mean a member who has reported himself to the merge leader; but for whom no coordinator claims responsibility. Eg if A, B, C report {A,B}, {A, B}, {A, B, C} then C is a participant because his supposed coordinator A does not have C in view. Then I'm proposing that the merge leader sends INSTALL_MERGE_VIEW * to A, who broadcasts it to A and B * to C, who installs it directly (whereas today C will attempt to broadcast this to A,B,C - which I claim is pointless). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1484?page=com.atlassian.jira.plugin.... ] Bela Ban commented on JGRP-1484: -------------------------------- I think there's a misunderstanding about what merge participants are. In A: {A,B,C,D} and X: {X,Y,Z}, B, C, D, Y and Z are *not* merge participants ! Merge particpants are only members which are not in any of the merge coordinators' views. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1484?page=com.atlassian.jira.plugin.... ] David Hotham commented on JGRP-1484: ------------------------------------ I think we're in agreement about what a participant is! I have _only_ changed the behaviour on a (non-coordinator) participant who receives an INSTALL_VIEW. The change is that such a member does not attempt to broadcast the view, but simply installs it. In your example at 2:48PM, the fix that I have submitted makes no change to the behaviour at all. * merge leader sends INSTALL_VIEW to A and X * A and X both broadcast the new view to their subgroup In the example at Util.determineMergeParticipants, all that happens is: * B is a coordinator, he receives INSTALL_VIEW and broadcasts it to {B,C} * A is a participant, he receives INSTALL_VIEW and just installs it (but does not broadcast it to {B,A,C}, which is what happens today). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1484?page=com.atlassian.jira.plugin.... ] Bela Ban commented on JGRP-1484: -------------------------------- IC. But since participants which are not in the coordinators' view are seldom (most merge partitions are disjunct), I don't mind sending out that multicast... :-) So I'll stick with my solution. Re changes: there is 1 small change in GMS (marking a message as NO_TOTAL_ORDER), and that's low enough risk-wise for me. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira