Merge doesn't work properly --------------------------- Key: JGRP-2197 URL: https://issues.jboss.org/browse/JGRP-2197 Project: JGroups Issue Type: Bug Affects Versions: 3.4.2 Reporter: Allen Zhao Assignee: Bela Ban Priority: Critical Attachments: MergeTest.java, jgroups.log, jgroupsConfig.xml Using the attached jGroups configuration file and sample code based on the Draw example, run two MergeTest instances alanz-dev-54589 and alanz-dev-53593 on one machine, and another instance , AllenMAC-4773 on the second machine; both machines are in the same network connected by a cable. Initially all three instances are in the same group, then unplug the cable from the second machine, two groups formed as expected: group1: alanz-dev-54589 and alanz-dev-53593 group2: AllenMAC-4773 Then plug the cable into the second machine again, the groups merged into one group with all the three members. These are good as expected. Keeping doing these by unplug/plug the cable into the second machine, it took around 5 times until the following happened, which is unexpected: when AllenMAC-4773 merged into the group composed by alanz-dev-54589 and alanz-dev-53593, it kicked alanz-dev-54589 out, and the two groups formed: MergeView::[alanz-dev-53593|14] (2) [alanz-dev-53593, AllenMAC-4773], and alanz-dev-54589 formed a group by itself. Please see the sample code output from the two instances run on the first machine as follows: ==============alanz-dev-54589=================== ------------------------------------------------------------------- GMS: address=alanz-dev-54589, cluster=draw-cluster, physical address=fe80:0:0:0:1c97:64f4:50a0:20b0%11:63608 ------------------------------------------------------------------- ** View=[alanz-dev-54589|0] (1) [alanz-dev-54589] ** View=[alanz-dev-54589|1] (2) [alanz-dev-54589, AllenMAC-4773] ** View=[alanz-dev-54589|2] (3) [alanz-dev-54589, AllenMAC-4773, alanz-dev-53593] ** View=[alanz-dev-54589|3] (2) [alanz-dev-54589, alanz-dev-53593] ** MergeView::[alanz-dev-54589|4] (3) [alanz-dev-54589, alanz-dev-53593, AllenMAC-4773], 2 subgroups: [alanz-dev-54589|3] (2) [alanz-dev-54589, alanz-dev-53593], [AllenMAC-4773|3] (1) [AllenMAC-4773] ** View=[alanz-dev-54589|5] (2) [alanz-dev-54589, alanz-dev-53593] ** MergeView::[alanz-dev-54589|6] (3) [alanz-dev-54589, alanz-dev-53593, AllenMAC-4773], 2 subgroups: [alanz-dev-54589|5] (2) [alanz-dev-54589, alanz-dev-53593], [AllenMAC-4773|5] (1) [AllenMAC-4773] ** View=[alanz-dev-54589|7] (2) [alanz-dev-54589, alanz-dev-53593] ** MergeView::[alanz-dev-54589|8] (3) [alanz-dev-54589, alanz-dev-53593, AllenMAC-4773], 2 subgroups: [alanz-dev-54589|7] (2) [alanz-dev-54589, alanz-dev-53593], [AllenMAC-4773|7] (1) [AllenMAC-4773] ** View=[alanz-dev-54589|9] (2) [alanz-dev-54589, alanz-dev-53593] ** MergeView::[alanz-dev-54589|10] (3) [alanz-dev-54589, alanz-dev-53593, AllenMAC-4773], 2 subgroups: [alanz-dev-54589|9] (2) [alanz-dev-54589, alanz-dev-53593], [AllenMAC-4773|9] (1) [AllenMAC-4773] ** View=[alanz-dev-54589|11] (2) [alanz-dev-54589, alanz-dev-53593] ** MergeView::[alanz-dev-54589|12] (3) [alanz-dev-54589, alanz-dev-53593, AllenMAC-4773], 2 subgroups: [alanz-dev-54589|11] (2) [alanz-dev-54589, alanz-dev-53593], [AllenMAC-4773|11] (1) [AllenMAC-4773] ** View=[alanz-dev-54589|13] (2) [alanz-dev-54589, alanz-dev-53593] ** MergeView::[alanz-dev-53593|15] (3) [alanz-dev-53593, AllenMAC-4773, alanz-dev-54589], 2 subgroups: [alanz-dev-53593|14] (2) [alanz-dev-53593, AllenMAC-4773], [alanz-dev-54589|13] (1) [alanz-dev-54589] ================alanz-dev-53593======================= --------------------------------------------------------------------- GMS: address=alanz-dev-53593, cluster=draw-cluster, physical address=fe80:0:0:0:1c97:64f4:50a0:20b0%11:63610 ------------------------------------------------------------------- ** View=[alanz-dev-54589|2] (3) [alanz-dev-54589, AllenMAC-4773, alanz-dev-53593] ** View=[alanz-dev-54589|3] (2) [alanz-dev-54589, alanz-dev-53593] ** MergeView::[alanz-dev-54589|4] (3) [alanz-dev-54589, alanz-dev-53593, AllenMAC-4773], 2 subgroups: [alanz-dev-54589|3] (2) [alanz-dev-54589, alanz-dev-53593], [AllenMAC-4773|3] (1) [AllenMAC-4773] ** View=[alanz-dev-54589|5] (2) [alanz-dev-54589, alanz-dev-53593] ** MergeView::[alanz-dev-54589|6] (3) [alanz-dev-54589, alanz-dev-53593, AllenMAC-4773], 2 subgroups: [alanz-dev-54589|5] (2) [alanz-dev-54589, alanz-dev-53593], [AllenMAC-4773|5] (1) [AllenMAC-4773] ** View=[alanz-dev-54589|7] (2) [alanz-dev-54589, alanz-dev-53593] ** MergeView::[alanz-dev-54589|8] (3) [alanz-dev-54589, alanz-dev-53593, AllenMAC-4773], 2 subgroups: [alanz-dev-54589|7] (2) [alanz-dev-54589, alanz-dev-53593], [AllenMAC-4773|7] (1) [AllenMAC-4773] ** View=[alanz-dev-54589|9] (2) [alanz-dev-54589, alanz-dev-53593] ** MergeView::[alanz-dev-54589|10] (3) [alanz-dev-54589, alanz-dev-53593, AllenMAC-4773], 2 subgroups: [alanz-dev-54589|9] (2) [alanz-dev-54589, alanz-dev-53593], [AllenMAC-4773|9] (1) [AllenMAC-4773] ** View=[alanz-dev-54589|11] (2) [alanz-dev-54589, alanz-dev-53593] ** MergeView::[alanz-dev-54589|12] (3) [alanz-dev-54589, alanz-dev-53593, AllenMAC-4773], 2 subgroups: [alanz-dev-54589|11] (2) [alanz-dev-54589, alanz-dev-53593], [AllenMAC-4773|11] (1) [AllenMAC-4773] ** View=[alanz-dev-54589|13] (2) [alanz-dev-54589, alanz-dev-53593] ** MergeView::[alanz-dev-53593|14] (2) [alanz-dev-53593, AllenMAC-4773], 2 subgroups: [alanz-dev-54589|13] (1) [alanz-dev-53593], [AllenMAC-4773|13] (1) [AllenMAC-4773] ** MergeView::[alanz-dev-53593|15] (3) [alanz-dev-53593, AllenMAC-4773, alanz-dev-54589], 2 subgroups: [alanz-dev-53593|14] (2) [alanz-dev-53593, AllenMAC-4773], [alanz-dev-54589|13] (1) [alanz-dev-54589] The issue happened in our system which uses jGroups for data replication. I turned on all jGroups log, and got some useful log information as the attached. It seems that at some point, a merge request from AllenMAC-4773 was received by alanz-dev-53593, and in that message, mbrs only contained alanz-dev-53593.