[ https://jira.jboss.org/jira/browse/JGRP-837?page=com.atlassian.jira.plugi... ] Richard Achmatowicz commented on JGRP-837: ------------------------------------------ I've currently added these changes to the Simulator class, but haven't committed anything yet. The changes to the operation of Simulator are fairly compact. I've also created a test case called SimulatorTest which seta up a group of three simulators with a non-existennt protocol layer (so that we are effectively simulating the operation of a TP layer with no FIFO, no reliability, etc) and test each feature. There is the possibility of keeping Simulator as is, and adding in the new functionality using a subclass SimulatorWithFaultInjection. Any preferences? -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JGRP-837?page=com.atlassian.jira.plugi... ] Richard Achmatowicz updated JGRP-837: ------------------------------------- Summary: Add failure simulation capabilities to Simulator (was: Add failure simulation capabiliries to Simulator ) Complexity: Low -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JGRP-837?page=com.atlassian.jira.plugi... ] Richard Achmatowicz commented on JGRP-837: ------------------------------------------ I've now implemented a test suite called NakackRebroadcastTest which contains two tests: 1. testRebroadcastWithoutFailure - two senders S1 and S2 each send three messages two a receiver R1 - the second message of S1 is dropped - the digest on S1 is obtained and the FIFO state of S1 shows that only one message is delivered - a rebroadcast is initiated and completes - after a time, the missing messages from S1 should be retransmitted and all three messages delivered 2. testRebroadcastWithFailure - two senders S1 and S2 each send three messages two a receiver R1 - the second message of S1 is dropped - the digest on S1 is obtained and the FIFO state of S1 shows that only one message is delivered - a rebroadcast is initiated and completes - the NAKACK.XMIT_RSP messages from S1 are dropped - after a time, the missing messages from S1 should be retransmitted and all three messages delivered, but they are not due to dropping - the rebroadcast completes in an incorrect state, with no warning Here is the output from the two test cases: [junit] Starting test testRebroadcastWithoutFailure [junit] <127.0.0.1:2222>: ==> 1 [junit] <127.0.0.1:2222>: ==> 2 [junit] <127.0.0.1:2222>: ==> 3 [junit] <127.0.0.1:1111>:PASS: received msg #1 from 127.0.0.1:2222 [junit] <127.0.0.1:1111>:PASS: received msg #1 from 127.0.0.1:3333 [junit] <127.0.0.1:3333>: ==> 1 [junit] Dropping 2nd (mcast) MSG from sender 127.0.0.1:2222 to dest null [junit] <127.0.0.1:3333>: ==> 2 [junit] <127.0.0.1:1111>:PASS: received msg #2 from 127.0.0.1:3333 [junit] <127.0.0.1:3333>: ==> 3 [junit] <127.0.0.1:1111>:PASS: received msg #3 from 127.0.0.1:3333 [junit] Getting digest before rebroadcast [junit] Dropping 2nd (mcast) MSG from sender 127.0.0.1:2222 to dest null [junit] Dropping 2nd (mcast) MSG from sender 127.0.0.1:2222 to dest null [junit] actual digest = 127.0.0.1:1111: [0 : 0 (0)], 127.0.0.1:2222: [0 : 1 (1)], 127.0.0.1:3333: [0 : 3 (3)] [junit] Starting rebroadcast [junit] Rebroadcast completed [junit] Sleeping to allow rebroadcast to take hold [junit] <127.0.0.1:1111>:PASS: received msg #2 from 127.0.0.1:2222 [junit] <127.0.0.1:1111>:PASS: received msg #3 from 127.0.0.1:2222 [junit] Getting digest after rebroadcast [junit] actual digest = 127.0.0.1:1111: [0 : 0 (0)], 127.0.0.1:2222: [0 : 3 (3)], 127.0.0.1:3333: [0 : 3 (3)] [junit] Ending test testRebroadcastWithoutFailure [junit] Starting test testRebroadcastWithFailure [junit] <127.0.0.1:1111>:PASS: received msg #1 from 127.0.0.1:2222 [junit] <127.0.0.1:2222>: ==> 1 [junit] Dropping 2nd (mcast) MSG from sender 127.0.0.1:2222 to dest null [junit] Dropping 2nd (mcast) MSG from sender 127.0.0.1:2222 to dest null [junit] Dropping 2nd (mcast) MSG from sender 127.0.0.1:2222 to dest null [junit] <127.0.0.1:2222>: ==> 2 [junit] <127.0.0.1:2222>: ==> 3 [junit] <127.0.0.1:3333>: ==> 1 [junit] <127.0.0.1:1111>:PASS: received msg #1 from 127.0.0.1:3333 [junit] <127.0.0.1:3333>: ==> 2 [junit] <127.0.0.1:1111>:PASS: received msg #2 from 127.0.0.1:3333 [junit] <127.0.0.1:3333>: ==> 3 [junit] <127.0.0.1:1111>:PASS: received msg #3 from 127.0.0.1:3333 [junit] Getting digest before rebroadcast [junit] actual digest = 127.0.0.1:1111: [0 : 0 (0)], 127.0.0.1:2222: [0 : 1 (3)], 127.0.0.1:3333: [0 : 3 (3)] [junit] Starting rebroadcast [junit] Rebroadcast completed [junit] Sleeping to allow rebroadcast to take hold [junit] Dropping NAKACK.XMIT_RSP (unicast) from sender 127.0.0.1:2222 to dest 127.0.0.1:1111 [junit] Dropping NAKACK.XMIT_RSP (unicast) from sender 127.0.0.1:2222 to dest 127.0.0.1:3333 [junit] Dropping NAKACK.XMIT_RSP (unicast) from sender 127.0.0.1:2222 to dest 127.0.0.1:1111 [junit] Dropping NAKACK.XMIT_RSP (unicast) from sender 127.0.0.1:2222 to dest 127.0.0.1:3333 [junit] Dropping NAKACK.XMIT_RSP (unicast) from sender 127.0.0.1:2222 to dest 127.0.0.1:1111 [junit] Dropping NAKACK.XMIT_RSP (unicast) from sender 127.0.0.1:2222 to dest 127.0.0.1:3333 [junit] Dropping NAKACK.XMIT_RSP (unicast) from sender 127.0.0.1:2222 to dest 127.0.0.1:3333 [junit] Dropping NAKACK.XMIT_RSP (unicast) from sender 127.0.0.1:2222 to dest 127.0.0.1:1111 [junit] Dropping NAKACK.XMIT_RSP (unicast) from sender 127.0.0.1:2222 to dest 127.0.0.1:3333 [junit] Dropping NAKACK.XMIT_RSP (unicast) from sender 127.0.0.1:2222 to dest 127.0.0.1:1111 [junit] Getting digest after rebroadcast [junit] actual digest = 127.0.0.1:1111: [0 : 0 (0)], 127.0.0.1:2222: [0 : 1 (3)], 127.0.0.1:3333: [0 : 3 (3)] [junit] Ending test testRebroadcastWithFailure [junit] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 10.342 sec [junit] Test org.jgroups.tests.NakackRebroadcastTest FAILED When message are dropped, a message is written to sysout to indicate such. The second test is marked as failed as the rebroadcast completes without actually causing the retransmissions to occur correctly. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JGRP-837?page=com.atlassian.jira.plugi... ] Bela Ban commented on JGRP-837: ------------------------------- Please commit your changes. I assume they won't affect existing unit tests, right ? Regarding public void simulateSlowProcess(long delay), maybe we should add a randomization parameter, same for dropping of messages. I also wonder whether we cannot simply add a protocol (like DISCARD or DELAY) to a stack with Simulator, rather than simulating dropped messages in the Simulator directly ? Regarding rebroadcasting of messages in NAKACK: let's move this issue to a separate JIRA task and discuss it from there. A member cannot cease receiving/sending messages forever, because it would get suspected and excluded, so we would have a recusrive FLUSH phase. Meaning, that if we flush {A,B,C,D} and - before the flush completes, C crashes - we'd continue the FLUSH phase, this time excluding C and reconciling messages for C. This scenario should be IMO also tested by your new simulator tests. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JGRP-837?page=com.atlassian.jira.plugi... ] Richard Achmatowicz commented on JGRP-837: ------------------------------------------ Committed a version of the Simulator to 2.7 which handles delays, partitions and dropped messages. An example of its use is in the functional test case SimulatorTest.java. I changed the DropMessage interface from public interface DropMessage { public boolean drop(Message msg) ; } to public interface DropMessage { public boolean drop(Message msg, Address dest) ; } The Simulator implelements multicast by transferrng a copy of the multicast message to the receive queue of each configured Simulator instance. So when a message is multicast from one sender to a group {A,B,C}, the drop function gets called once for each peer: drop(msg, A) drop(msg,B) drop(msg,C) This allows the drop function to drop the multicast in its entirety, or only one or more of the messages making up the multicast (set of unicasts), as we now know, for each unicast, which peer it is destined for. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JGRP-837?page=com.atlassian.jira.plugi... ] Richard Achmatowicz closed JGRP-837. ------------------------------------ Resolution: Partially Completed Closing until I have time to return to this issue. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira