[jboss-jira] [JBoss JIRA] Commented: (JGRP-837) Add failure simulation capabilities to Simulator

Tue Oct 21 16:59:30 EDT 2008

    [ https://jira.jboss.org/jira/browse/JGRP-837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12434768#action_12434768 ] 

Richard Achmatowicz commented on JGRP-837:
------------------------------------------

I've now implemented a test suite called NakackRebroadcastTest which contains two tests:

1. testRebroadcastWithoutFailure 
- two senders S1 and S2 each send three messages two a receiver R1
- the second message of S1 is dropped
- the digest on S1 is obtained and the FIFO state of S1 shows that only one message is delivered
- a rebroadcast is initiated and completes
- after a time, the missing messages from S1 should be retransmitted and all three messages delivered

2. testRebroadcastWithFailure
- two senders S1 and S2 each send three messages two a receiver R1
- the second message of S1 is dropped
- the digest on S1 is obtained and the FIFO state of S1 shows that only one message is delivered
- a rebroadcast is initiated and completes
- the NAKACK.XMIT_RSP messages from S1 are dropped
- after a time, the missing messages from S1 should be retransmitted and all three messages delivered, but they are not due to dropping
- the rebroadcast completes in an incorrect state, with no warning

Here is the output from the two test cases:

    [junit] Starting test testRebroadcastWithoutFailure

    [junit] <127.0.0.1:2222>: ==> 1
    [junit] <127.0.0.1:2222>: ==> 2
    [junit] <127.0.0.1:2222>: ==> 3
    [junit] <127.0.0.1:1111>:PASS: received msg #1 from 127.0.0.1:2222
    [junit] <127.0.0.1:1111>:PASS: received msg #1 from 127.0.0.1:3333
    [junit] <127.0.0.1:3333>: ==> 1
    [junit] Dropping 2nd (mcast) MSG from sender 127.0.0.1:2222 to dest null
    [junit] <127.0.0.1:3333>: ==> 2
    [junit] <127.0.0.1:1111>:PASS: received msg #2 from 127.0.0.1:3333
    [junit] <127.0.0.1:3333>: ==> 3
    [junit] <127.0.0.1:1111>:PASS: received msg #3 from 127.0.0.1:3333
    [junit] Getting digest before rebroadcast
    [junit] Dropping 2nd (mcast) MSG from sender 127.0.0.1:2222 to dest null
    [junit] Dropping 2nd (mcast) MSG from sender 127.0.0.1:2222 to dest null
    [junit] actual digest = 127.0.0.1:1111: [0 : 0 (0)], 127.0.0.1:2222: [0 : 1 (1)], 127.0.0.1:3333: [0 : 3 (3)]
    [junit] Starting rebroadcast
    [junit] Rebroadcast completed
    [junit] Sleeping to allow rebroadcast to take hold
    [junit] <127.0.0.1:1111>:PASS: received msg #2 from 127.0.0.1:2222
    [junit] <127.0.0.1:1111>:PASS: received msg #3 from 127.0.0.1:2222
    [junit] Getting digest after rebroadcast
    [junit] actual digest = 127.0.0.1:1111: [0 : 0 (0)], 127.0.0.1:2222: [0 : 3 (3)], 127.0.0.1:3333: [0 : 3 (3)]

    [junit] Ending test testRebroadcastWithoutFailure

    [junit] Starting test testRebroadcastWithFailure

    [junit] <127.0.0.1:1111>:PASS: received msg #1 from 127.0.0.1:2222
    [junit] <127.0.0.1:2222>: ==> 1
    [junit] Dropping 2nd (mcast) MSG from sender 127.0.0.1:2222 to dest null
    [junit] Dropping 2nd (mcast) MSG from sender 127.0.0.1:2222 to dest null
    [junit] Dropping 2nd (mcast) MSG from sender 127.0.0.1:2222 to dest null
    [junit] <127.0.0.1:2222>: ==> 2
    [junit] <127.0.0.1:2222>: ==> 3
    [junit] <127.0.0.1:3333>: ==> 1
    [junit] <127.0.0.1:1111>:PASS: received msg #1 from 127.0.0.1:3333
    [junit] <127.0.0.1:3333>: ==> 2
    [junit] <127.0.0.1:1111>:PASS: received msg #2 from 127.0.0.1:3333
    [junit] <127.0.0.1:3333>: ==> 3
    [junit] <127.0.0.1:1111>:PASS: received msg #3 from 127.0.0.1:3333
    [junit] Getting digest before rebroadcast
    [junit] actual digest = 127.0.0.1:1111: [0 : 0 (0)], 127.0.0.1:2222: [0 : 1 (3)], 127.0.0.1:3333: [0 : 3 (3)]
    [junit] Starting rebroadcast
    [junit] Rebroadcast completed
    [junit] Sleeping to allow rebroadcast to take hold
    [junit] Dropping NAKACK.XMIT_RSP (unicast) from sender 127.0.0.1:2222 to dest 127.0.0.1:1111
    [junit] Dropping NAKACK.XMIT_RSP (unicast) from sender 127.0.0.1:2222 to dest 127.0.0.1:3333
    [junit] Dropping NAKACK.XMIT_RSP (unicast) from sender 127.0.0.1:2222 to dest 127.0.0.1:1111
    [junit] Dropping NAKACK.XMIT_RSP (unicast) from sender 127.0.0.1:2222 to dest 127.0.0.1:3333
    [junit] Dropping NAKACK.XMIT_RSP (unicast) from sender 127.0.0.1:2222 to dest 127.0.0.1:1111
    [junit] Dropping NAKACK.XMIT_RSP (unicast) from sender 127.0.0.1:2222 to dest 127.0.0.1:3333
    [junit] Dropping NAKACK.XMIT_RSP (unicast) from sender 127.0.0.1:2222 to dest 127.0.0.1:3333
    [junit] Dropping NAKACK.XMIT_RSP (unicast) from sender 127.0.0.1:2222 to dest 127.0.0.1:1111
    [junit] Dropping NAKACK.XMIT_RSP (unicast) from sender 127.0.0.1:2222 to dest 127.0.0.1:3333
    [junit] Dropping NAKACK.XMIT_RSP (unicast) from sender 127.0.0.1:2222 to dest 127.0.0.1:1111
    [junit] Getting digest after rebroadcast
    [junit] actual digest = 127.0.0.1:1111: [0 : 0 (0)], 127.0.0.1:2222: [0 : 1 (3)], 127.0.0.1:3333: [0 : 3 (3)]

    [junit] Ending test testRebroadcastWithFailure

    [junit] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 10.342 sec
    [junit] Test org.jgroups.tests.NakackRebroadcastTest FAILED

When message are dropped, a message is written to sysout to indicate such.
The second test is marked as failed as the rebroadcast completes without actually causing the retransmissions to occur correctly.

> Add failure simulation capabilities to Simulator 
> -------------------------------------------------
>
>                 Key: JGRP-837
>                 URL: https://jira.jboss.org/jira/browse/JGRP-837
>             Project: JGroups
>          Issue Type: Feature Request
>            Reporter: Richard Achmatowicz
>            Assignee: Richard Achmatowicz
>            Priority: Minor
>
> The Simulator can be used to test protocol layers in isolation. Multiple Simulator instances can be configured so that they model a multicast group.
> In its present state, messages are transported from one Simulator instance to another by the send_thread, which moves messages from send_queue to recv_queue in a reliable fashion.  Futhermore, all Simulators perform at the same speed. I'd like to use the Simulator to additionally simulate failures, in order to check the robustness of the protocols. In particular, i'd like to model:
> * dropped, reordered, corrupted messages
> * failed processors ('crash' failure)
> * network partitions
> * slow processes
> This JIRA issue will track progress on this and allow for discussion.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira