The data above is taken from the log: https://hudson.qa.jboss.com/hudson/user/admin/my-views/view/rachmatowicz/job/eap-6x-failover-http-session-shutdown-repl-sync-RA/15/

This log is the client's view of sessions and their jvmRoutes.

When a node fails, sessions are moved from the failing node to only a subset of the available nodes, not all three nodes which will be abailble.
The source of the problem may have to do with DistributableCacheManager.locate() which determines which new node a session's jvmRoute should be reset to.

So, for example, when node perf21 fails (the last node to fail in the test), nodes 18, 19, and 20 are available, but all sessions have jvmRoute of perf19:

2012/06/12 16:50:07:824 EDT [INFO ][Runner - 510] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failover detected, JvmRoute changed. perf21 -> perf19
2012/06/12 16:50:07:824 EDT [INFO ][Runner - 444] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failover detected, JvmRoute changed. perf21 -> perf19
2012/06/12 16:50:07:824 EDT [INFO ][Runner - 408] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failover detected, JvmRoute changed. perf21 -> perf19
2012/06/12 16:50:07:824 EDT [INFO ][Runner - 197] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failover detected, JvmRoute changed. perf21 -> perf19
2012/06/12 16:50:07:824 EDT [INFO ][Runner - 1533] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failover detected, JvmRoute changed. perf21 -> perf19
2012/06/12 16:50:07:826 EDT [INFO ][Runner - 715] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failover detected, JvmRoute changed. perf21 -> perf19
2012/06/12 16:50:07:826 EDT [INFO ][Runner - 1578] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failover detected, JvmRoute changed. perf21 -> perf19
2012/06/12 16:50:07:827 EDT [INFO ][Runner - 868] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failover detected, JvmRoute changed. perf21 -> perf19

[snip]

2012/06/12 16:50:12:726 EDT [INFO ][Runner - 141] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failover detected, JvmRoute changed. perf21 -> perf19
2012/06/12 16:50:12:726 EDT [INFO ][Runner - 1865] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failover detected, JvmRoute changed. perf21 -> perf19
2012/06/12 16:50:12:726 EDT [INFO ][Runner - 929] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failover detected, JvmRoute changed. perf21 -> perf19
2012/06/12 16:50:12:730 EDT [INFO ][Runner - 450] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failover detected, JvmRoute changed. perf21 -> perf19
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira