[infinispan-issues] [JBoss JIRA] (ISPN-5454) XSite: RetryMechanismTest random failures

[Dan Berindei (JIRA)]

     [ https://issues.jboss.org/browse/ISPN-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dan Berindei updated ISPN-5454:
-------------------------------
    Description: 
{{ClusteredCacheBackupReceiver.awaitRemoteTask()}} doesn't respect the state push command's timeout, at least when it's smaller than the sync replication timeout in the target cache. When that happens, the state provider will resend the state, and there will be 2 state push commands executing at the same time.

RetryMechanismTest changes the state push timeout to 2 seconds, but the sync replication timeout stays at 15 seconds. This causes  failures in {{testRetryLocally}} and {{testFailRetryLocally}}, if it takes more than 2 seconds to suspect the killed node.

{noformat}
10:02:13,007 TRACE (asyncTransportThread-8,NodeN:) [RetryOnFailureXSiteCommand] Sending XSiteStatePushCommand{cacheName=___defaultcache, timeout=2000 (1 keys)} to [NYC (sync, timeout=2000)]
10:02:16,008 TRACE (asyncTransportThread-8,NodeN:) [RetryOnFailureXSiteCommand] Sending XSiteStatePushCommand{cacheName=___defaultcache, timeout=2000 (1 keys)} to [NYC (sync, timeout=2000)]
10:02:16,040 TRACE (asyncTransportThread-4,NodeP:) [RpcManagerImpl] replication exception: 
org.infinispan.remoting.transport.jgroups.SuspectException: Node NodeQ-56809 was suspected
10:02:16,040 TRACE (asyncTransportThread-0,NodeP:) [RpcManagerImpl] replication exception: 
org.infinispan.remoting.transport.jgroups.SuspectException: Node NodeQ-56809 was suspected
10:02:19,147 ERROR (testng-RetryMechanismTest:) [UnitTestTestNGListener] Test testFailRetryLocally(org.infinispan.xsite.statetransfer.failures.RetryMechanismTest) failed.
java.lang.AssertionError: expected:<2> but was:<3>
	at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
	at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:364)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:80)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:245)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:252)
	at org.infinispan.xsite.statetransfer.failures.RetryMechanismTest.testFailRetryLocally(RetryMechanismTest.java:227)
{noformat}

  was:
{{ClusteredCacheBackupReceiver.awaitRemoteTask()}} doesn't respect the state push command's timeout, at least when it's smaller than the sync replication timeout in the target cache. When that happens, the state provider will resend the state, and there will be 2 state push commands executing at the same time.

RetryMechanismTest changes the state push timeout to 2 seconds, but the sync replication timeout stays at 15 seconds. This causes  failures in {{testRetryLocally}} and {{testFailRetryLocally}}, if it takes more than 2 seconds to suspect the killed node.

{noformat}
10:02:13,007 TRACE (asyncTransportThread-8,NodeN:) [RetryOnFailureXSiteCommand] Sending XSiteStatePushCommand{cacheName=___defaultcache, timeout=2000 (1 keys)} to [NYC (sync, timeout=2000)]
10:02:16,008 TRACE (asyncTransportThread-8,NodeN:) [RetryOnFailureXSiteCommand] Sending XSiteStatePushCommand{cacheName=___defaultcache, timeout=2000 (1 keys)} to [NYC (sync, timeout=2000)]
10:02:16,040 TRACE (asyncTransportThread-4,NodeP:) [RpcManagerImpl] replication exception: 
org.infinispan.remoting.transport.jgroups.SuspectException: Node NodeQ-56809 was suspected
10:02:16,040 TRACE (asyncTransportThread-0,NodeP:) [RpcManagerImpl] replication exception: 
org.infinispan.remoting.transport.jgroups.SuspectException: Node NodeQ-56809 was suspected
10:02:19,147 ERROR (testng-RetryMechanismTest:) [UnitTestTestNGListener] Test testFailRetryLocally(org.infinispan.xsite.statetransfer.failures.RetryMechanismTest) failed.
java.lang.AssertionError: expected:<2> but was:<3>
	at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
	at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:364)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:80)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:245)
	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:252)
	at org.infinispan.xsite.statetransfer.failures.RetryMechanismTest.testFailRetryLocally(RetryMechanismTest.java:227)
{noformat}
{noformat}



> XSite: RetryMechanismTest random failures
> -----------------------------------------
>
>                 Key: ISPN-5454
>                 URL: https://issues.jboss.org/browse/ISPN-5454
>             Project: Infinispan
>          Issue Type: Bug
>          Components: Cross-Site Replication
>    Affects Versions: 7.2.1.Final
>            Reporter: Dan Berindei
>            Priority: Blocker
>              Labels: testsuite_stability
>             Fix For: 8.0.0.Alpha1
>
>
> {{ClusteredCacheBackupReceiver.awaitRemoteTask()}} doesn't respect the state push command's timeout, at least when it's smaller than the sync replication timeout in the target cache. When that happens, the state provider will resend the state, and there will be 2 state push commands executing at the same time.
> RetryMechanismTest changes the state push timeout to 2 seconds, but the sync replication timeout stays at 15 seconds. This causes  failures in {{testRetryLocally}} and {{testFailRetryLocally}}, if it takes more than 2 seconds to suspect the killed node.
> {noformat}
> 10:02:13,007 TRACE (asyncTransportThread-8,NodeN:) [RetryOnFailureXSiteCommand] Sending XSiteStatePushCommand{cacheName=___defaultcache, timeout=2000 (1 keys)} to [NYC (sync, timeout=2000)]
> 10:02:16,008 TRACE (asyncTransportThread-8,NodeN:) [RetryOnFailureXSiteCommand] Sending XSiteStatePushCommand{cacheName=___defaultcache, timeout=2000 (1 keys)} to [NYC (sync, timeout=2000)]
> 10:02:16,040 TRACE (asyncTransportThread-4,NodeP:) [RpcManagerImpl] replication exception: 
> org.infinispan.remoting.transport.jgroups.SuspectException: Node NodeQ-56809 was suspected
> 10:02:16,040 TRACE (asyncTransportThread-0,NodeP:) [RpcManagerImpl] replication exception: 
> org.infinispan.remoting.transport.jgroups.SuspectException: Node NodeQ-56809 was suspected
> 10:02:19,147 ERROR (testng-RetryMechanismTest:) [UnitTestTestNGListener] Test testFailRetryLocally(org.infinispan.xsite.statetransfer.failures.RetryMechanismTest) failed.
> java.lang.AssertionError: expected:<2> but was:<3>
> 	at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
> 	at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:364)
> 	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:80)
> 	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:245)
> 	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:252)
> 	at org.infinispan.xsite.statetransfer.failures.RetryMechanismTest.testFailRetryLocally(RetryMechanismTest.java:227)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)