Random failures in DistributedExecutorTest.testBasicTargetRemoteDistributedCallableWithHighFutureAndLowTaskTimeout and subclasses --------------------------------------------------------------------------------------------------------------------------------- Key: ISPN-2975 URL: https://issues.jboss.org/browse/ISPN-2975 Project: Infinispan Issue Type: Bug Components: RPC, Test Suite Affects Versions: 5.2.0.Final Reporter: Dan Berindei Assignee: Dan Berindei Fix For: 5.3.0.Alpha1 DistributedTaskBuilder supports setting a timeout for the distributed task. That timeout is used in two places: both a timeout for the FutureTask submitted to the local executor, and as a timeout for RPC invocations. The test testBasicTargetRemoteDistributedCallableWithHighFutureAndLowTaskTimeout expects a TimeoutException, which I think is correct. But the distributed task future doesn't throw a TimeoutException if the RPC timed out, instead it throws something like this: {noformat} org.testng.TestException: Expected exception java.util.concurrent.TimeoutException but got java.util.concurrent.ExecutionException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to ReplSyncDistributedExecutorTest-NodeV-34575org.testng.TestException:Expected exception java.util.concurrent.TimeoutException but got java.util.concurrent.ExecutionException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to ReplSyncDistributedExecutorTest-NodeV-34575 at org.testng.internal.Invoker.handleInvocationResults(Invoker.java:1497) at org.testng.internal.Invoker.invokeMethod(Invoker.java:754) at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901) at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231) at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127) at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111) at org.testng.TestRunner.privateRun(TestRunner.java:767) at org.testng.TestRunner.run(TestRunner.java:617) at org.testng.SuiteRunner.runTest(SuiteRunner.java:334) at org.testng.SuiteRunner.access$000(SuiteRunner.java:37) at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368) at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to ReplSyncDistributedExecutorTest-NodeV-34575 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232) at java.util.concurrent.FutureTask.get(FutureTask.java:91) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:948) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:927) at org.infinispan.distexec.DistributedExecutorTest.testBasicTargetRemoteDistributedCallableWithHighFutureAndLowTaskTimeout(DistributedExecutorTest.java:159) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80) at org.testng.internal.Invoker.invokeMethod(Invoker.java:714) ... 15 more Caused by: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to ReplSyncDistributedExecutorTest-NodeV-34575 at org.infinispan.util.Util.rewrapAsCacheException(Util.java:542) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:186) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247) at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:79) at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:281) ... 5 more Caused by: org.jgroups.TimeoutException: timeout sending message to ReplSyncDistributedExecutorTest-NodeV-34575 at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:418) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:301) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179) ... 11 more {noformat} I believe CommandAwareRpcDispatcher should wrap org.jgroups.TimeoutException in a org.infinispan.util.concurrent.TimeoutException, and DistributedTaskPart.innerGet should also treat ExecutionExceptions containing TimeoutExceptions specially.