DefaultExecutorService failoverExecution does not rethrow the root cause of the failure --------------------------------------------------------------------------------------- Key: ISPN-2409 URL: https://issues.jboss.org/browse/ISPN-2409 Project: Infinispan Issue Type: Bug Components: Distributed Execution and Map/Reduce Reporter: Anna Manukyan Assignee: Vladimir Blagojevic Fix For: 5.2.0.Beta6, 5.2.0.Final Hi, in case if Exception is thrown in Callable/DistributedCallable during execution with DistributedExecutor, it often happens that the real root cause of the exception is not shown - rethrown, but the following exception appears (see below). In this case, the the code maintenance becomes harder, as it is not visible what caused the issue. The pull request for the reproduction is: https://github.com/infinispan/infinispan/pull/1398 I'm just thinking, maybe this issue relates to ISPN-1636 - where it is said that "injection should happen again on each node, avoiding serialization of the injected components", but in case of failover - serialization is performed. Just thoughts. java.util.concurrent.ExecutionException: Failover execution failed at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.failoverExecution(DefaultExecutorService.java:839) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:801) at org.infinispan.distexec.DistributedExecutorTest.basicInvocation(DistributedExecutorTest.java:86) at org.infinispan.cdi.test.distexec.DistributedExecutorCDITest.testInvocationException(DistributedExecutorCDITest.java:89) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:74) at org.testng.internal.MethodInvocationHelper$1.runTestMethod(MethodInvocationHelper.java:176) at org.jboss.arquillian.testng.Arquillian$1.invoke(Arquillian.java:111) at org.jboss.arquillian.container.test.impl.execution.LocalTestExecuter.execute(LocalTestExecuter.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:99) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:81) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:134) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114) at org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67) at org.jboss.arquillian.container.test.impl.client.protocol.local.LocalContainerMethodExecutor.invoke(LocalContainerMethodExecutor.java:50) at org.jboss.arquillian.container.test.impl.execution.RemoteTestExecuter.execute(RemoteTestExecuter.java:120) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:99) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:81) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:134) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114) at org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67) at org.jboss.arquillian.container.test.impl.execution.ClientTestExecuter.execute(ClientTestExecuter.java:57) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:99) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:81) at org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:130) at org.jboss.arquillian.container.test.impl.client.ContainerEventController.createTestContext(ContainerEventController.java:117) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:82) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:68) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:54) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:134) at org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.test(EventTestRunnerAdaptor.java:111) at org.jboss.arquillian.testng.Arquillian.run(Arquillian.java:102) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.testng.internal.MethodInvocationHelper.invokeHookable(MethodInvocationHelper.java:189) at org.testng.internal.Invoker.invokeMethod(Invoker.java:666) at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:846) at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1170) at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125) at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109) at org.testng.TestRunner.runWorkers(TestRunner.java:1147) at org.testng.TestRunner.privateRun(TestRunner.java:749) at org.testng.TestRunner.run(TestRunner.java:600) at org.testng.SuiteRunner.runTest(SuiteRunner.java:317) at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:312) at org.testng.SuiteRunner.privateRun(SuiteRunner.java:274) at org.testng.SuiteRunner.run(SuiteRunner.java:223) at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52) at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86) at org.testng.TestNG.runSuitesSequentially(TestNG.java:1039) at org.testng.TestNG.runSuitesLocally(TestNG.java:964) at org.testng.TestNG.run(TestNG.java:900) at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:77) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:110) at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:106) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:113) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) Caused by: java.lang.Exception: Failover execution failed ... 108 more Caused by: java.util.concurrent.ExecutionException: Failover execution failed at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.failoverExecution(DefaultExecutorService.java:839) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:801) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.failoverExecution(DefaultExecutorService.java:834) ... 107 more Caused by: java.lang.Exception: Failover execution failed ... 110 more Caused by: java.util.concurrent.ExecutionException: Failover execution failed at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.failoverExecution(DefaultExecutorService.java:836) ... 109 more Caused by: java.lang.Exception: Failover execution failed ... 110 more Caused by: java.util.concurrent.ExecutionException: org.infinispan.CacheException: java.lang.RuntimeException: Failure to marshal argument(s) at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) at java.util.concurrent.FutureTask.get(FutureTask.java:111) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:798) ... 108 more Caused by: org.infinispan.CacheException: java.lang.RuntimeException: Failure to marshal argument(s) at org.infinispan.util.Util.rewrapAsCacheException(Util.java:532) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:184) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:514) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:176) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:198) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:255) at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:81) at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:289) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.RuntimeException: Failure to marshal argument(s) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:278) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:296) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:177) ... 11 more Caused by: org.infinispan.marshall.NotSerializableException: org.infinispan.CacheImpl Caused by: an exception which occurred: in field cache in object org.infinispan.cdi.test.distexec.DistributedExecutorCDITest$DistributedCacheCallable@4c824d8 -> toString = org.infinispan.cdi.test.distexec.DistributedExecutorCDITest$DistributedCacheCallable@4c824d8 in object org.infinispan.commands.read.DistributedExecuteCommand@1f -> toString = DistributedExecuteCommand{cache=Cache 'DistributedExecutorTest-DIST_SYNC'@DistributedExecutorCDITest-NodeA-37398, keys=[], callable=org.infinispan.cdi.test.distexec.DistributedExecutorCDITest$DistributedCacheCallable@4c824d8} in object org.infinispan.commands.remote.SingleRpcCommand@e8fbe1d0 -> toString = SingleRpcCommand{cacheName='DistributedExecutorTest-DIST_SYNC', command=DistributedExecuteCommand{cache=Cache 'DistributedExecutorTest-DIST_SYNC'@DistributedExecutorCDITest-NodeA-37398, keys=[], callable=org.infinispan.cdi.test.distexec.DistributedExecutorCDITest$DistributedCacheCallable@4c824d8}}