[arquillian-issues] [JBoss JIRA] (ARQ-563) @ArquillianResource in TestNG based tests causes test body not to be evaluated

[Kristoffer Richardsson (JIRA)]

    [ https://issues.jboss.org/browse/ARQ-563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710040#comment-12710040 ] 

Kristoffer Richardsson commented on ARQ-563:
--------------------------------------------

I have seen this problem as well, similar to the Hanspeter Gisler scenario. A test class like this will pass without any errors.

{code}
public class TestNotEvaluatedExample extends Arquillian {
    private ClassThatIsNotIncludedInJar instance;

    @Deployment
    public static JavaArchive createTestArchive() {
        return ShrinkWrap.create(JavaArchive.class, "test.jar");
    }

    @Test
    public void testThatIsNotEvaluated() {
        Assert.fail("Expecting this assert to fail but it is not executed");
    }
}
{code}

It seems as testNG actually catches the error but that arquillian ignores it. In {{TestListener.getTestResult()}}, {{context}} is examined for failedTests, skippedTests and passedTests, but testNG reports this problem as a failed configuration, not a failed test. 
{code}
   public TestResult getTestResult() 
   {
      if(context.getFailedTests().size() > 0) 
      {
         return new TestResult(
               Status.FAILED, 
               context.getFailedTests().getAllResults().iterator().next().getThrowable());
      } 
      else if(context.getSkippedTests().size() > 0)
      {
         return new TestResult(Status.SKIPPED);
      }
      if(context.getPassedTests().size() > 0) 
      {
         return new TestResult(
               Status.PASSED, 
               context.getPassedTests().getAllResults().iterator().next().getThrowable());
      } 
      return new TestResult(
            Status.FAILED, 
            new RuntimeException("Unknown test result: " + context).fillInStackTrace());
   }
{code}

The problem can be solved by checking {{context.getFailedConfigurations()}} as well.
{code}
   public TestResult getTestResult() 
   {
       if(context.getFailedConfigurations().size() > 0)
       {
          return new TestResult(
             Status.FAILED,
             context.getFailedConfigurations().getAllResults().iterator().next().getThrowable());
       }
       else if(context.getFailedTests().size() > 0)
       {
          return new TestResult(
             Status.FAILED,
             context.getFailedTests().getAllResults().iterator().next().getThrowable());
      }...
{code}

I'll add a pull request for this.
                
> @ArquillianResource in TestNG based tests causes test body not to be evaluated
> ------------------------------------------------------------------------------
>
>                 Key: ARQ-563
>                 URL: https://issues.jboss.org/browse/ARQ-563
>             Project: Arquillian
>          Issue Type: Bug
>      Security Level: Public(Everyone can see) 
>          Components: Test Protocol SPIs and Implementation
>    Affects Versions: 1.0.0.CR4
>         Environment: Surefire 2.7.2, 2.9
> TestNG 5.14.6, 5.14.9
> AS 7.0.1 web
>            Reporter: Karel Piwko
>            Assignee: Andrew Rubinger
>            Priority: Blocker
>             Fix For: 1.1.0.Beta1
>
>
> Suppose following test case:
> {code}
> public class LowercaseConverterTest extends Arquillian {
> 	@Inject
> 	private LowercaseConverter lowercaseConverter;
> 	@Deployment
> 	public static JavaArchive createTestArchive() {
> 		return ShrinkWrap
> 				.create(JavaArchive.class, "test.jar")
> 				.addClasses(LowercaseConverter.class)
> 				.addAsManifestResource(EmptyAsset.INSTANCE,
> 						ArchivePaths.create("beans.xml"));
> 	}
> 	@Test
> 	public void testConvertToLowercase() {	    
> 	    
> 		Assert.assertEquals("martin",
> 				lowercaseConverter.convertToLowercase("Martin"));
> 	}
> }
> {code}
> However, if @ArquillianResource is used for a field injection, such as:
> {code}
> @ArquillianResource URL field;	
> {code}
> than test body is no longer executed and *all tests are reported as passed* even an obvious failure like:
> {code}
> @Test
> 	public void testConvertToLowercase() {	    
> 	    
> 	    Assert.fail();
> 	    
> 		Assert.assertEquals("karel",
> 				lowercaseConverter.convertToLowercase("Martin"));
> 	}
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira