[arquillian-issues] [JBoss JIRA] (ARQ-2086) Test failures not reported with recent versions of TestNG

[Dipak Pawar (JIRA)]

    [ https://issues.jboss.org/browse/ARQ-2086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402929#comment-13402929 ] 

Dipak Pawar edited comment on ARQ-2086 at 5/8/17 5:48 AM:
----------------------------------------------------------

Following are few findings for this:

* All failed tests are passing? 
** As in arquillian-core we are not setting status as failed when our test is failing, we are setting only throwable which is AssertionError in failed tests.
** This was working with previous version of testng as in previous version it's checking only exception thrown, not it's status.
** So to work with testng 6.11 we have to set status as failed with throwable.
** https://github.com/cbeust/testng/commit/8fba9498efaaec18ce23572fc5a95afb6d9554c3#diff-b3080bc9d91f177f63b5fd4d9c501364R1326

* Why new SkipExecption() is not able to skip tests in case of remote execution.
** If we have some tests as Skipped then in the result we are not setting
it's throwable, we are setting only status as skipped, but after this testng is overriding it's status as success as it doesn't have throwable which is skipException here
** So to fix it set throwable even if test is skipping.
** https://github.com/dipak-pawar/arquillian-core/commit/d93431d38e18b2b54e9c17bd17c88adc0080c151#diff-4417a7a63a141484c8ecb8e3958a3924R64

* Root cause of failure of TestNGIntegrationTestCase is testng
** Look for fix https://github.com/cbeust/testng/pull/1427


All work is done, look [here|https://github.com/dipak-pawar/arquillian-core/tree/testng_issue], waiting to merge upstream pr. 

Thanks,
Dipak


was (Author: dipakpawar231):
Following are few findings for this:

* All failed tests are passing? 
** As in arquillian-core we are not setting status as failed when our test is failing, we are setting only throwable which is AssertionError in failed tests.
** This was working with previous version of testng as in previous version it's checking only exception thrown, not it's status.
** So to work with testng 6.11 we have to set status as failed with throwable.
** https://github.com/cbeust/testng/commit/8fba9498efaaec18ce23572fc5a95afb6d9554c3#diff-b3080bc9d91f177f63b5fd4d9c501364R1326

* Why new SkipExecption() is not able to skip tests in case of remote execution.
** If we have some tests as Skipped then in the result we are not setting
it's throwable, we are setting only status as skipped, but after this testng is overriding it's status as success as it doesn't have throwable which is skipException here
** So to fix it set throwable even if test is skipping.
** https://github.com/dipak-pawar/arquillian-core/commit/d93431d38e18b2b54e9c17bd17c88adc0080c151#diff-4417a7a63a141484c8ecb8e3958a3924R64

* Root cause of failure of TestNGIntegrationTestCase is testng
** Look for fix https://github.com/cbeust/testng/pull/1427


All work is done look [here|https://github.com/dipak-pawar/arquillian-core/tree/testng_issue], waiting to merge upstream pr. 

Thanks,
Dipak

> Test failures not reported with recent versions of TestNG
> ---------------------------------------------------------
>
>                 Key: ARQ-2086
>                 URL: https://issues.jboss.org/browse/ARQ-2086
>             Project: Arquillian
>          Issue Type: Bug
>    Affects Versions: 1.1.12.Final
>            Reporter: Gunnar Morling
>            Assignee: Bartosz Majsak
>            Priority: Blocker
>
> When using recent TestNG versions (> 6.8.21) with Arquillian, failing tests are accidentally reported as passing.
> A reproducer can be found at https://github.com/gunnarmorling/arquillian-testng-bug.
> There is a single test which is expected to fail, but it actually is reported as successful with TestNG 6.11. It is reported as failed as expected with 6.8.21. Run {{mvn clean install}} for reproducing.



--
This message was sent by Atlassian JIRA
(v7.2.3#72005)