[jboss-as7-dev] Keeping testsuite "API" consistent // -DskipTests argument no longer ...

[Ondřej Žižka]

Jaikiran Pai píše v Po 28. 11. 2011 v 15:05 +0530:

> Comments inline.

-"-



> Since these flags apply to the EAP testing, why 
> not have a specific script which deals with all these flags and that 
> script can then be used for EAP testing? 

Well, since the testsuite is maven-based, the script can only use what's
really in pom.xml's.
I could move some steps (e.g. server config) to the script.
But I think that having all in one place is beneficial: If a bug in EAP
is found, then it's easy for the developer to reproduce - they'll simply
use same set of maven properties.

Maybe my words "more and more" scared you a bit :)   It's not that bad,
especially after splitting to modules, the pom.xml's are not that hard
to read anymore.
And since we have gone the modularized way, I can keep the functionality
outside of scope of the "normal" build, i.e. not include some modules at
all - only for EAP builds / test runs.

What I mean is:
testsuite
  + integration
     + configure-database
     + configure-ipv6
     + group-smoke
     + group-basic
     + group-iiop

And activate the submodules on demand.



> My point is not about adding 
> these new flags but is more about having to expect the non-QA developers 
> (for whom those are irrelevant) to understand and use these flags for 
> the builds.

Actually, I try not to change the behavior unless someone asks for some
change - that's the case of -DskipTests btw.
I didn't assume anyone relying on that.
And, this was not in testsuite requirements, which we discussed for a
while on this list.
But as I wrote - it's coming back.


> 
> > But I can keep you uninformed and keep the testsuite running without 
> > changes of "API" if that helps.
> I guess, you meant informed. Yes, that would help - if adding the flags 
> helps QA testing but doesn't have any impact on the "API" and helps keep 
> the testsuite running, then I have no objection to those changes/flags.

Okay. How about others?



> >>   The actual
> >> question for this specific issue is what's the real usecase of skipping
> >> unit tests (which potentially can have failures) but still want to run
> >> the integration tests?
> > Often it happens that some unit tests fail and block one step AS build 
> > & testsuite execution.
> So far I haven't seen any unit test fail in AS7 (and if some of them are 
> failing then those need to be fixed). Yes, there are integration test 
> failures which are intermittent. But those issues need to be fixed. Like 
> I said in my earlier reply, building the entire project by using 
> -DskipTests (to skip all tests) and then cd testsuite/integration; mvn 
> clean install (to run the integration tests) is what you are looking for 
> in such cases. Ofcourse it's a 2 step process but to me that's more 
> logical (and can even be added to a script) than adding and integrating 
> some new property.

Okay. Someone from QA wanted to be able to run in one step due to some
Jenkins issue with Maven-based build, I'll discuss that.


> 
> > And let me ask the other way around - what is the use case of 
> > -DskipTests, skipping all tests?
> Yes.

That was like a part of the question - what is the use case of skipping
_all_ tests?
* As you said, unit tests should be rock solid and you rarely see a
failure.
* So you skip tests to make the build faster, not because of unit tests
failures.
* So you'd probably like to skip testsuite module to make it even
faster.
Then I don't understand why you dislike the idea of having a separated
flag to skip the testsuite. ?


Ondra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/jboss-as7-dev/attachments/20111128/3a8ac5e7/attachment-0001.html