[jboss-as7-dev] Testsuite modularization? // Re: Testsuite refactor

[Jason T. Greene]

On 11/22/11 2:31 PM, Ondřej Žižka wrote:
> Hi,
>
> 1) Relevant resources:
>
> Tracking jira: https://issues.jboss.org/browse/AS7-2007
> Requirements: http://community.jboss.org/wiki/ASTestsuiteRequirements
> Docs: https://docs.jboss.org/author/display/AS71/Testsuite
>
>
> 2) Modularization & moving the test sources
>
>  > This is one thing I am wondering. If we all agree that many smaller
> modules is the way to go, since its the standard maven approach
> (supported by all existing maven tools), and it solves that various
> issues introduced by doing the grouping approach then maybe we should
> just do one refactor instead of doing this one and refactoring it yet again?
>
> I'd like to see the problems with the solution in PR 710. Please send me
> instructions how to check that the IDE has problems coping with test
> sources on ../src/test/java.
> (I don't know about Eclipse or IDEA, but NetBeans uses quite rational
> approach, which is simply running Maven for the active module. Which works.
> It works for smoke, for -DallTests if you add that param, and also when
> running a single test. I haven't tried debugging yet but in worst case,
> I'd run with -Ddebug and attach the debugger.
)

I think its the same issues Stuart already brought up a few weeks ago. 
One megaclasspath, larger build footprint, consistency with the modules 
directory etc.

>
>  > I know you put a lot of effort into this, but there is a good chance
> this may net less work in the end since we wont be constantly tweaking
> this version.
>
> It's not like switching from version A to B1 or B2, it's like staying at
> B (pull request 710) or going further to C (which would be moving
> sources to modules).
> This harness is quite ready for both alternatives.
> The reason I am proposing this as an intermediate state is, as I
> previously stated, merely technical: Moving thousands of files is a big
> operation from SCM PoV, so I want to get this tests relocation get
> merged as quickly as possible - i.e. move, test, make pull request, get
> it reviewed and merged, at best in a single day, otherwise I'll drown
> with new tests coming.

I guess I don't see what if anything in this pull we would want to keep 
if we went to a small module approach?

  Maybe you could explain that?

>
>  > Or is there some set of requirements we have to meet ASAP that our
> existing testsuite infrastructure doesnt cover?
>
> There are many problems - I've mentioned them before - like:
> * Smoke profile can either be default and need to be disabled
> explicitely for any other group; but in that case many people will
> forget to disable it which causes inference with other profiles.
> Or it would have to be defined explicitely - i.e. "no smoke tests run by
> default"
> * Huge unmaintainable pom.xml
> * Impossible to define multiple groups

These all sound livable for awhile to me.

> * Hard to separate web/full or even make all tests run with full
> Etc etc. See the resources above, and the issues discussed on as7 list
> recently.

This works today in master. I just added multiple executions.

>
>  > Although the problem stuart brought up is that we can't have
> different classpaths in a single testsrun.
>  > As an example compatibility tests would likley not be able to be ran
> in this way.
>
> Compat tests are in different module anyway. This change is refactoring
> the integration module. The testsuite/pom.xml is affected too but only
> slightly - properties etc.
> Do/will the integration submodules need different classpath? BTW slight
> differences could be handled by tweaking deps in profiles.

Yeah so this is double maintenance, both of these have to be in sync right?

>
> Regarding moving the test sources - is there anyone seeing any advantage
> in having them all in testsuite/integration/src/test/java ? It looked
> quite ellegant solution but if there's really no one liking it, let's
> move it :)
<
>
> 3) Motivation driving the modularization
>
> In general, I'd like to:
>
> * Keep the number of modules at rational number - not fine-grained
> (difficult pom.xml maintainance), but not keeping it all-in-one at all
> costs (as it is now).
> * Keep the pom.xml's easy to read and maintain - profiles interwoven
> like a fancy bread isn't exactly something I'd like to add features to.
> * Keep the pure maven api concise and locical. Please keep in mind that
> some configs will need to be distributed across whole testsuite, and
> _every_ config "axis" (databases, IPv6, security,...) will need to be
> reflected in _every_ module. What's now few-lines pom.xml for each
> module may become a big ball of mud, pulling in things from parent
> pom.xml's.

Yeah i think referencing external files is a nice solution to this problem.


-- 
Jason T. Greene
JBoss AS Lead / EAP Platform Architect
JBoss, a division of Red Hat