[jboss-as7-dev] Simplifying testsuite structure

[Andrew Lee Rubinger]

I should have made explicit here, yes, just as Carlo and Jai have 
pointed out, docs are key.  If we come to some consensus on this I'll 
just type up a Wiki.

Current state of upstream has only the "smoke" submodule of "testsuite" 
enabled now, so we need to get this resolved.  Tests aren't running 
through, opening each commit to have the potential to break things. :)

I've a push to my AS7-999 branch coming today which corrects this and 
moves yet more tests away from the unified structure.

S,
ALR

On 06/14/2011 01:23 AM, Jaikiran Pai wrote:
> I personally had like the module split that was there in testsuite2,
> where api, spec, and internals were separated out. So yes, I do agree
> with Andrew, that clubbing this all into one single "integration" module
> is going to make it a dumping ground for tests. Having them separate
> looks like a good idea. Once these modules are in place, as Carlo says,
> it's just a matter of documenting explicitly which test module is
> expected to contain what kind of tests.
>
> -Jaikiran
> On Tuesday 14 June 2011 10:47 AM, Andrew Lee Rubinger wrote:
>> Hi, guys.
>>
>> So we have some disagreement towards unification of the testsuite here,
>> and I'd like to get that ironed out as soon as we can.  So anyone with
>> strong opinions, please weigh.
>>
>> Thomas has proposed collapsing "spec" and "api" into a single point:
>> "integration".  I think this is actually a step back from where we
>> should be heading:
>>
>> https://issues.jboss.org/browse/AS7-999
>> http://lists.jboss.org/pipermail/jboss-as7-dev/2011-March/000851.html
>>
>> Now, it's taken me quite some time to do the prereq work in
>> ShrinkWrap/Arquillian/AS7 ARQ fork unwinding to get to the point where
>> I'm actually gunning fulltime now on that proposal, but the basic
>> premise is this:
>>
>> * Maven modules imply a dependency structure
>> * We should write tests/demos/samples as our users would
>> * Be weary of limiting the dependencies to *only* what's required
>>
>> To this end, I've actually been working in the reverse direction from
>> the points outlined in this Thread.  Instead of moving everything into a
>> unified project to address all integration testing concerns, I've been
>> breaking them into 3 categories:
>>
>> * spec (Java EE and SE APIs only)
>> * api (spec, plus APIs we expose as public via AS - Hibernate
>> annotations, EJB3 annotations, HornetQ API, etc)
>> * internals (The whole runtime)
>>
>> Each of these represent integration tests, and the only difference is
>> the dependencies available on the Compilation Classpath.  In other
>> words, what a user will see as available in src/main/java.
>>
>> My work is being done here:
>>
>> https://github.com/ALRubinger/jboss-as/tree/AS7-999
>>
>> ...though I do need to rebase that and add my latest of today's commits
>> to it.
>>
>> Also, there's a corollary issue to expose out all of the AS7 public APIs
>> in one POM:
>>
>> https://issues.jboss.org/browse/AS7-650
>>
>> I do feel strongly that when it comes to testing, we must act as users,
>> so I suspect this above is the best approach. :)
>>
>> S,
>> ALR
>>
>> On 06/10/2011 03:38 AM, Thomas Diesler wrote:
>>> Folks,
>>>
>>> the current state is
>>>
>>> + demos
>>> + api
>>> + internals
>>> + legacy
>>> + spec
>>> + testsuite
>>> + api
>>> + benchmark
>>> + domain
>>> + integration
>>> + smoke
>>> + spec
>>> + stress
>>>
>>> some of these modules are empty. It is hard to find the "right location"
>>> for new tests. The demos are not self verifying. Instead I propose a
>>> simplified structure like this
>>>
>>> + (demos removed)
>>> + testsuite
>>> + benchmark
>>> + domain
>>> + integration
>>> + smoke
>>> + stress
>>>
>>> #1 demos removed
>>>
>>> The artefacts in the various demos modules are currently reused by
>>> testsuite modules (mainly smoke). demos are collapsed into smoke. I
>>> believe a well documented smoke testsuite can serve the purpose of demos
>>> and be a standalone deliverable. To be standalone you need to add a mvn
>>> repository entry to smoke/pom.xml. Users can download this as a binary
>>> and run 'mvn test' on it. Test cases should be organised in packages
>>> according to their functional area. It is guaranteed that the demos work
>>> because the smoke tests run on every build. Smoke tests should be well
>>> documented.
>>>
>>> #2 Collapse testsuite api+spec into integration
>>>
>>> The integration testsuite should only have dependencies on spec and api
>>> modules. Where this is not the case (i.e. a test depends on internal
>>> impl) the dependency could be flagged and an api module could be made
>>> available. Test cases should be organised in packages according to their
>>> functional area. There may be test packages that reference jiras (e.g.
>>> as835
>>> <https://github.com/tdiesler/jboss-as/tree/master/testsuite/integration/src/test/java/org/jboss/as/testsuite/integration/as835>).
>>>
>>> The main motivation is, that it is intuitively clear where to put a
>>> test. Even more importantly, where to look for already existing test
>>> coverage.
>>>
>>> cheers
>>> -thomas
>>>
>>>
>>>
>>> --
>>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx
>>> Thomas Diesler
>>> JBoss OSGi Lead
>>> JBoss, a division of Red Hat
>>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx
>>>
>>>
>>>
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> jboss-as7-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>> _______________________________________________
>> jboss-as7-dev mailing list
>> jboss-as7-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev