[jboss-as7-dev] Revisited: Integration TestSuite Organization and Maintenance

[Andrew Lee Rubinger]

On 08/15/2011 08:39 AM, Scott Marlow wrote:
> On 08/15/2011 03:27 AM, Andrew Lee Rubinger wrote:
>> A summary of where I believe we stand on the 3 points raised here:
>>
>> 1) TestSuite Organization
>>
>> Stuart and Kabir have voiced concerns that separating src/main and
>> src/test make authoring more complex.
>
> I think keeping it simple/easy to create new tests, is more important
> than being able to prove that our api/spec-api modules are incomplete.
> We depend on the tests to prove that our AS build works correctly (keep
> it simple/easy so people will create the tests).

Is it really that difficult to split your tests vs. the deployables into 
two source folders?  Arquillian makes it all so easy to begin with, I'm 
honestly surprised this is such a debate.

And if we're not validating that api/spec-api is complete, what is?

Try to answer this question: What composes the AS7 API?  What artifacts 
can users depend upon in order to write software to run on our server? 
These are very important to usability, and I want to be sure they're 
addressed here.  Because the integration testsuite should by and large 
be written as if we are users.

S,
ALR

>
>>
>> While I'll concede that may be true to a small extent, I believe the
>> benefits of separation far outweigh the drawbacks. By separating these
>> out I uncovered 4 JIRAs which proved that our "api" and "spec-api"
>> modules were incomplete. In short, paying attention to user
>> dependencies to validate against them is very important.
>>
>> 2) Run Modes, Test Subsets
>>
>> I don't think there's been much discussion here, with the exception of
>> the QE team who have provided some use cases that may not be possible
>> given a standard layout. For instance the CLI and clustering tests are
>> more than the standard "deploy something and make assertions" format
>> we cover in point 1), so these will likely get their own modules as is
>> appropriate.
>>
>> 3) An authoritative maintainer
>
> If someone sets the standard, by creating many of the unit tests by
> using ARQ correctly. Others are more likely to follow their example.
> Perhaps someone could tag certain tests as good examples to follow.
>
>>
>> No one has commented on this.
>>
>> By Tuesday evening I need to report back on at least the 3 points
>> above, and also have approval for getting my patch committed. Failing
>> that, we need to agree on an alternate path forward. I'm happy with
>> any solution which addresses the respect for dependencies in the
>> testsuite as I've outlined.
>>
>> S,
>> ALR
>>

-- 
Andrew Lee Rubinger
Senior Software Engineer
JBoss by Red Hat
Twitter: @ALRubinger
http://about.me/alrubinger