[jboss-as7-dev] Revisited: Integration TestSuite Organization and Maintenance

[Jason T. Greene]

On 8/15/11 8:34 PM, Stuart Douglas wrote:
>
> On 15/08/2011, at 5:27 PM, Andrew Lee Rubinger wrote:
>
>> A summary of where I believe we stand on the 3 points raised here:
>>
>> 1) TestSuite Organization
>>
>> Stuart and Kabir have voiced concerns that separating src/main and
>> src/test make authoring more complex.
>>
>> While I'll concede that may be true to a small extent, I believe
>> the benefits of separation far outweigh the drawbacks.  By
>> separating these out I uncovered 4 JIRAs which proved that our
>> "api" and "spec-api" modules were incomplete.  In short, paying
>> attention to user dependencies to validate against them is very
>> important.
>>
>
> Personally I still don' t think that the additional testing of the
> API module makes it worth the extra inconvenience. The API does not
> change often, and bugs are both easy to identify and work around
> (e.g. if we are missing an artefact it a end user can for around it
> by adding the dependency to their pom, not ideal, but but not the end
> of the world either). On the other hand if this change does result in
> less tests being written, then this may result in more bugs in the
> application server code itself, which will be much more difficult to
> work around.
>
> With that said though if everyone else agrees with this I am prepared
> to give it a go, with the caveat that if it causes problems we go
> back to the current way of doing things.

I have been doing a lot of thinking about this since the discussions 
this morning, and I am wondering if this really is the "right" way to 
verify an API. What bothers me is that it is an indirect mechanism.

The approach might catch a missing dependency, but it will not catch an 
unnecessary dependency. It seems that only human review can get the most 
accurate results. Alternatively, it might be more correct to actually 
write real API verification tests, that does something like look for an 
expected list of packages, and erroring when an unexpected package is 
found (or a missing one).

-- 
Jason T. Greene
JBoss AS Lead / EAP Platform Architect
JBoss, a division of Red Hat