[jboss-as7-dev] Revisited: Integration TestSuite Organization and Maintenance

[Richard Achmatowicz]

On 08/16/2011 01:12 PM, Jason T. Greene wrote:
> On 8/16/11 11:55 AM, Andrew Lee Rubinger wrote:
>> On 08/16/2011 12:40 PM, Jason T. Greene wrote:
>>> On 8/16/11 11:37 AM, Jason T. Greene wrote:
>>>> On 8/16/11 11:29 AM, Andrew Lee Rubinger wrote:
>>>>> On 08/16/2011 09:33 AM, Jason T. Greene wrote:
>>>>>> On 8/16/11 2:51 AM, Andrew Lee Rubinger wrote:
>>>>>>> In short:
>>>>>>>
>>>>>>> https://issues.jboss.org/browse/AS7-1489
>>>>>>> https://issues.jboss.org/browse/AS7-1479
>>>>>>> https://issues.jboss.org/browse/AS7-1478
>>>>>>> https://issues.jboss.org/browse/AS7-1493
>>>>>>>
>>>>>>> ...are the issues I uncovered after moving to a restricted dependency
>>>>>>> chain. With the status quo in place, issues like these go unnoticed.
>>>>>> Yeah but no one made any attempt to verify this pom was correct.
>>>>> Thanks for making my point for me. :) It's about maintenance. No one is
>>>>> going to be verifying that these POMs are complete.
>>>>> Even if we make some extra suite, who is going to think to go in there
>>>>> and add
>>>>> @SecurityDomain to it? It was the testsuite which exposed that this was
>>>>> needed, not a manual review.
>>>> No actually I didn't :) You do admit that the indirect process proposal
>>>> has holes (e.g. it does not catch a leaked API)? Right?
>>> Basically my point is that the only way you get this pom correct is if
>>> you have a human involved in creating the requirements/verfication list.
>>> Tools can help with that, but hacking up the AS testsuite to verify it
>>> indirectly is no substitute.
>> Let me take a different tack.
>>
>> I proposed we organize the current testsuite by dependency. If we don't
>> want to do that, what would we like to do?
>>
>> * Status Quo?
>> * Some other organizational criteria?
> The door is certainly open to ideas here. Probably the most useful
> criteria is making it easy to run targeted areas of the testsuite.
>
>> I've heard a lot of criticisms with the proposal, but not too much in
>> the way of alternate solutions. So the concerns I'm putting forth here
>> are not too important to others, or we're happy with the way things
>> currently are?
> I did propose one alternative to the api verification problem. The spec
> API verification is perhaps the easiest:
>
> 1. Take javaee javadocs and grab a list of all packages (perhaps even
> classes)
> 2. Write a testcase that verifies that spec-api is 100% equivalent to
> the above list
>
> Also the only argument I have seen is just around the split of src/test
> and src/main and whether the benefits are worth the cost. Ultimately I
> think the number 1 goal should be to make it easy to write tests so that
> those writing them write lots more :)
Jason, Andrew

This issue is generating a lot of noise for what appears to be limited 
benefit. We have been locked into a discussion for a long time now (it 
started in March) about a particular aspect of the testing of externally 
facing APIs (involving the need to control classpaths and dependencies), 
and there is now a dispute as to whether this should be used as a 
testsuite structuring mechanism.

It would be of great help if you could come up with a *proper* test plan 
for this type of testing (describing at a minimum test objectives, the 
test methodology and most importantly its scope), so that we could  
determine:
(i) whether the testing is important enough to warrant investment, given 
other priorities and
(ii) whether structuring the testsuite around dependencies is the only 
way to achieve it

The situation at the moment is that we are proposing to base the 
structure of the testsuite (which we will be using for the next 5 years) 
on issues which aren't even written down clearly and concisely, and that 
is a serious risk. The structure of the testsuite needs to be designed 
in a way that it caters to all of the requirements we have and not just 
this particular one.

If the testing is important, then it should have a test plan; if we 
can't take the time to produce a test plan and gauge its impact on the 
testsuite, then we should drop it.

My 2p.

Richard