[jboss-as7-dev] Revisited: Integration TestSuite Organization and Maintenance

[Jason T. Greene]

On 8/16/11 11:55 AM, Andrew Lee Rubinger wrote:
> On 08/16/2011 12:40 PM, Jason T. Greene wrote:
>> On 8/16/11 11:37 AM, Jason T. Greene wrote:
>>> On 8/16/11 11:29 AM, Andrew Lee Rubinger wrote:
>>>> On 08/16/2011 09:33 AM, Jason T. Greene wrote:
>>>>> On 8/16/11 2:51 AM, Andrew Lee Rubinger wrote:
>>>>>> In short:
>>>>>>
>>>>>> https://issues.jboss.org/browse/AS7-1489
>>>>>> https://issues.jboss.org/browse/AS7-1479
>>>>>> https://issues.jboss.org/browse/AS7-1478
>>>>>> https://issues.jboss.org/browse/AS7-1493
>>>>>>
>>>>>> ...are the issues I uncovered after moving to a restricted dependency
>>>>>> chain. With the status quo in place, issues like these go unnoticed.
>>>>>
>>>>> Yeah but no one made any attempt to verify this pom was correct.
>>>>
>>>> Thanks for making my point for me. :) It's about maintenance. No one is
>>>> going to be verifying that these POMs are complete.
>>>> Even if we make some extra suite, who is going to think to go in there
>>>> and add
>>>> @SecurityDomain to it? It was the testsuite which exposed that this was
>>>> needed, not a manual review.
>>>
>>> No actually I didn't :) You do admit that the indirect process proposal
>>> has holes (e.g. it does not catch a leaked API)? Right?
>>
>> Basically my point is that the only way you get this pom correct is if
>> you have a human involved in creating the requirements/verfication list.
>> Tools can help with that, but hacking up the AS testsuite to verify it
>> indirectly is no substitute.
>
> Let me take a different tack.
>
> I proposed we organize the current testsuite by dependency. If we don't
> want to do that, what would we like to do?
>
> * Status Quo?
> * Some other organizational criteria?

The door is certainly open to ideas here. Probably the most useful 
criteria is making it easy to run targeted areas of the testsuite.

>
> I've heard a lot of criticisms with the proposal, but not too much in
> the way of alternate solutions. So the concerns I'm putting forth here
> are not too important to others, or we're happy with the way things
> currently are?

I did propose one alternative to the api verification problem. The spec 
API verification is perhaps the easiest:

1. Take javaee javadocs and grab a list of all packages (perhaps even 
classes)
2. Write a testcase that verifies that spec-api is 100% equivalent to 
the above list

Also the only argument I have seen is just around the split of src/test 
and src/main and whether the benefits are worth the cost. Ultimately I 
think the number 1 goal should be to make it easy to write tests so that 
those writing them write lots more :)

-- 
Jason T. Greene
JBoss AS Lead / EAP Platform Architect
JBoss, a division of Red Hat