[jboss-as7-dev] Revisited: Integration TestSuite Organization and Maintenance

[Andrew Lee Rubinger]

On 08/16/2011 01:12 PM, Jason T. Greene wrote:
> On 8/16/11 11:55 AM, Andrew Lee Rubinger wrote:
>> On 08/16/2011 12:40 PM, Jason T. Greene wrote:
>>> On 8/16/11 11:37 AM, Jason T. Greene wrote:
>>>> On 8/16/11 11:29 AM, Andrew Lee Rubinger wrote:
>>>>> On 08/16/2011 09:33 AM, Jason T. Greene wrote:
>>>>>> On 8/16/11 2:51 AM, Andrew Lee Rubinger wrote:
>>>>>>> In short:
>>>>>>>
>>>>>>> https://issues.jboss.org/browse/AS7-1489
>>>>>>> https://issues.jboss.org/browse/AS7-1479
>>>>>>> https://issues.jboss.org/browse/AS7-1478
>>>>>>> https://issues.jboss.org/browse/AS7-1493
>>>>>>>
>>>>>>> ...are the issues I uncovered after moving to a restricted
>>>>>>> dependency
>>>>>>> chain. With the status quo in place, issues like these go unnoticed.
>>>>>>
>>>>>> Yeah but no one made any attempt to verify this pom was correct.
>>>>>
>>>>> Thanks for making my point for me. :) It's about maintenance. No
>>>>> one is
>>>>> going to be verifying that these POMs are complete.
>>>>> Even if we make some extra suite, who is going to think to go in there
>>>>> and add
>>>>> @SecurityDomain to it? It was the testsuite which exposed that this
>>>>> was
>>>>> needed, not a manual review.
>>>>
>>>> No actually I didn't :) You do admit that the indirect process proposal
>>>> has holes (e.g. it does not catch a leaked API)? Right?
>>>
>>> Basically my point is that the only way you get this pom correct is if
>>> you have a human involved in creating the requirements/verfication list.
>>> Tools can help with that, but hacking up the AS testsuite to verify it
>>> indirectly is no substitute.
>>
>> Let me take a different tack.
>>
>> I proposed we organize the current testsuite by dependency. If we don't
>> want to do that, what would we like to do?
>>
>> * Status Quo?
>> * Some other organizational criteria?
>
> The door is certainly open to ideas here. Probably the most useful
> criteria is making it easy to run targeted areas of the testsuite.

For that I'd previously thought profiles are one answer, though they 
certainly have drawbacks.  Alternatively I'm not against exploring the 
antrun plugin to launch JUnit and set the test includes properly.  In 
fact that may give us the best control.

So:

mvn test -Dsubset="jpa"

...and the antrun plugin, bound to "test" phase, will execute all JPA tests.

What do we think of that?

>> I've heard a lot of criticisms with the proposal, but not too much in
>> the way of alternate solutions. So the concerns I'm putting forth here
>> are not too important to others, or we're happy with the way things
>> currently are?
>
> I did propose one alternative to the api verification problem. The spec
> API verification is perhaps the easiest:

I was more referring to the testsuite org here, but OK on your spec 
verification points. :)

> 1. Take javaee javadocs and grab a list of all packages (perhaps even
> classes)
> 2. Write a testcase that verifies that spec-api is 100% equivalent to
> the above list
>
> Also the only argument I have seen is just around the split of src/test
> and src/main and whether the benefits are worth the cost. Ultimately I
> think the number 1 goal should be to make it easy to write tests so that
> those writing them write lots more :)

Honestly ARQ makes the whole thing so easy that I'm surprised this is 
even a debate. :)  Split src folder structure and all.

What we can do maybe is unify EVERY deployment-based test into ONE 
location: "testsuite/integration".  Which means moving "smoke" and some 
other submodules there too.  Collapse the source folder structure and 
dump everything into one place armed with all dependencies.  Then go 
ahead and make separate spec verification (as you propose) to satisfy 
that need.

I'm not tremendously happy with that approach as it makes for a huge dep 
chain in a singular testsuite (which just feels icky), but combined w/ 
the Spec/API verification tests meets my criteria.

Stuart/Jason - approve the above?

S,
ALR

-- 
Andrew Lee Rubinger
Senior Software Engineer
JBoss by Red Hat
Twitter: @ALRubinger
http://about.me/alrubinger