[jboss-as7-dev] Revisited: Integration TestSuite Organization and Maintenance

Andrew Lee Rubinger andrew.rubinger at redhat.com
Tue Aug 16 15:04:19 EDT 2011


Current state after #jboss-as7 chat w/ Jason and Richard (and earlier in 
the week w/ DML and Stuart):

1) TestSuite Organization

Deferred until requirements are complete, which won't block incoming/new 
test contributions.  Validating the specification and AS7 exported APIs 
as a separate concern may be handled outside the testsuite, allowing us 
to structure the testsuite not by dependency as I'd proposed, but by 
some other top-level criteria.

2) Run Modes, Test Subsets

To be proposed alongside the completed requirements doc, as Richard 
suggests.

3) An authoritative maintainer

David and Jason note that while push access is restricted, sign-off 
approval can come in from a designated test and/or ARQ lead.  I continue 
to volunteer to review pull requests which include commits into these 
subsystems for signoff.

S,
ALR

On 08/16/2011 01:59 PM, Andrew Lee Rubinger wrote:
>
>
> On 08/16/2011 01:12 PM, Jason T. Greene wrote:
>> On 8/16/11 11:55 AM, Andrew Lee Rubinger wrote:
>>> On 08/16/2011 12:40 PM, Jason T. Greene wrote:
>>>> On 8/16/11 11:37 AM, Jason T. Greene wrote:
>>>>> On 8/16/11 11:29 AM, Andrew Lee Rubinger wrote:
>>>>>> On 08/16/2011 09:33 AM, Jason T. Greene wrote:
>>>>>>> On 8/16/11 2:51 AM, Andrew Lee Rubinger wrote:
>>>>>>>> In short:
>>>>>>>>
>>>>>>>> https://issues.jboss.org/browse/AS7-1489
>>>>>>>> https://issues.jboss.org/browse/AS7-1479
>>>>>>>> https://issues.jboss.org/browse/AS7-1478
>>>>>>>> https://issues.jboss.org/browse/AS7-1493
>>>>>>>>
>>>>>>>> ...are the issues I uncovered after moving to a restricted
>>>>>>>> dependency
>>>>>>>> chain. With the status quo in place, issues like these go unnoticed.
>>>>>>>
>>>>>>> Yeah but no one made any attempt to verify this pom was correct.
>>>>>>
>>>>>> Thanks for making my point for me. :) It's about maintenance. No
>>>>>> one is
>>>>>> going to be verifying that these POMs are complete.
>>>>>> Even if we make some extra suite, who is going to think to go in there
>>>>>> and add
>>>>>> @SecurityDomain to it? It was the testsuite which exposed that this
>>>>>> was
>>>>>> needed, not a manual review.
>>>>>
>>>>> No actually I didn't :) You do admit that the indirect process proposal
>>>>> has holes (e.g. it does not catch a leaked API)? Right?
>>>>
>>>> Basically my point is that the only way you get this pom correct is if
>>>> you have a human involved in creating the requirements/verfication list.
>>>> Tools can help with that, but hacking up the AS testsuite to verify it
>>>> indirectly is no substitute.
>>>
>>> Let me take a different tack.
>>>
>>> I proposed we organize the current testsuite by dependency. If we don't
>>> want to do that, what would we like to do?
>>>
>>> * Status Quo?
>>> * Some other organizational criteria?
>>
>> The door is certainly open to ideas here. Probably the most useful
>> criteria is making it easy to run targeted areas of the testsuite.
>
> For that I'd previously thought profiles are one answer, though they
> certainly have drawbacks.  Alternatively I'm not against exploring the
> antrun plugin to launch JUnit and set the test includes properly.  In
> fact that may give us the best control.
>
> So:
>
> mvn test -Dsubset="jpa"
>
> ...and the antrun plugin, bound to "test" phase, will execute all JPA tests.
>
> What do we think of that?
>
>>> I've heard a lot of criticisms with the proposal, but not too much in
>>> the way of alternate solutions. So the concerns I'm putting forth here
>>> are not too important to others, or we're happy with the way things
>>> currently are?
>>
>> I did propose one alternative to the api verification problem. The spec
>> API verification is perhaps the easiest:
>
> I was more referring to the testsuite org here, but OK on your spec
> verification points. :)
>
>> 1. Take javaee javadocs and grab a list of all packages (perhaps even
>> classes)
>> 2. Write a testcase that verifies that spec-api is 100% equivalent to
>> the above list
>>
>> Also the only argument I have seen is just around the split of src/test
>> and src/main and whether the benefits are worth the cost. Ultimately I
>> think the number 1 goal should be to make it easy to write tests so that
>> those writing them write lots more :)
>
> Honestly ARQ makes the whole thing so easy that I'm surprised this is
> even a debate. :)  Split src folder structure and all.
>
> What we can do maybe is unify EVERY deployment-based test into ONE
> location: "testsuite/integration".  Which means moving "smoke" and some
> other submodules there too.  Collapse the source folder structure and
> dump everything into one place armed with all dependencies.  Then go
> ahead and make separate spec verification (as you propose) to satisfy
> that need.
>
> I'm not tremendously happy with that approach as it makes for a huge dep
> chain in a singular testsuite (which just feels icky), but combined w/
> the Spec/API verification tests meets my criteria.
>
> Stuart/Jason - approve the above?
>
> S,
> ALR
>

-- 
Andrew Lee Rubinger
Senior Software Engineer
JBoss by Red Hat
Twitter: @ALRubinger
http://about.me/alrubinger


More information about the jboss-as7-dev mailing list