[jboss-as7-dev] Revisited: Integration TestSuite Organization and Maintenance

Fri Aug 12 08:41:28 EDT 2011

Was able to catch up w/ Stuart on #jboss-as7, and thought it'd make 
sense to document that conversation here; it clearly identifies the gaps 
in our approaches.

(07:49:54 AM) ALR at Freenode: stuartdouglas: At this point, we're mainly 
at a gap just on the source folder separation>
(07:49:54 AM) ALR at Freenode: ?
(07:51:13 AM) stuartdouglas: yes, I just don't think that it actually 
buys you much in terms of testing the API's, and it makes it harder to 
write and read the tests
(07:52:12 AM) stuartdouglas: I also don't think that restricting the 
tests to a the API's counts as testing the API's
(07:52:55 AM) ALR at Freenode: stuartdouglas: What else would you propose 
to address the testing that the APIs are complete?
(07:53:14 AM) ALR at Freenode: For instance in doing the separation, I 
uncovered 4 JIRAs.
(07:53:29 AM) ALR at Freenode: We would have missed these otherwise (and we 
had been missing them until now).
(07:53:35 AM) ALR at Freenode: That's proof enough for me. :)
(07:54:02 AM) ALR at Freenode: There's 2 things you need to do to fully 
test an exported API:
(07:54:11 AM) ALR at Freenode: 1) Ensure it's complete (which is what this 
does)
(07:54:17 AM) tcrawley [~tcrawley at redhat/jboss/tcrawley] entered the room.
(07:54:17 AM) mode (+v tcrawley) by ChanServ
(07:54:24 AM) ALR at Freenode: 2) Ensure it's not leaking out too much 
(which is a much tougher problem)
(07:56:45 AM) stuartdouglas: I don't think it really does either, the 
tests have not really been designed with the intention of fully 
exercising the API, even though it may expose some issues
(07:58:02 AM) ALR at Freenode: stuartdouglas: Which is why we should be 
continuing to write them as if we were users :)
(07:58:12 AM) ALR at Freenode: stuartdouglas: Though honestly I think they 
do a very good job.
(07:58:20 AM) ALR at Freenode: Arquillian is helping a lot with that.
(07:58:41 AM) ALR at Freenode: Most of them basically just define a 
deployment, maybe do an injection, and then perform their assertions.
(07:59:01 AM) stuartdouglas: Which is why I don't see why it is so 
important to restrict the class path
(07:59:02 AM) ALR at Freenode: Yours in particular are very clear and 
well-written IMO
(07:59:11 AM) ALR at Freenode: stuartdouglas: How can that be?
(07:59:24 AM) ALR at Freenode: I've shown you quite a few reasons why it's 
problematic now.
(07:59:55 AM) ALR at Freenode: The JIRAs I discovered for one.  How the ARQ 
runtime deps would leak for another.
(08:00:21 AM) ALR at Freenode: So it looks like the basis of your argument 
is that you don't refute these points, but just don't consider them 
important.
(08:00:34 AM) ALR at Freenode: And are instead placing greater importance 
over the convenience of unified source folders.
(08:02:32 AM) stuartdouglas: It's not just convenience, it makes them 
easier to read and easier to write, which is important. If people are 
disciplined when they are writing tests then the fact that there are 
dependencies available on the class path should not matter.
(08:02:45 AM) stuartdouglas: I also don't think that it really counts as 
'testing the API'
(08:03:06 AM) pilhuhn is now known as pil-afk-biab
(08:03:39 AM) stuartdouglas: if you want to test the API I think you 
should have a test/module for this testing that actually tests it properly
(08:04:22 AM) ALR at Freenode: stuartdouglas: Hehe, I think we've proven 
that the testsuite is not approached w/ discipline.
(08:04:39 AM) ALR at Freenode: It's a bit wild west, with many styles mixed 
in depending upon the author.
(08:05:26 AM) ALR at Freenode: stuartdouglas: Also we need to settle upon a 
categorization criteria.  Mine proposed is by dependency.  Unless you 
separate it out into separate source folders, you might as well not 
categorize at all.
(08:05:42 AM) ALR at Freenode: Or propose another criteria by which 
integration tests should be categorized.
(08:06:35 AM) ALR at Freenode: stuartdouglas: And the key to testing an API 
is simply coverage.
(08:06:54 AM) ALR at Freenode: By that measure, I think the integration 
testsuite as it stands is a perfect candidate.
(08:07:15 AM) ALR at Freenode: Hitting all corners of the types of apps 
that users may write.
(08:07:50 AM) ALR at Freenode: So yeah, I don't want to trade technical 
concerns for (IMO relatively) small convenience.
(08:10:28 AM) bbrowning: Is it intentional that the ProcessController is 
hard-coded to bind to 127.0.0.1 and a random port? The code at 
https://github.com/jbossas/jboss-as/blob/master/process-controller/src/main/java/org/jboss/as/process/Main.java#L210 
looks like maybe it's meant to be configurable?
(08:10:37 AM) bbrowning: lines 210 - 216
(08:16:24 AM) ALR at Freenode: stuartdouglas: I'm considering your points 
pretty thoroughly, but I just can't agree.  I've uncovered some serious 
issues with regards to our dependency management, and fixed them in my 
AS7-999 patch.  I don't see how keeping a unified source folder 
structure is worth re-exposing those risks.
(08:16:26 AM) jbossbot: jira [AS7-999] Move tests from 
testsuite/integration (legacy) to api/spec/internals [Open (Unresolved) 
Task, Major, Andrew Rubinger] https://issues.jboss.org/browse/AS7-999
(08:17:37 AM) ALR at Freenode: Sorry, I'd really like to keep everyone 
happy here.
(08:17:54 AM) ALR at Freenode: But I'll favor strict and correct over easy.
(08:18:05 AM) ALR at Freenode: Hehe, just take a look at what a PITA 
modules config is!
(08:18:27 AM) ALR at Freenode: Not easy.  But keeps isolation and keeps 
things from colliding.

S,
ALR

On 08/12/2011 08:38 AM, Andrew Lee Rubinger wrote:
> Inline again.
>
> On 08/11/2011 11:28 PM, Stuart Douglas wrote:
>> I am ok with splitting the tests up in this manner, but I think we will
>> need additional test suite modules for purely technical reasons.
>>
>> For instance, at the moment we also have compat for hibernate 3 testing,
>> and timer service, to test the ejb timer service.
>>
>> Compat is needed because maven cannot handle having multiple hibernate
>> versions in the same module, and timer service
>> is separate because it needs the preview configuration, and need to run
>> multiple times in order to test restoring persistent timers.
>
> Agreed.  At the moment I've addressed only the standard "we need to test
> a deployment" type of integration case.  There will be the need for
> other modules as you note, also for stuff like:
>
> 1) Tests which do manual control of multinode domain
> 2) CLI
>
> ...etc.  QA is feeding us some great requirements for these, and I'd
> like to discuss those once we resolve the first 3 points I've raised so
> far. :)
>
> S,
> ALR
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev