Re: [jboss-as7-dev] Revisited: Integration TestSuite Organization and Maintenance

Was able to catch up w/ Stuart on #jboss-as7, and thought it'd make sense to document that conversation here; it clearly identifies the gaps in our approaches. (07:49:54 AM) ALR@Freenode: stuartdouglas: At this point, we're mainly at a gap just on the source folder separation> (07:49:54 AM) ALR@Freenode: ? (07:51:13 AM) stuartdouglas: yes, I just don't think that it actually buys you much in terms of testing the API's, and it makes it harder to write and read the tests (07:52:12 AM) stuartdouglas: I also don't think that restricting the tests to a the API's counts as testing the API's (07:52:55 AM) ALR@Freenode: stuartdouglas: What else would you propose to address the testing that the APIs are complete? (07:53:14 AM) ALR@Freenode: For instance in doing the separation, I uncovered 4 JIRAs. (07:53:29 AM) ALR@Freenode: We would have missed these otherwise (and we had been missing them until now). (07:53:35 AM) ALR@Freenode: That's proof enough for me. :) (07:54:02 AM) ALR@Freenode: There's 2 things you need to do to fully test an exported API: (07:54:11 AM) ALR@Freenode: 1) Ensure it's complete (which is what this does) (07:54:17 AM) tcrawley [~tcrawley@redhat/jboss/tcrawley] entered the room. (07:54:17 AM) mode (+v tcrawley) by ChanServ (07:54:24 AM) ALR@Freenode: 2) Ensure it's not leaking out too much (which is a much tougher problem) (07:56:45 AM) stuartdouglas: I don't think it really does either, the tests have not really been designed with the intention of fully exercising the API, even though it may expose some issues (07:58:02 AM) ALR@Freenode: stuartdouglas: Which is why we should be continuing to write them as if we were users :) (07:58:12 AM) ALR@Freenode: stuartdouglas: Though honestly I think they do a very good job. (07:58:20 AM) ALR@Freenode: Arquillian is helping a lot with that. (07:58:41 AM) ALR@Freenode: Most of them basically just define a deployment, maybe do an injection, and then perform their assertions. (07:59:01 AM) stuartdouglas: Which is why I don't see why it is so important to restrict the class path (07:59:02 AM) ALR@Freenode: Yours in particular are very clear and well-written IMO (07:59:11 AM) ALR@Freenode: stuartdouglas: How can that be? (07:59:24 AM) ALR@Freenode: I've shown you quite a few reasons why it's problematic now. (07:59:55 AM) ALR@Freenode: The JIRAs I discovered for one. How the ARQ runtime deps would leak for another. (08:00:21 AM) ALR@Freenode: So it looks like the basis of your argument is that you don't refute these points, but just don't consider them important. (08:00:34 AM) ALR@Freenode: And are instead placing greater importance over the convenience of unified source folders. (08:02:32 AM) stuartdouglas: It's not just convenience, it makes them easier to read and easier to write, which is important. If people are disciplined when they are writing tests then the fact that there are dependencies available on the class path should not matter. (08:02:45 AM) stuartdouglas: I also don't think that it really counts as 'testing the API' (08:03:06 AM) pilhuhn is now known as pil-afk-biab (08:03:39 AM) stuartdouglas: if you want to test the API I think you should have a test/module for this testing that actually tests it properly (08:04:22 AM) ALR@Freenode: stuartdouglas: Hehe, I think we've proven that the testsuite is not approached w/ discipline. (08:04:39 AM) ALR@Freenode: It's a bit wild west, with many styles mixed in depending upon the author. (08:05:26 AM) ALR@Freenode: stuartdouglas: Also we need to settle upon a categorization criteria. Mine proposed is by dependency. Unless you separate it out into separate source folders, you might as well not categorize at all. (08:05:42 AM) ALR@Freenode: Or propose another criteria by which integration tests should be categorized. (08:06:35 AM) ALR@Freenode: stuartdouglas: And the key to testing an API is simply coverage. (08:06:54 AM) ALR@Freenode: By that measure, I think the integration testsuite as it stands is a perfect candidate. (08:07:15 AM) ALR@Freenode: Hitting all corners of the types of apps that users may write. (08:07:50 AM) ALR@Freenode: So yeah, I don't want to trade technical concerns for (IMO relatively) small convenience. (08:10:28 AM) bbrowning: Is it intentional that the ProcessController is hard-coded to bind to 127.0.0.1 and a random port? The code at https://github.com/jbossas/jboss-as/blob/master/process-controller/src/ma... looks like maybe it's meant to be configurable? (08:10:37 AM) bbrowning: lines 210 - 216 (08:16:24 AM) ALR@Freenode: stuartdouglas: I'm considering your points pretty thoroughly, but I just can't agree. I've uncovered some serious issues with regards to our dependency management, and fixed them in my AS7-999 patch. I don't see how keeping a unified source folder structure is worth re-exposing those risks. (08:16:26 AM) jbossbot: jira [AS7-999] Move tests from testsuite/integration (legacy) to api/spec/internals [Open (Unresolved) Task, Major, Andrew Rubinger] https://issues.jboss.org/browse/AS7-999 (08:17:37 AM) ALR@Freenode: Sorry, I'd really like to keep everyone happy here. (08:17:54 AM) ALR@Freenode: But I'll favor strict and correct over easy. (08:18:05 AM) ALR@Freenode: Hehe, just take a look at what a PITA modules config is! (08:18:27 AM) ALR@Freenode: Not easy. But keeps isolation and keeps things from colliding. S, ALR On 08/12/2011 08:38 AM, Andrew Lee Rubinger wrote: