Hi guys: I'd like to reopen the discussion regarding the testsuite organization and its ongoing maintenance. This issue dates back a few months with some debates and differing opinions, so I'll do my best to outline the guiding principles I'd like to see put in place concisely. To start off, I've a Proof-of-Concept for many of the following points now located: https://github.com/ALRubinger/jboss-as/tree/AS7-999 The relevant JIRA I've been using to track things: https://issues.jboss.org/browse/AS7-999 So: 1) TestSuite Organization I believe we need a single top-level categorization by which we may organize integration tests which are deployment-based and run within the context of the server. Because we use Maven modules (which are bound to a dependency structure), it makes sense to file these modules by the compile-time dependencies they require. So in place I've put: testsuite/spec - Java SE and Java EE APIs only testsuite/api - AS7 APIs + Spec testsuite/internals - Use anything in the AS7 runtime in your deployments; not guaranteed to be back-compat across releases

The primary motivation here is to ensure that the dependencies we export (ie. "spec-api", and "api" modules) are complete enough for users to create their own deployments. In this setup, we act as *users* of our own APIs, and everything in src/main is limited to the relevant dependencies. I know the source of some disagreements earlier centered around placing the tests right next to the deployments, and some folks consider the deployments as part of the test itself. That's not a bad argument at all, but again consider that we then lose the ability to validate our tests in the context of our exported APIs. 2) Run Modes, Test Subsets Because the primary organizational criteria proposed in 1) is by dependency, these modules will grow large over time. The AS build over time will take longer and longer to run. Additionally, there are runtime options to consider when starting tests. So consider the following requirements: * Running the testsuite in IPv6 * Running only a subset of tests as part of the main build

These lend themselves well to using build profiles. By default, I think the "smoke tests" should simply be a set of tests we deem important or indicative of the general health of AS7 with respect to each subsystem. As it stands now, "smoke" is its own module with a bunch of Embedded-based tests, and I think these should move to the organizational structure in 1) and instead we can apply some filtering to make the "smoke" some default set of includes. 3) An authoritative maintainer I'd like to treat the Arquillian and TestSuite modules as true subsystems of the Application Server, and as such we'll need someone to assume the responsibility to review incoming commits/pull requests and ensure they fit the criteria for acceptance. Simple things like consistent package names, using ARQ correctly, and not leaking dependencies are very important.

So assuming we come to agreement on these points, I'd like to request push access to the AS7 repo to field testsuite and ARQ-related pull requests. ...there's much more to discuss (I've more issues to raise alongside the upcoming EAP requirements), but let's start with those first 3 major points and my POC, and run from there. S, ALR _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

Hi guys: I'd like to reopen the discussion regarding the testsuite organization and its ongoing maintenance. This issue dates back a few months with some debates and differing opinions, so I'll do my best to outline the guiding principles I'd like to see put in place concisely. To start off, I've a Proof-of-Concept for many of the following points now located: https://github.com/ALRubinger/jboss-as/tree/AS7-999 The relevant JIRA I've been using to track things: https://issues.jboss.org/browse/AS7-999 So: 1) TestSuite Organization I believe we need a single top-level categorization by which we may organize integration tests which are deployment-based and run within the context of the server. Because we use Maven modules (which are bound to a dependency structure), it makes sense to file these modules by the compile-time dependencies they require. So in place I've put: testsuite/spec - Java SE and Java EE APIs only testsuite/api - AS7 APIs + Spec testsuite/internals - Use anything in the AS7 runtime in your deployments; not guaranteed to be back-compat across releases The primary motivation here is to ensure that the dependencies we export (ie. "spec-api", and "api" modules) are complete enough for users to create their own deployments. In this setup, we act as *users* of our own APIs, and everything in src/main is limited to the relevant dependencies. I know the source of some disagreements earlier centered around placing the tests right next to the deployments, and some folks consider the deployments as part of the test itself. That's not a bad argument at all, but again consider that we then lose the ability to validate our tests in the context of our exported APIs. 2) Run Modes, Test Subsets Because the primary organizational criteria proposed in 1) is by dependency, these modules will grow large over time. The AS build over time will take longer and longer to run. Additionally, there are runtime options to consider when starting tests. So consider the following requirements: * Running the testsuite in IPv6 * Running only a subset of tests as part of the main build These lend themselves well to using build profiles. By default, I think the "smoke tests" should simply be a set of tests we deem important or indicative of the general health of AS7 with respect to each subsystem. As it stands now, "smoke" is its own module with a bunch of Embedded-based tests, and I think these should move to the organizational structure in 1) and instead we can apply some filtering to make the "smoke" some default set of includes. 3) An authoritative maintainer I'd like to treat the Arquillian and TestSuite modules as true subsystems of the Application Server, and as such we'll need someone to assume the responsibility to review incoming commits/pull requests and ensure they fit the criteria for acceptance. Simple things like consistent package names, using ARQ correctly, and not leaking dependencies are very important. So assuming we come to agreement on these points, I'd like to request push access to the AS7 repo to field testsuite and ARQ-related pull requests. ...there's much more to discuss (I've more issues to raise alongside the upcoming EAP requirements), but let's start with those first 3 major points and my POC, and run from there. S, ALR _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

Inline. On 08/11/2011 05:12 PM, Richard Achmatowicz wrote: > Andrew > > Can you give a simple, concrete example of a spec test case (citing part > of the J2EE spec, say), which illustrates: Hehe, my PoC is filled with 'em. :) But keep in mind we're not directly necessarily citing the EE specs, because we're *not* building a TCK in AS7. > - what dependencies you want to be on the classpath, as well as those > you don't want to be only the classpath For this discussion, I'm primarily concerned with the *compilation* ClassPath, as this is the view users see when building their applications. It's about explicitly defining what the AS7 API is. So the deps for the compilation ClassPath should be as they are in by AS7-999 branch and as described in the last email: * spec - Java SE and Java EE * api - JBoss-specific extensions to "spec" which comprise our AS7 API * internals - Anything in the AS7 runtime

> - what sample artifacts you would put in src/main and what you would put > in src/test * src/main - Stuff that is part of the deployment. Anything that you'd be including in the @Deployment archive defined by an Arquillian-based test case. * src/test - The test infrastructure itself, like the JUnit test and other util/helpers

> - what compile and test execution time constraints you would want to enforce > This would help me to see the motivation for the refactoring and your > planned organization of artifacts in src/main and src/test, which > differ from the standard organization of putting all test related > artifacts in src/test. Test execution time is an orthogonal concern, and relates instead back to using "smoke" tests to run by default instead of the entire AS7 integration test suite (which won't scale to run on every build over time). IMO it's already taking too long for standard builds. Again, the motivation for moving/organizing in this fashion is to honor dependencies and assert that the APIs we export ("api" and "spec-api" modules) are complete. For instance, while moving tests around I discovered that our POMs were incompete: https://issues.jboss.org/browse/AS7-1489 https://issues.jboss.org/browse/AS7-1493 https://issues.jboss.org/browse/AS7-1479 https://issues.jboss.org/browse/AS7-1478 > Also, is it not possible to control what is on the classpath by maven > elements like<classpathDependencyExcludes/> and the like? That's for test runtime. Because Arquillian is starting (or connecting to) the container in a remote process, we're less concerned with the client runtime ClassPath, so long as it's enough for Arquillian to do its thing. S, ALR > > Richard > > On 08/11/2011 07:38 AM, Andrew Lee Rubinger wrote: >> Hi guys: >> >> I'd like to reopen the discussion regarding the testsuite organization >> and its ongoing maintenance. This issue dates back a few months with >> some debates and differing opinions, so I'll do my best to outline the >> guiding principles I'd like to see put in place concisely. >> >> To start off, I've a Proof-of-Concept for many of the following points >> now located: >> >> https://github.com/ALRubinger/jboss-as/tree/AS7-999 >> >> The relevant JIRA I've been using to track things: >> >> https://issues.jboss.org/browse/AS7-999 >> >> So: >> >> 1) TestSuite Organization >> >> I believe we need a single top-level categorization by which we may >> organize integration tests which are deployment-based and run within the >> context of the server. Because we use Maven modules (which are bound to >> a dependency structure), it makes sense to file these modules by the >> compile-time dependencies they require. So in place I've put: >> >> testsuite/spec - Java SE and Java EE APIs only >> testsuite/api - AS7 APIs + Spec >> testsuite/internals - Use anything in the AS7 runtime in your >> deployments; not guaranteed to be back-compat across releases >> >> The primary motivation here is to ensure that the dependencies we export >> (ie. "spec-api", and "api" modules) are complete enough for users to >> create their own deployments. In this setup, we act as *users* of our >> own APIs, and everything in src/main is limited to the relevant >> dependencies. >> >> I know the source of some disagreements earlier centered around placing >> the tests right next to the deployments, and some folks consider the >> deployments as part of the test itself. That's not a bad argument at >> all, but again consider that we then lose the ability to validate our >> tests in the context of our exported APIs. >> >> 2) Run Modes, Test Subsets >> >> Because the primary organizational criteria proposed in 1) is by >> dependency, these modules will grow large over time. The AS build over >> time will take longer and longer to run. Additionally, there are >> runtime options to consider when starting tests. So consider the >> following requirements: >> >> * Running the testsuite in IPv6 >> * Running only a subset of tests as part of the main build >> >> These lend themselves well to using build profiles. By default, I think >> the "smoke tests" should simply be a set of tests we deem important or >> indicative of the general health of AS7 with respect to each subsystem. >> As it stands now, "smoke" is its own module with a bunch of >> Embedded-based tests, and I think these should move to the >> organizational structure in 1) and instead we can apply some filtering >> to make the "smoke" some default set of includes. >> >> 3) An authoritative maintainer >> >> I'd like to treat the Arquillian and TestSuite modules as true >> subsystems of the Application Server, and as such we'll need someone to >> assume the responsibility to review incoming commits/pull requests and >> ensure they fit the criteria for acceptance. Simple things like >> consistent package names, using ARQ correctly, and not leaking >> dependencies are very important. >> >> So assuming we come to agreement on these points, I'd like to request >> push access to the AS7 repo to field testsuite and ARQ-related pull >> requests. >> >> ...there's much more to discuss (I've more issues to raise alongside the >> upcoming EAP requirements), but let's start with those first 3 major >> points and my POC, and run from there. >> >> S, >> ALR >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

On 12/08/2011, at 9:06 AM, Andrew Lee Rubinger wrote: > Inline. > > On 08/11/2011 05:12 PM, Richard Achmatowicz wrote: >> Andrew >> >> Can you give a simple, concrete example of a spec test case (citing part >> of the J2EE spec, say), which illustrates: > > Hehe, my PoC is filled with 'em. :) But keep in mind we're not directly > necessarily citing the EE specs, because we're *not* building a TCK in AS7. > >> - what dependencies you want to be on the classpath, as well as those >> you don't want to be only the classpath > > For this discussion, I'm primarily concerned with the *compilation* > ClassPath, as this is the view users see when building their > applications. It's about explicitly defining what the AS7 API is. So > the deps for the compilation ClassPath should be as they are in by > AS7-999 branch and as described in the last email: > > * spec - Java SE and Java EE > * api - JBoss-specific extensions to "spec" which comprise our AS7 API > * internals - Anything in the AS7 runtime I am ok with splitting the tests up in this manner, but I think we will need additional test suite modules for purely technical reasons. For instance, at the moment we also have compat for hibernate 3 testing, and timer service, to test the ejb timer service. Compat is needed because maven cannot handle having multiple hibernate versions in the same module, and timer service is separate because it needs the preview configuration, and need to run multiple times in order to test restoring persistent timers. > >> - what sample artifacts you would put in src/main and what you would put >> in src/test > > * src/main - Stuff that is part of the deployment. Anything that you'd > be including in the @Deployment archive defined by an Arquillian-based > test case. > * src/test - The test infrastructure itself, like the JUnit test and > other util/helpers I am very much opposed to this split. IMHO splitting the test classes into two different places makes it much harder to read and write tests. I also don't see how having additional test scoped dependencies on the class path is a problem, as in practice this works out to being Arquillian and JUnit.

I agree here, I don't like having tests split up over two source folders

Stuart > >> - what compile and test execution time constraints you would want to enforce >> This would help me to see the motivation for the refactoring and your >> planned organization of artifacts in src/main and src/test, which >> differ from the standard organization of putting all test related >> artifacts in src/test. > > Test execution time is an orthogonal concern, and relates instead back > to using "smoke" tests to run by default instead of the entire AS7 > integration test suite (which won't scale to run on every build over > time). IMO it's already taking too long for standard builds. > > Again, the motivation for moving/organizing in this fashion is to honor > dependencies and assert that the APIs we export ("api" and "spec-api" > modules) are complete. For instance, while moving tests around I > discovered that our POMs were incompete: > > https://issues.jboss.org/browse/AS7-1489 > https://issues.jboss.org/browse/AS7-1493 > https://issues.jboss.org/browse/AS7-1479 > https://issues.jboss.org/browse/AS7-1478 > >> Also, is it not possible to control what is on the classpath by maven >> elements like<classpathDependencyExcludes/> and the like? > > That's for test runtime. Because Arquillian is starting (or connecting > to) the container in a remote process, we're less concerned with the > client runtime ClassPath, so long as it's enough for Arquillian to do > its thing. > > S, > ALR > >> >> Richard >> >> On 08/11/2011 07:38 AM, Andrew Lee Rubinger wrote: >>> Hi guys: >>> >>> I'd like to reopen the discussion regarding the testsuite organization >>> and its ongoing maintenance. This issue dates back a few months with >>> some debates and differing opinions, so I'll do my best to outline the >>> guiding principles I'd like to see put in place concisely. >>> >>> To start off, I've a Proof-of-Concept for many of the following points >>> now located: >>> >>> https://github.com/ALRubinger/jboss-as/tree/AS7-999 >>> >>> The relevant JIRA I've been using to track things: >>> >>> https://issues.jboss.org/browse/AS7-999 >>> >>> So: >>> >>> 1) TestSuite Organization >>> >>> I believe we need a single top-level categorization by which we may >>> organize integration tests which are deployment-based and run within the >>> context of the server. Because we use Maven modules (which are bound to >>> a dependency structure), it makes sense to file these modules by the >>> compile-time dependencies they require. So in place I've put: >>> >>> testsuite/spec - Java SE and Java EE APIs only >>> testsuite/api - AS7 APIs + Spec >>> testsuite/internals - Use anything in the AS7 runtime in your >>> deployments; not guaranteed to be back-compat across releases >>> >>> The primary motivation here is to ensure that the dependencies we export >>> (ie. "spec-api", and "api" modules) are complete enough for users to >>> create their own deployments. In this setup, we act as *users* of our >>> own APIs, and everything in src/main is limited to the relevant >>> dependencies. >>> >>> I know the source of some disagreements earlier centered around placing >>> the tests right next to the deployments, and some folks consider the >>> deployments as part of the test itself. That's not a bad argument at >>> all, but again consider that we then lose the ability to validate our >>> tests in the context of our exported APIs. >>> >>> 2) Run Modes, Test Subsets >>> >>> Because the primary organizational criteria proposed in 1) is by >>> dependency, these modules will grow large over time. The AS build over >>> time will take longer and longer to run. Additionally, there are >>> runtime options to consider when starting tests. So consider the >>> following requirements: >>> >>> * Running the testsuite in IPv6 >>> * Running only a subset of tests as part of the main build >>> >>> These lend themselves well to using build profiles. By default, I think >>> the "smoke tests" should simply be a set of tests we deem important or >>> indicative of the general health of AS7 with respect to each subsystem. >>> As it stands now, "smoke" is its own module with a bunch of >>> Embedded-based tests, and I think these should move to the >>> organizational structure in 1) and instead we can apply some filtering >>> to make the "smoke" some default set of includes. >>> >>> 3) An authoritative maintainer >>> >>> I'd like to treat the Arquillian and TestSuite modules as true >>> subsystems of the Application Server, and as such we'll need someone to >>> assume the responsibility to review incoming commits/pull requests and >>> ensure they fit the criteria for acceptance. Simple things like >>> consistent package names, using ARQ correctly, and not leaking >>> dependencies are very important. >>> >>> So assuming we come to agreement on these points, I'd like to request >>> push access to the AS7 repo to field testsuite and ARQ-related pull >>> requests. >>> >>> ...there's much more to discuss (I've more issues to raise alongside the >>> upcoming EAP requirements), but let's start with those first 3 major >>> points and my POC, and run from there. >>> >>> S, >>> ALR >>> _______________________________________________ >>> jboss-as7-dev mailing list >>> jboss-as7-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >> >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

> On 12 Aug 2011, at 04:28, Stuart Douglas wrote: > >> On 12/08/2011, at 9:06 AM, Andrew Lee Rubinger wrote: >> >>> Inline. >>> >>> On 08/11/2011 05:12 PM, Richard Achmatowicz wrote: >>>> Andrew >>>> >>>> Can you give a simple, concrete example of a spec test case >>>> (citing part >>>> of the J2EE spec, say), which illustrates: >>> Hehe, my PoC is filled with 'em. :) But keep in mind we're not >>> directly >>> necessarily citing the EE specs, because we're *not* building a TCK >>> in AS7. >>> >>>> - what dependencies you want to be on the classpath, as well as >>>> those >>>> you don't want to be only the classpath >>> For this discussion, I'm primarily concerned with the *compilation* >>> ClassPath, as this is the view users see when building their >>> applications. It's about explicitly defining what the AS7 API is. >>> So >>> the deps for the compilation ClassPath should be as they are in by >>> AS7-999 branch and as described in the last email: >>> >>> * spec - Java SE and Java EE >>> * api - JBoss-specific extensions to "spec" which comprise our AS7 >>> API >>> * internals - Anything in the AS7 runtime >> I am ok with splitting the tests up in this manner, but I think we >> will need additional test suite modules for purely technical >> reasons. >> >> For instance, at the moment we also have compat for hibernate 3 >> testing, and timer service, to test the ejb timer service. >> >> Compat is needed because maven cannot handle having multiple >> hibernate versions in the same module, and timer service >> is separate because it needs the preview configuration, and need to >> run multiple times in order to test restoring persistent timers. Another example for new module should be tests on several AS7 instances - for some cases you would need 2, 4 and more instances, sometimes you will need to shutdown/kill AS7 instance (failover, replication tests). How would be AS7 instances managed in this proposal? It seems that everybody is interested only to run one instance of standalone server and run tests against it.

What about 'Clustering tests on multiple AS7 instances proposal' by Karel Piwko (http://lists.jboss.org/pipermail/jboss-as7-dev/2011-August/003434.html) ? Clustering tests are not mentioned in this proposal. And what about tests for management APIs? Should they be placed in api module? Management APIs are not extension to "spec". Andrew wants to use profiles to specify parameters for tests and maven plugins + to select tests which should be executed. In QA we would like to execute all tests with one command (mvn test) and I think that wouldn't be easily possible with this approach. I can imagine array of parameters, Maven is called in cycles with different parameters. This is complicated to maintain - you must cover all combinations + when new profile is added all test jobs should be reconfigured to use new configurations. And you must be sure that test results are not overwritten with new run. We need to run testsuite on RHEL/Solaris/Windows, one command to execute all tests is necessary. >>>> - what sample artifacts you would put in src/main and what you >>>> would put >>>> in src/test >>> * src/main - Stuff that is part of the deployment. Anything that >>> you'd >>> be including in the @Deployment archive defined by an >>> Arquillian-based >>> test case. >>> * src/test - The test infrastructure itself, like the JUnit test >>> and >>> other util/helpers >> I am very much opposed to this split. IMHO splitting the test >> classes into two different places makes it much harder to read and >> write tests. >> >> I also don't see how having additional test scoped dependencies on >> the class path is a problem, as in practice this works out to being >> Arquillian and JUnit. > I agree here, I don't like having tests split up over two source > folders +1 Rosta >> Stuart >> >>>> - what compile and test execution time constraints you would want >>>> to enforce >>>> This would help me to see the motivation for the refactoring and >>>> your >>>> planned organization of artifacts in src/main and src/test, which >>>> differ from the standard organization of putting all test related >>>> artifacts in src/test. >>> Test execution time is an orthogonal concern, and relates instead >>> back >>> to using "smoke" tests to run by default instead of the entire AS7 >>> integration test suite (which won't scale to run on every build >>> over >>> time). IMO it's already taking too long for standard builds. >>> >>> Again, the motivation for moving/organizing in this fashion is to >>> honor >>> dependencies and assert that the APIs we export ("api" and >>> "spec-api" >>> modules) are complete. For instance, while moving tests around I >>> discovered that our POMs were incompete: >>> >>> https://issues.jboss.org/browse/AS7-1489 >>> https://issues.jboss.org/browse/AS7-1493 >>> https://issues.jboss.org/browse/AS7-1479 >>> https://issues.jboss.org/browse/AS7-1478 >>> >>>> Also, is it not possible to control what is on the classpath by >>>> maven >>>> elements like<classpathDependencyExcludes/> and the like? >>> That's for test runtime. Because Arquillian is starting (or >>> connecting >>> to) the container in a remote process, we're less concerned with >>> the >>> client runtime ClassPath, so long as it's enough for Arquillian to >>> do >>> its thing. >>> >>> S, >>> ALR >>> >>>> Richard >>>> >>>> On 08/11/2011 07:38 AM, Andrew Lee Rubinger wrote: >>>>> Hi guys: >>>>> >>>>> I'd like to reopen the discussion regarding the testsuite >>>>> organization >>>>> and its ongoing maintenance. This issue dates back a few months >>>>> with >>>>> some debates and differing opinions, so I'll do my best to >>>>> outline the >>>>> guiding principles I'd like to see put in place concisely. >>>>> >>>>> To start off, I've a Proof-of-Concept for many of the following >>>>> points >>>>> now located: >>>>> >>>>> https://github.com/ALRubinger/jboss-as/tree/AS7-999 >>>>> >>>>> The relevant JIRA I've been using to track things: >>>>> >>>>> https://issues.jboss.org/browse/AS7-999 >>>>> >>>>> So: >>>>> >>>>> 1) TestSuite Organization >>>>> >>>>> I believe we need a single top-level categorization by which we >>>>> may >>>>> organize integration tests which are deployment-based and run >>>>> within the >>>>> context of the server. Because we use Maven modules (which are >>>>> bound to >>>>> a dependency structure), it makes sense to file these modules by >>>>> the >>>>> compile-time dependencies they require. So in place I've put: >>>>> >>>>> testsuite/spec - Java SE and Java EE APIs only >>>>> testsuite/api - AS7 APIs + Spec >>>>> testsuite/internals - Use anything in the AS7 runtime in your >>>>> deployments; not guaranteed to be back-compat across releases >>>>> >>>>> The primary motivation here is to ensure that the dependencies we >>>>> export >>>>> (ie. "spec-api", and "api" modules) are complete enough for users >>>>> to >>>>> create their own deployments. In this setup, we act as *users* of >>>>> our >>>>> own APIs, and everything in src/main is limited to the relevant >>>>> dependencies. >>>>> >>>>> I know the source of some disagreements earlier centered around >>>>> placing >>>>> the tests right next to the deployments, and some folks consider >>>>> the >>>>> deployments as part of the test itself. That's not a bad argument >>>>> at >>>>> all, but again consider that we then lose the ability to validate >>>>> our >>>>> tests in the context of our exported APIs. >>>>> >>>>> 2) Run Modes, Test Subsets >>>>> >>>>> Because the primary organizational criteria proposed in 1) is by >>>>> dependency, these modules will grow large over time. The AS build >>>>> over >>>>> time will take longer and longer to run. Additionally, there are >>>>> runtime options to consider when starting tests. So consider the >>>>> following requirements: >>>>> >>>>> * Running the testsuite in IPv6 >>>>> * Running only a subset of tests as part of the main build >>>>> >>>>> These lend themselves well to using build profiles. By default, I >>>>> think >>>>> the "smoke tests" should simply be a set of tests we deem >>>>> important or >>>>> indicative of the general health of AS7 with respect to each >>>>> subsystem. >>>>> As it stands now, "smoke" is its own module with a bunch of >>>>> Embedded-based tests, and I think these should move to the >>>>> organizational structure in 1) and instead we can apply some >>>>> filtering >>>>> to make the "smoke" some default set of includes. >>>>> >>>>> 3) An authoritative maintainer >>>>> >>>>> I'd like to treat the Arquillian and TestSuite modules as true >>>>> subsystems of the Application Server, and as such we'll need >>>>> someone to >>>>> assume the responsibility to review incoming commits/pull >>>>> requests and >>>>> ensure they fit the criteria for acceptance. Simple things like >>>>> consistent package names, using ARQ correctly, and not leaking >>>>> dependencies are very important. >>>>> >>>>> So assuming we come to agreement on these points, I'd like to >>>>> request >>>>> push access to the AS7 repo to field testsuite and ARQ-related >>>>> pull >>>>> requests. >>>>> >>>>> ...there's much more to discuss (I've more issues to raise >>>>> alongside the >>>>> upcoming EAP requirements), but let's start with those first 3 >>>>> major >>>>> points and my POC, and run from there. >>>>> >>>>> S, >>>>> ALR >>>>> _______________________________________________ >>>>> jboss-as7-dev mailing list >>>>> jboss-as7-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>> _______________________________________________ >>>> jboss-as7-dev mailing list >>>> jboss-as7-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>> _______________________________________________ >>> jboss-as7-dev mailing list >>> jboss-as7-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

I am ok with splitting the tests up in this manner, but I think we will need additional test suite modules for purely technical reasons. For instance, at the moment we also have compat for hibernate 3 testing, and timer service, to test the ejb timer service. Compat is needed because maven cannot handle having multiple hibernate versions in the same module, and timer service is separate because it needs the preview configuration, and need to run multiple times in order to test restoring persistent timers.

On 08/11/2011 05:12 PM, Richard Achmatowicz wrote: > Andrew > > Can you give a simple, concrete example of a spec test case (citing part > of the J2EE spec, say), which illustrates: Hehe, my PoC is filled with 'em. :) But keep in mind we're not directly necessarily citing the EE specs, because we're *not* building a TCK in AS7.

> - what dependencies you want to be on the classpath, as well as those > you don't want to be only the classpath For this discussion, I'm primarily concerned with the *compilation* ClassPath, as this is the view users see when building their applications. It's about explicitly defining what the AS7 API is. So the deps for the compilation ClassPath should be as they are in by AS7-999 branch and as described in the last email: * spec - Java SE and Java EE * api - JBoss-specific extensions to "spec" which comprise our AS7 API * internals - Anything in the AS7 runtime > - what sample artifacts you would put in src/main and what you would put > in src/test * src/main - Stuff that is part of the deployment. Anything that you'd be including in the @Deployment archive defined by an Arquillian-based test case. * src/test - The test infrastructure itself, like the JUnit test and other util/helpers > - what compile and test execution time constraints you would want to enforce > This would help me to see the motivation for the refactoring and your > planned organization of artifacts in src/main and src/test, which > differ from the standard organization of putting all test related > artifacts in src/test. Test execution time is an orthogonal concern, and relates instead back to using "smoke" tests to run by default instead of the entire AS7 integration test suite (which won't scale to run on every build over time). IMO it's already taking too long for standard builds. Again, the motivation for moving/organizing in this fashion is to honor dependencies and assert that the APIs we export ("api" and "spec-api" modules) are complete. For instance, while moving tests around I discovered that our POMs were incompete: https://issues.jboss.org/browse/AS7-1489 https://issues.jboss.org/browse/AS7-1493 https://issues.jboss.org/browse/AS7-1479 https://issues.jboss.org/browse/AS7-1478 > Also, is it not possible to control what is on the classpath by maven > elements like<classpathDependencyExcludes/> and the like? That's for test runtime. Because Arquillian is starting (or connecting to) the container in a remote process, we're less concerned with the client runtime ClassPath, so long as it's enough for Arquillian to do its thing. S, ALR > > Richard > > On 08/11/2011 07:38 AM, Andrew Lee Rubinger wrote: >> Hi guys: >> >> I'd like to reopen the discussion regarding the testsuite organization >> and its ongoing maintenance. This issue dates back a few months with >> some debates and differing opinions, so I'll do my best to outline the >> guiding principles I'd like to see put in place concisely. >> >> To start off, I've a Proof-of-Concept for many of the following points >> now located: >> >> https://github.com/ALRubinger/jboss-as/tree/AS7-999 >> >> The relevant JIRA I've been using to track things: >> >> https://issues.jboss.org/browse/AS7-999 >> >> So: >> >> 1) TestSuite Organization >> >> I believe we need a single top-level categorization by which we may >> organize integration tests which are deployment-based and run within the >> context of the server. Because we use Maven modules (which are bound to >> a dependency structure), it makes sense to file these modules by the >> compile-time dependencies they require. So in place I've put: >> >> testsuite/spec - Java SE and Java EE APIs only >> testsuite/api - AS7 APIs + Spec >> testsuite/internals - Use anything in the AS7 runtime in your >> deployments; not guaranteed to be back-compat across releases >> >> The primary motivation here is to ensure that the dependencies we export >> (ie. "spec-api", and "api" modules) are complete enough for users to >> create their own deployments. In this setup, we act as *users* of our >> own APIs, and everything in src/main is limited to the relevant >> dependencies. >> >> I know the source of some disagreements earlier centered around placing >> the tests right next to the deployments, and some folks consider the >> deployments as part of the test itself. That's not a bad argument at >> all, but again consider that we then lose the ability to validate our >> tests in the context of our exported APIs. >> >> 2) Run Modes, Test Subsets >> >> Because the primary organizational criteria proposed in 1) is by >> dependency, these modules will grow large over time. The AS build over >> time will take longer and longer to run. Additionally, there are >> runtime options to consider when starting tests. So consider the >> following requirements: >> >> * Running the testsuite in IPv6 >> * Running only a subset of tests as part of the main build >> >> These lend themselves well to using build profiles. By default, I think >> the "smoke tests" should simply be a set of tests we deem important or >> indicative of the general health of AS7 with respect to each subsystem. >> As it stands now, "smoke" is its own module with a bunch of >> Embedded-based tests, and I think these should move to the >> organizational structure in 1) and instead we can apply some filtering >> to make the "smoke" some default set of includes. >> >> 3) An authoritative maintainer >> >> I'd like to treat the Arquillian and TestSuite modules as true >> subsystems of the Application Server, and as such we'll need someone to >> assume the responsibility to review incoming commits/pull requests and >> ensure they fit the criteria for acceptance. Simple things like >> consistent package names, using ARQ correctly, and not leaking >> dependencies are very important. >> >> So assuming we come to agreement on these points, I'd like to request >> push access to the AS7 repo to field testsuite and ARQ-related pull >> requests. >> >> ...there's much more to discuss (I've more issues to raise alongside the >> upcoming EAP requirements), but let's start with those first 3 major >> points and my POC, and run from there. >> >> S, >> ALR >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

A summary of where I believe we stand on the 3 points raised here: 1) TestSuite Organization Stuart and Kabir have voiced concerns that separating src/main and src/test make authoring more complex.

While I'll concede that may be true to a small extent, I believe the benefits of separation far outweigh the drawbacks. By separating these out I uncovered 4 JIRAs which proved that our "api" and "spec-api" modules were incomplete. In short, paying attention to user dependencies to validate against them is very important. 2) Run Modes, Test Subsets I don't think there's been much discussion here, with the exception of the QE team who have provided some use cases that may not be possible given a standard layout. For instance the CLI and clustering tests are more than the standard "deploy something and make assertions" format we cover in point 1), so these will likely get their own modules as is appropriate. 3) An authoritative maintainer

No one has commented on this. By Tuesday evening I need to report back on at least the 3 points above, and also have approval for getting my patch committed. Failing that, we need to agree on an alternate path forward. I'm happy with any solution which addresses the respect for dependencies in the testsuite as I've outlined. S, ALR

A summary of where I believe we stand on the 3 points raised here: 1) TestSuite Organization Stuart and Kabir have voiced concerns that separating src/main and src/test make authoring more complex. While I'll concede that may be true to a small extent, I believe the benefits of separation far outweigh the drawbacks. By separating these out I uncovered 4 JIRAs which proved that our "api" and "spec-api" modules were incomplete. In short, paying attention to user dependencies to validate against them is very important.

2) Run Modes, Test Subsets I don't think there's been much discussion here, with the exception of the QE team who have provided some use cases that may not be possible given a standard layout. For instance the CLI and clustering tests are more than the standard "deploy something and make assertions" format we cover in point 1), so these will likely get their own modules as is appropriate.

3) An authoritative maintainer No one has commented on this. By Tuesday evening I need to report back on at least the 3 points above, and also have approval for getting my patch committed. Failing that, we need to agree on an alternate path forward. I'm happy with any solution which addresses the respect for dependencies in the testsuite as I've outlined. S, ALR ----- Original Message ----- From: "Andrew Lee Rubinger&quot;&lt;andrew.rubinger(a)redhat.com&gt; To: "JBoss AS7 Development&quot;&lt;jboss-as7-dev(a)lists.jboss.org&gt; Sent: Thursday, August 11, 2011 7:38:51 AM Subject: [jboss-as7-dev] Revisited: Integration TestSuite Organization and Maintenance Hi guys: I'd like to reopen the discussion regarding the testsuite organization and its ongoing maintenance. This issue dates back a few months with some debates and differing opinions, so I'll do my best to outline the guiding principles I'd like to see put in place concisely. To start off, I've a Proof-of-Concept for many of the following points now located: https://github.com/ALRubinger/jboss-as/tree/AS7-999 The relevant JIRA I've been using to track things: https://issues.jboss.org/browse/AS7-999 So: 1) TestSuite Organization I believe we need a single top-level categorization by which we may organize integration tests which are deployment-based and run within the context of the server. Because we use Maven modules (which are bound to a dependency structure), it makes sense to file these modules by the compile-time dependencies they require. So in place I've put: testsuite/spec - Java SE and Java EE APIs only testsuite/api - AS7 APIs + Spec testsuite/internals - Use anything in the AS7 runtime in your deployments; not guaranteed to be back-compat across releases The primary motivation here is to ensure that the dependencies we export (ie. "spec-api", and "api" modules) are complete enough for users to create their own deployments. In this setup, we act as *users* of our own APIs, and everything in src/main is limited to the relevant dependencies. I know the source of some disagreements earlier centered around placing the tests right next to the deployments, and some folks consider the deployments as part of the test itself. That's not a bad argument at all, but again consider that we then lose the ability to validate our tests in the context of our exported APIs. 2) Run Modes, Test Subsets Because the primary organizational criteria proposed in 1) is by dependency, these modules will grow large over time. The AS build over time will take longer and longer to run. Additionally, there are runtime options to consider when starting tests. So consider the following requirements: * Running the testsuite in IPv6 * Running only a subset of tests as part of the main build These lend themselves well to using build profiles. By default, I think the "smoke tests" should simply be a set of tests we deem important or indicative of the general health of AS7 with respect to each subsystem. As it stands now, "smoke" is its own module with a bunch of Embedded-based tests, and I think these should move to the organizational structure in 1) and instead we can apply some filtering to make the "smoke" some default set of includes. 3) An authoritative maintainer I'd like to treat the Arquillian and TestSuite modules as true subsystems of the Application Server, and as such we'll need someone to assume the responsibility to review incoming commits/pull requests and ensure they fit the criteria for acceptance. Simple things like consistent package names, using ARQ correctly, and not leaking dependencies are very important. So assuming we come to agreement on these points, I'd like to request push access to the AS7 repo to field testsuite and ARQ-related pull requests. ...there's much more to discuss (I've more issues to raise alongside the upcoming EAP requirements), but let's start with those first 3 major points and my POC, and run from there. S, ALR _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

On 15/08/2011, at 5:27 PM, Andrew Lee Rubinger wrote: > A summary of where I believe we stand on the 3 points raised here: > > 1) TestSuite Organization > > Stuart and Kabir have voiced concerns that separating src/main and > src/test make authoring more complex. > > While I'll concede that may be true to a small extent, I believe > the benefits of separation far outweigh the drawbacks. By > separating these out I uncovered 4 JIRAs which proved that our > "api" and "spec-api" modules were incomplete. In short, paying > attention to user dependencies to validate against them is very > important. > Personally I still don' t think that the additional testing of the API module makes it worth the extra inconvenience. The API does not change often, and bugs are both easy to identify and work around (e.g. if we are missing an artefact it a end user can for around it by adding the dependency to their pom, not ideal, but but not the end of the world either). On the other hand if this change does result in less tests being written, then this may result in more bugs in the application server code itself, which will be much more difficult to work around. With that said though if everyone else agrees with this I am prepared to give it a go, with the caveat that if it causes problems we go back to the current way of doing things.

On 08/16/2011 12:54 AM, Jason T. Greene wrote: > The approach might catch a missing dependency, but it will not catch an > unnecessary dependency. It seems that only human review can get the most > accurate results. Alternatively, it might be more correct to actually > write real API verification tests, that does something like look for an > expected list of packages, and erroring when an unexpected package is > found (or a missing one). I think about this all the time. Unfortunately, you can't catch a leaked API unless you test for all possible leaks. If anyone's got a fine idea how to deal with this, I'm all ears.

WRT using the testsuite as a verification mechanism to validate the completeness of the API, yes, it's a convenience mechanism. But in this case, convenience is likely to equal correctness over time (implicit maintenance). Take the case of @SecurityDomain or JavaMail, both missing from the Spec-API and API. If we didn't remember to export it, we definitely weren't going to remember to test that it was exported. :) Only when we added tests for EJB3 security or something that used javax.mail did we catch it (after I moved to a restricted dependency chain). S, ALR -- Andrew Lee Rubinger Senior Software Engineer JBoss by Red Hat Twitter: @ALRubinger http://about.me/alrubinger

In short: https://issues.jboss.org/browse/AS7-1489 https://issues.jboss.org/browse/AS7-1479 https://issues.jboss.org/browse/AS7-1478 https://issues.jboss.org/browse/AS7-1493 ...are the issues I uncovered after moving to a restricted dependency chain. With the status quo in place, issues like these go unnoticed.

On 08/16/2011 09:33 AM, Jason T. Greene wrote: > On 8/16/11 2:51 AM, Andrew Lee Rubinger wrote: >> In short: >> >> https://issues.jboss.org/browse/AS7-1489 >> https://issues.jboss.org/browse/AS7-1479 >> https://issues.jboss.org/browse/AS7-1478 >> https://issues.jboss.org/browse/AS7-1493 >> >> ...are the issues I uncovered after moving to a restricted dependency >> chain. With the status quo in place, issues like these go unnoticed. > > Yeah but no one made any attempt to verify this pom was correct. Thanks for making my point for me. :) It's about maintenance. No one is going to be verifying that these POMs are complete. Even if we make some extra suite, who is going to think to go in there and add @SecurityDomain to it? It was the testsuite which exposed that this was needed, not a manual review.

On 8/16/11 11:37 AM, Jason T. Greene wrote: > On 8/16/11 11:29 AM, Andrew Lee Rubinger wrote: >> On 08/16/2011 09:33 AM, Jason T. Greene wrote: >>> On 8/16/11 2:51 AM, Andrew Lee Rubinger wrote: >>>> In short: >>>> >>>> https://issues.jboss.org/browse/AS7-1489 >>>> https://issues.jboss.org/browse/AS7-1479 >>>> https://issues.jboss.org/browse/AS7-1478 >>>> https://issues.jboss.org/browse/AS7-1493 >>>> >>>> ...are the issues I uncovered after moving to a restricted dependency >>>> chain. With the status quo in place, issues like these go unnoticed. >>> >>> Yeah but no one made any attempt to verify this pom was correct. >> >> Thanks for making my point for me. :) It's about maintenance. No one is >> going to be verifying that these POMs are complete. >> Even if we make some extra suite, who is going to think to go in there >> and add >> @SecurityDomain to it? It was the testsuite which exposed that this was >> needed, not a manual review. > > No actually I didn't :) You do admit that the indirect process proposal > has holes (e.g. it does not catch a leaked API)? Right? Basically my point is that the only way you get this pom correct is if you have a human involved in creating the requirements/verfication list. Tools can help with that, but hacking up the AS testsuite to verify it indirectly is no substitute.

On 8/16/11 11:55 AM, Andrew Lee Rubinger wrote: > On 08/16/2011 12:40 PM, Jason T. Greene wrote: >> On 8/16/11 11:37 AM, Jason T. Greene wrote: >>> On 8/16/11 11:29 AM, Andrew Lee Rubinger wrote: >>>> On 08/16/2011 09:33 AM, Jason T. Greene wrote: >>>>> On 8/16/11 2:51 AM, Andrew Lee Rubinger wrote: >>>>>> In short: >>>>>> >>>>>> https://issues.jboss.org/browse/AS7-1489 >>>>>> https://issues.jboss.org/browse/AS7-1479 >>>>>> https://issues.jboss.org/browse/AS7-1478 >>>>>> https://issues.jboss.org/browse/AS7-1493 >>>>>> >>>>>> ...are the issues I uncovered after moving to a restricted dependency >>>>>> chain. With the status quo in place, issues like these go unnoticed. >>>>> Yeah but no one made any attempt to verify this pom was correct. >>>> Thanks for making my point for me. :) It's about maintenance. No one is >>>> going to be verifying that these POMs are complete. >>>> Even if we make some extra suite, who is going to think to go in there >>>> and add >>>> @SecurityDomain to it? It was the testsuite which exposed that this was >>>> needed, not a manual review. >>> No actually I didn't :) You do admit that the indirect process proposal >>> has holes (e.g. it does not catch a leaked API)? Right? >> Basically my point is that the only way you get this pom correct is if >> you have a human involved in creating the requirements/verfication list. >> Tools can help with that, but hacking up the AS testsuite to verify it >> indirectly is no substitute. > Let me take a different tack. > > I proposed we organize the current testsuite by dependency. If we don't > want to do that, what would we like to do? > > * Status Quo? > * Some other organizational criteria? The door is certainly open to ideas here. Probably the most useful criteria is making it easy to run targeted areas of the testsuite. > I've heard a lot of criticisms with the proposal, but not too much in > the way of alternate solutions. So the concerns I'm putting forth here > are not too important to others, or we're happy with the way things > currently are? I did propose one alternative to the api verification problem. The spec API verification is perhaps the easiest: 1. Take javaee javadocs and grab a list of all packages (perhaps even classes) 2. Write a testcase that verifies that spec-api is 100% equivalent to the above list Also the only argument I have seen is just around the split of src/test and src/main and whether the benefits are worth the cost. Ultimately I think the number 1 goal should be to make it easy to write tests so that those writing them write lots more :)

On 8/16/11 11:55 AM, Andrew Lee Rubinger wrote: > On 08/16/2011 12:40 PM, Jason T. Greene wrote: >> On 8/16/11 11:37 AM, Jason T. Greene wrote: >>> On 8/16/11 11:29 AM, Andrew Lee Rubinger wrote: >>>> On 08/16/2011 09:33 AM, Jason T. Greene wrote: >>>>> On 8/16/11 2:51 AM, Andrew Lee Rubinger wrote: >>>>>> In short: >>>>>> >>>>>> https://issues.jboss.org/browse/AS7-1489 >>>>>> https://issues.jboss.org/browse/AS7-1479 >>>>>> https://issues.jboss.org/browse/AS7-1478 >>>>>> https://issues.jboss.org/browse/AS7-1493 >>>>>> >>>>>> ...are the issues I uncovered after moving to a restricted >>>>>> dependency >>>>>> chain. With the status quo in place, issues like these go unnoticed. >>>>> >>>>> Yeah but no one made any attempt to verify this pom was correct. >>>> >>>> Thanks for making my point for me. :) It's about maintenance. No >>>> one is >>>> going to be verifying that these POMs are complete. >>>> Even if we make some extra suite, who is going to think to go in there >>>> and add >>>> @SecurityDomain to it? It was the testsuite which exposed that this >>>> was >>>> needed, not a manual review. >>> >>> No actually I didn't :) You do admit that the indirect process proposal >>> has holes (e.g. it does not catch a leaked API)? Right? >> >> Basically my point is that the only way you get this pom correct is if >> you have a human involved in creating the requirements/verfication list. >> Tools can help with that, but hacking up the AS testsuite to verify it >> indirectly is no substitute. > > Let me take a different tack. > > I proposed we organize the current testsuite by dependency. If we don't > want to do that, what would we like to do? > > * Status Quo? > * Some other organizational criteria? The door is certainly open to ideas here. Probably the most useful criteria is making it easy to run targeted areas of the testsuite.

> I've heard a lot of criticisms with the proposal, but not too much in > the way of alternate solutions. So the concerns I'm putting forth here > are not too important to others, or we're happy with the way things > currently are? I did propose one alternative to the api verification problem. The spec API verification is perhaps the easiest:

1. Take javaee javadocs and grab a list of all packages (perhaps even classes) 2. Write a testcase that verifies that spec-api is 100% equivalent to the above list Also the only argument I have seen is just around the split of src/test and src/main and whether the benefits are worth the cost. Ultimately I think the number 1 goal should be to make it easy to write tests so that those writing them write lots more :)

On 08/16/2011 12:54 AM, Jason T. Greene wrote: > The approach might catch a missing dependency, but it will not catch an > unnecessary dependency. It seems that only human review can get the most > accurate results. Alternatively, it might be more correct to actually > write real API verification tests, that does something like look for an > expected list of packages, and erroring when an unexpected package is > found (or a missing one). I think about this all the time. Unfortunately, you can't catch a leaked API unless you test for all possible leaks. If anyone's got a fine idea how to deal with this, I'm all ears.