Why @IntegrationTest? By simply being in the AS Integration Test module, wouldn't that be self-explanatory?

On 03/20/2011 10:58 AM, Shelly McGowan wrote: > > > > Why don't we consider what was done for the CDI and BV TCKs. > Annotations[1] are used to define the test type. > If it is a spec assertion, the spec reference is clearly identified: > > @SpecAssertion(section = "2.1.1", id = "c") > > or you could annotate the Test with @IntegrationTest which could be > used to denote that AS-specific test. > > Test (not code) coverage reports can be generated to identify what > the coverage is for each component. The report > provides links to fisheye and the test code in svn. > > > [1]http://community.jboss.org/wiki/BeanValidationTCK#What_do_all_the_annot... > Other references: > http://docs.jboss.org/hibernate/stable/beanvalidation/tck/reference/html_... > http://anonsvn.jboss.org/repos/weld/cdi-tck/trunk/impl/src/main/java/org/... > > > Shelly McGowan > JBoss, by Red Hat > > ----- Original Message ----- > From: "Andrew Lee Rubinger"<andrew.rubinger(a)redhat.com> > To: jboss-as7-dev(a)lists.jboss.org > Sent: Saturday, March 19, 2011 5:34:30 PM > Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 > TestSuite > > It occurs to me I might not have been clear about the JIRA-per-test > thing. > > Say you're working on Feature X. You might commit the feature and > the > test alongside one another, citing the same JIRA, with the commit > message: > > "[JBAS-XXX] Implement Feature X and tests". > > That's great. No separate issues needed for the feature and the test > each. > > What's no good is going in and adding a bunch of tests for EJB, Web, > etc > while we pump up our coverage, alongside a commit message: > > "[JBAS-XXX] Add spec coverage to the testsuite" > > ...that tells us nothing about each test being made, just that we > dumped > 'em in there. Which is obvious, because they're there. > > S, > ALR > > On 03/19/2011 05:25 PM, Andrew Lee Rubinger wrote: >> Inline. >> >> On 03/19/2011 03:19 PM, Jason Greene wrote: >>> I really like the idea of our primary test suite covering API >>> interaction and your typical unit test which is tightly coupled to >>> internals. I also like your idea to separate spec Apis from JBoss >>> aAPIs. >>> >>> However, I disagree on the Jira per test requirement. This is only >>> because I worked with a similar model (all jiras require a test), >>> and you ended up with lots of duplicate tests, and you always had >>> to read Jira to understand what was being tested. Also, the tests >>> ended up being too specific, typically an exact replication of the >>> reporter's workflow. >> >> Consider this: If you had to read JIRA to understand what was being >> tested, that's a clear violation of point 3): Document the test. >> And >> failing good documentation, with no JIRA to turn to, you're flying >> blind. >> >> Also, how exactly does registering a JIRA encourage test >> duplication? >> Whether the tests are in an issue tracker or not seems to be >> unrelated >> to folks writing duplicate tests. >>> >>> IMO a better way to organize it would be by functional area, with >>> generous descriptions. All spec tests should reference spec >>> sections, for example. If we have a Jira that demonstrates lack of >>> coverage, we should translate the problem into meaningful and >>> reusable tests for the gaps we missed. Those tests should >>> primarily refer to the API contract and not the Jira. Although >>> commenting a list of jiras for extra info may be useful provided >>> the Jira doesn't confuse what the correct behavior should be. >> >> I'm a fan of organizing by functional areas and spec references. >> We'd >> done that in EJB3 with success: some packages were even named for >> the >> spec. Regardless, every commit had a JIRA association. >> >> In the end, so long as it's perfectly clear what we're testing and >> why, >> I'll have no objections. But I do believe that JIRA is the obvious >> answer to that question, and requires minimal effort to make a new >> issue. Especially since every commit is isolated to one concern and >> linked back to JIRA anyway, right? ;) >> >> S, >> ALR >> >>> >>> Sent from my iPad >>> >>> On Mar 18, 2011, at 2:21 AM, Andrew Lee >>> Rubinger<andrew.rubinger(a)redhat.com> wrote: >>> >>>> Looks like a lot of us have different ideas for what the AS7 >>>> Integration >>>> TestSuite should consist of, so I'll kickoff with what I believe >>>> is the >>>> first design proposal towards getting coverage focused on the >>>> end-user >>>> (not certifying our own internals). >>>> >>>> I suspect this breaks down into two categories, which may be >>>> modelled as >>>> separate modules under the existing "testsuite" aggregator >>>> parent: >>>> >>>> * Specification >>>> * AS-specific APIs >>>> >>>> This isn't difficult work, though I do think it's important we >>>> consider >>>> some hard rules. IMO we should be developing these suites as if >>>> we were >>>> application developers, not wearing our server dev hats. >>>> >>>> ---------------- >>>> >>>> [End Goal] >>>> >>>> 1) No compile-time dependencies in the module except for what's >>>> absolutely necessary. >>>> >>>> For the spec suite, this means: JDK and EE Spec APIs only in the >>>> compilation classpath. Testable asset sources and resources (ie. >>>> EJBs, >>>> Servlets, etc) would live under src/main/* to enforce that. Only >>>> the >>>> tests themselves would be located under "src/test/*". >>>> >>>> The AS-specific API suite may also add in our own APIs to the >>>> compilation classpath, but the line should end there. In "test" >>>> scope >>>> we can place all runtime dependencies. >>>> >>>> For the specification suite, AS-specific grammars like our own >>>> deployment descriptors are fine; these are in many ways >>>> equivalent to >>>> the TCK porting layer. We're not building a TCK; we're showing >>>> that our >>>> implementation supports the features advertised. >>>> >>>> 2) Every single new test created is to have an associated JIRA. >>>> >>>> We all remember the nightmare it was when the old AS4-6 suite >>>> would fall >>>> down. We'd comb through each test, at times trying to determine >>>> its >>>> purpose. By linking to JIRA we get history of intent, which acts >>>> as a >>>> nice record even in the case that the test isn't so >>>> well-documented. >>>> I'd argue that tests are a bigger asset than our code, and we >>>> should be >>>> thinking about these in terms of long-term maintenance to outlive >>>> any >>>> specific impl. >>>> >>>> 3) Documentation >>>> >>>> Alongside the JIRA reference, a quick note about we're looking to >>>> accomplish is something I find very helpful. I don't personally >>>> buy the >>>> argument that code is self-documenting if written well. It gets >>>> refactored and stale over time. >>>> >>>> 4) Run-mode profiles >>>> >>>> Arquillian provides a wonderful abstraction such that we can get >>>> coverage for AS in both remote managed *and* embedded modes >>>> without >>>> changing the test itself. To certify that everything is working >>>> as >>>> advertised no matter the runtime, we should be able to run the >>>> same >>>> suite in standalone, domain, and embedded modes (generally >>>> speaking). >>>> >>>> 5) Porting of AS6 Tests >>>> >>>> There's no discounting the value this coverage has given us, >>>> though I >>>> question the purpose of a lot of these tests. I think a great >>>> majority >>>> of these need to come into the new codebase, refactored to align >>>> if needed. >>>> >>>> ---------------- >>>> >>>> [Current State] >>>> >>>> Here[1] is an example of what I believe to be a simple, >>>> well-written >>>> test, with the exception that the tested Servlet and EJB are in >>>> the same >>>> test source folder. >>>> >>>> The current "testsuite" aggregator contains modules which mix our >>>> end-user certification stuff alongside internals, so I think >>>> these >>>> should be separated out. >>>> >>>> A lot of this is set up in some fashion already, but I would like >>>> to see us: >>>> >>>> 1) Agree upon a strict scope for each type of testsuite along the >>>> lines >>>> of my points above, once we reach agreement >>>> 2) Upgrade to ARQ 1.0.0.Alpha5 (which implies ShrinkWrap >>>> 1.0.0-alpha-12), just released tonight. Currently AS is on a >>>> forked >>>> release of ARQ for OSGi purposes, and these changes, if >>>> necessary, need >>>> to get upstream so we can do upgrades. >>>> >>>> It's clear that AS7 has made full-steam-ahead progress since last >>>> summer, and with a little organization our testsuite can give us >>>> a great >>>> view of where we stand, from an end-user's perspective, with >>>> minimal >>>> investment. >>>> >>>> S, >>>> ALR >>>> >>>> >>>> [1] >>>> https://github.com/jbossas/jboss-as/blob/master/testsuite/integration/src... >>>> _______________________________________________ >>>> jboss-as7-dev mailing list >>>> jboss-as7-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>> >>> _______________________________________________ >>> jboss-as7-dev mailing list >>> jboss-as7-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

I realize the intent of @IntegrationTest annotation for the CDI TCK. It was just an example in response to the comments about: Jason: IMO a better way to organize it would be by functional area, with generous descriptions. All spec tests should reference spec sections, for example. ALR: I'm a fan of organizing by functional areas and spec references. therefore proposing to use annotations for such organization instead of JIRA. We can devise our own annotations that make sense.

Another consideration is because they are integration tests, it is common that more than one technology is used in the test cases. Having tests organized by directory structure does not always reflect that. integration/src/test/java/org/jboss/as/testsuite/integration/jms/JMSTestCase.java These tests could contain test cases that may also cover MDB, XA, JCA. So as in [3] below, we could do something like: @Test(groups = { "JMS", "XA", "MDB"}) public void testJms() throws Exception {} or @Test(groups = { "web_profile" } ) public void testEJBLite throws Exception {}

Shelly McGowan JBoss, by Red Hat ----- Original Message ----- From: "Aslak Knutsen"<aknutsen(a)redhat.com> To: "Andrew Lee Rubinger"<andrew.rubinger(a)redhat.com> Cc: jboss-as7-dev(a)lists.jboss.org Sent: Tuesday, March 22, 2011 10:38:08 AM Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 TestSuite > > Why @IntegrationTest? By simply being in the AS Integration Test > module, wouldn't that be self-explanatory? > The use of @IntegrationTest in CDI TCK is basically to separate tests that can run in Weld SE (mocked EE v) vs the ones that need a full EE container. Not sure the AS Suite needs that kind of separation. -aslak- > On 03/20/2011 10:58 AM, Shelly McGowan wrote: >> >> >> >> Why don't we consider what was done for the CDI and BV TCKs. >> Annotations[1] are used to define the test type. >> If it is a spec assertion, the spec reference is clearly identified: >> >> @SpecAssertion(section = "2.1.1", id = "c") >> >> or you could annotate the Test with @IntegrationTest which could be >> used to denote that AS-specific test. >> >> Test (not code) coverage reports can be generated to identify what >> the coverage is for each component. The report >> provides links to fisheye and the test code in svn. >> >> >> [1]http://community.jboss.org/wiki/BeanValidationTCK#What_do_all_the_annot... >> Other references: >> [2]http://docs.jboss.org/hibernate/stable/beanvalidation/tck/reference/htm... >> [3]http://anonsvn.jboss.org/repos/weld/cdi-tck/trunk/impl/src/main/java/or... >> >> >> Shelly McGowan >> JBoss, by Red Hat >> >> ----- Original Message ----- >> From: "Andrew Lee Rubinger"<andrew.rubinger(a)redhat.com> >> To: jboss-as7-dev(a)lists.jboss.org >> Sent: Saturday, March 19, 2011 5:34:30 PM >> Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 >> TestSuite >> >> It occurs to me I might not have been clear about the JIRA-per-test >> thing. >> >> Say you're working on Feature X. You might commit the feature and >> the >> test alongside one another, citing the same JIRA, with the commit >> message: >> >> "[JBAS-XXX] Implement Feature X and tests". >> >> That's great. No separate issues needed for the feature and the test >> each. >> >> What's no good is going in and adding a bunch of tests for EJB, Web, >> etc >> while we pump up our coverage, alongside a commit message: >> >> "[JBAS-XXX] Add spec coverage to the testsuite" >> >> ...that tells us nothing about each test being made, just that we >> dumped >> 'em in there. Which is obvious, because they're there. >> >> S, >> ALR >> >> On 03/19/2011 05:25 PM, Andrew Lee Rubinger wrote: >>> Inline. >>> >>> On 03/19/2011 03:19 PM, Jason Greene wrote: >>>> I really like the idea of our primary test suite covering API >>>> interaction and your typical unit test which is tightly coupled to >>>> internals. I also like your idea to separate spec Apis from JBoss >>>> aAPIs. >>>> >>>> However, I disagree on the Jira per test requirement. This is only >>>> because I worked with a similar model (all jiras require a test), >>>> and you ended up with lots of duplicate tests, and you always had >>>> to read Jira to understand what was being tested. Also, the tests >>>> ended up being too specific, typically an exact replication of the >>>> reporter's workflow. >>> >>> Consider this: If you had to read JIRA to understand what was being >>> tested, that's a clear violation of point 3): Document the test. >>> And >>> failing good documentation, with no JIRA to turn to, you're flying >>> blind. >>> >>> Also, how exactly does registering a JIRA encourage test >>> duplication? >>> Whether the tests are in an issue tracker or not seems to be >>> unrelated >>> to folks writing duplicate tests. >>>> >>>> IMO a better way to organize it would be by functional area, with >>>> generous descriptions. All spec tests should reference spec >>>> sections, for example. If we have a Jira that demonstrates lack of >>>> coverage, we should translate the problem into meaningful and >>>> reusable tests for the gaps we missed. Those tests should >>>> primarily refer to the API contract and not the Jira. Although >>>> commenting a list of jiras for extra info may be useful provided >>>> the Jira doesn't confuse what the correct behavior should be. >>> >>> I'm a fan of organizing by functional areas and spec references. >>> We'd >>> done that in EJB3 with success: some packages were even named for >>> the >>> spec. Regardless, every commit had a JIRA association. >>> >>> In the end, so long as it's perfectly clear what we're testing and >>> why, >>> I'll have no objections. But I do believe that JIRA is the obvious >>> answer to that question, and requires minimal effort to make a new >>> issue. Especially since every commit is isolated to one concern and >>> linked back to JIRA anyway, right? ;) >>> >>> S, >>> ALR >>> >>>> >>>> Sent from my iPad >>>> >>>> On Mar 18, 2011, at 2:21 AM, Andrew Lee >>>> Rubinger<andrew.rubinger(a)redhat.com> wrote: >>>> >>>>> Looks like a lot of us have different ideas for what the AS7 >>>>> Integration >>>>> TestSuite should consist of, so I'll kickoff with what I believe >>>>> is the >>>>> first design proposal towards getting coverage focused on the >>>>> end-user >>>>> (not certifying our own internals). >>>>> >>>>> I suspect this breaks down into two categories, which may be >>>>> modelled as >>>>> separate modules under the existing "testsuite" aggregator >>>>> parent: >>>>> >>>>> * Specification >>>>> * AS-specific APIs >>>>> >>>>> This isn't difficult work, though I do think it's important we >>>>> consider >>>>> some hard rules. IMO we should be developing these suites as if >>>>> we were >>>>> application developers, not wearing our server dev hats. >>>>> >>>>> ---------------- >>>>> >>>>> [End Goal] >>>>> >>>>> 1) No compile-time dependencies in the module except for what's >>>>> absolutely necessary. >>>>> >>>>> For the spec suite, this means: JDK and EE Spec APIs only in the >>>>> compilation classpath. Testable asset sources and resources (ie. >>>>> EJBs, >>>>> Servlets, etc) would live under src/main/* to enforce that. Only >>>>> the >>>>> tests themselves would be located under "src/test/*". >>>>> >>>>> The AS-specific API suite may also add in our own APIs to the >>>>> compilation classpath, but the line should end there. In "test" >>>>> scope >>>>> we can place all runtime dependencies. >>>>> >>>>> For the specification suite, AS-specific grammars like our own >>>>> deployment descriptors are fine; these are in many ways >>>>> equivalent to >>>>> the TCK porting layer. We're not building a TCK; we're showing >>>>> that our >>>>> implementation supports the features advertised. >>>>> >>>>> 2) Every single new test created is to have an associated JIRA. >>>>> >>>>> We all remember the nightmare it was when the old AS4-6 suite >>>>> would fall >>>>> down. We'd comb through each test, at times trying to determine >>>>> its >>>>> purpose. By linking to JIRA we get history of intent, which acts >>>>> as a >>>>> nice record even in the case that the test isn't so >>>>> well-documented. >>>>> I'd argue that tests are a bigger asset than our code, and we >>>>> should be >>>>> thinking about these in terms of long-term maintenance to outlive >>>>> any >>>>> specific impl. >>>>> >>>>> 3) Documentation >>>>> >>>>> Alongside the JIRA reference, a quick note about we're looking to >>>>> accomplish is something I find very helpful. I don't personally >>>>> buy the >>>>> argument that code is self-documenting if written well. It gets >>>>> refactored and stale over time. >>>>> >>>>> 4) Run-mode profiles >>>>> >>>>> Arquillian provides a wonderful abstraction such that we can get >>>>> coverage for AS in both remote managed *and* embedded modes >>>>> without >>>>> changing the test itself. To certify that everything is working >>>>> as >>>>> advertised no matter the runtime, we should be able to run the >>>>> same >>>>> suite in standalone, domain, and embedded modes (generally >>>>> speaking). >>>>> >>>>> 5) Porting of AS6 Tests >>>>> >>>>> There's no discounting the value this coverage has given us, >>>>> though I >>>>> question the purpose of a lot of these tests. I think a great >>>>> majority >>>>> of these need to come into the new codebase, refactored to align >>>>> if needed. >>>>> >>>>> ---------------- >>>>> >>>>> [Current State] >>>>> >>>>> Here[1] is an example of what I believe to be a simple, >>>>> well-written >>>>> test, with the exception that the tested Servlet and EJB are in >>>>> the same >>>>> test source folder. >>>>> >>>>> The current "testsuite" aggregator contains modules which mix our >>>>> end-user certification stuff alongside internals, so I think >>>>> these >>>>> should be separated out. >>>>> >>>>> A lot of this is set up in some fashion already, but I would like >>>>> to see us: >>>>> >>>>> 1) Agree upon a strict scope for each type of testsuite along the >>>>> lines >>>>> of my points above, once we reach agreement >>>>> 2) Upgrade to ARQ 1.0.0.Alpha5 (which implies ShrinkWrap >>>>> 1.0.0-alpha-12), just released tonight. Currently AS is on a >>>>> forked >>>>> release of ARQ for OSGi purposes, and these changes, if >>>>> necessary, need >>>>> to get upstream so we can do upgrades. >>>>> >>>>> It's clear that AS7 has made full-steam-ahead progress since last >>>>> summer, and with a little organization our testsuite can give us >>>>> a great >>>>> view of where we stand, from an end-user's perspective, with >>>>> minimal >>>>> investment. >>>>> >>>>> S, >>>>> ALR >>>>> >>>>> >>>>> [1] >>>>> https://github.com/jbossas/jboss-as/blob/master/testsuite/integration/src... >>>>> _______________________________________________ >>>>> jboss-as7-dev mailing list >>>>> jboss-as7-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>> >>>> _______________________________________________ >>>> jboss-as7-dev mailing list >>>> jboss-as7-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>> _______________________________________________ >>> jboss-as7-dev mailing list >>> jboss-as7-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

@Test(groups = { "web_profile" } ) public void testEJBLite throws Exception {}

Shelly McGowan JBoss, by Red Hat ----- Original Message ----- From: "Aslak Knutsen"<aknutsen(a)redhat.com> To: "Andrew Lee Rubinger"<andrew.rubinger(a)redhat.com> Cc: jboss-as7-dev(a)lists.jboss.org Sent: Tuesday, March 22, 2011 10:38:08 AM Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 TestSuite > > Why @IntegrationTest? By simply being in the AS Integration Test > module, wouldn't that be self-explanatory? > The use of @IntegrationTest in CDI TCK is basically to separate tests that can run in Weld SE (mocked EE v) vs the ones that need a full EE container. Not sure the AS Suite needs that kind of separation. -aslak- > On 03/20/2011 10:58 AM, Shelly McGowan wrote: >> >> >> >> Why don't we consider what was done for the CDI and BV TCKs. >> Annotations[1] are used to define the test type. >> If it is a spec assertion, the spec reference is clearly identified: >> >> @SpecAssertion(section = "2.1.1", id = "c") >> >> or you could annotate the Test with @IntegrationTest which could be >> used to denote that AS-specific test. >> >> Test (not code) coverage reports can be generated to identify what >> the coverage is for each component. The report >> provides links to fisheye and the test code in svn. >> >> >> [1]http://community.jboss.org/wiki/BeanValidationTCK#What_do_all_the_annot... >> Other references: >> [2]http://docs.jboss.org/hibernate/stable/beanvalidation/tck/reference/htm... >> [3]http://anonsvn.jboss.org/repos/weld/cdi-tck/trunk/impl/src/main/java/or... >> >> >> Shelly McGowan >> JBoss, by Red Hat >> >> ----- Original Message ----- >> From: "Andrew Lee Rubinger"<andrew.rubinger(a)redhat.com> >> To: jboss-as7-dev(a)lists.jboss.org >> Sent: Saturday, March 19, 2011 5:34:30 PM >> Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 >> TestSuite >> >> It occurs to me I might not have been clear about the JIRA-per-test >> thing. >> >> Say you're working on Feature X. You might commit the feature and >> the >> test alongside one another, citing the same JIRA, with the commit >> message: >> >> "[JBAS-XXX] Implement Feature X and tests". >> >> That's great. No separate issues needed for the feature and the test >> each. >> >> What's no good is going in and adding a bunch of tests for EJB, Web, >> etc >> while we pump up our coverage, alongside a commit message: >> >> "[JBAS-XXX] Add spec coverage to the testsuite" >> >> ...that tells us nothing about each test being made, just that we >> dumped >> 'em in there. Which is obvious, because they're there. >> >> S, >> ALR >> >> On 03/19/2011 05:25 PM, Andrew Lee Rubinger wrote: >>> Inline. >>> >>> On 03/19/2011 03:19 PM, Jason Greene wrote: >>>> I really like the idea of our primary test suite covering API >>>> interaction and your typical unit test which is tightly coupled to >>>> internals. I also like your idea to separate spec Apis from JBoss >>>> aAPIs. >>>> >>>> However, I disagree on the Jira per test requirement. This is only >>>> because I worked with a similar model (all jiras require a test), >>>> and you ended up with lots of duplicate tests, and you always had >>>> to read Jira to understand what was being tested. Also, the tests >>>> ended up being too specific, typically an exact replication of the >>>> reporter's workflow. >>> >>> Consider this: If you had to read JIRA to understand what was being >>> tested, that's a clear violation of point 3): Document the test. >>> And >>> failing good documentation, with no JIRA to turn to, you're flying >>> blind. >>> >>> Also, how exactly does registering a JIRA encourage test >>> duplication? >>> Whether the tests are in an issue tracker or not seems to be >>> unrelated >>> to folks writing duplicate tests. >>>> >>>> IMO a better way to organize it would be by functional area, with >>>> generous descriptions. All spec tests should reference spec >>>> sections, for example. If we have a Jira that demonstrates lack of >>>> coverage, we should translate the problem into meaningful and >>>> reusable tests for the gaps we missed. Those tests should >>>> primarily refer to the API contract and not the Jira. Although >>>> commenting a list of jiras for extra info may be useful provided >>>> the Jira doesn't confuse what the correct behavior should be. >>> >>> I'm a fan of organizing by functional areas and spec references. >>> We'd >>> done that in EJB3 with success: some packages were even named for >>> the >>> spec. Regardless, every commit had a JIRA association. >>> >>> In the end, so long as it's perfectly clear what we're testing and >>> why, >>> I'll have no objections. But I do believe that JIRA is the obvious >>> answer to that question, and requires minimal effort to make a new >>> issue. Especially since every commit is isolated to one concern and >>> linked back to JIRA anyway, right? ;) >>> >>> S, >>> ALR >>> >>>> >>>> Sent from my iPad >>>> >>>> On Mar 18, 2011, at 2:21 AM, Andrew Lee >>>> Rubinger<andrew.rubinger(a)redhat.com> wrote: >>>> >>>>> Looks like a lot of us have different ideas for what the AS7 >>>>> Integration >>>>> TestSuite should consist of, so I'll kickoff with what I believe >>>>> is the >>>>> first design proposal towards getting coverage focused on the >>>>> end-user >>>>> (not certifying our own internals). >>>>> >>>>> I suspect this breaks down into two categories, which may be >>>>> modelled as >>>>> separate modules under the existing "testsuite" aggregator >>>>> parent: >>>>> >>>>> * Specification >>>>> * AS-specific APIs >>>>> >>>>> This isn't difficult work, though I do think it's important we >>>>> consider >>>>> some hard rules. IMO we should be developing these suites as if >>>>> we were >>>>> application developers, not wearing our server dev hats. >>>>> >>>>> ---------------- >>>>> >>>>> [End Goal] >>>>> >>>>> 1) No compile-time dependencies in the module except for what's >>>>> absolutely necessary. >>>>> >>>>> For the spec suite, this means: JDK and EE Spec APIs only in the >>>>> compilation classpath. Testable asset sources and resources (ie. >>>>> EJBs, >>>>> Servlets, etc) would live under src/main/* to enforce that. Only >>>>> the >>>>> tests themselves would be located under "src/test/*". >>>>> >>>>> The AS-specific API suite may also add in our own APIs to the >>>>> compilation classpath, but the line should end there. In "test" >>>>> scope >>>>> we can place all runtime dependencies. >>>>> >>>>> For the specification suite, AS-specific grammars like our own >>>>> deployment descriptors are fine; these are in many ways >>>>> equivalent to >>>>> the TCK porting layer. We're not building a TCK; we're showing >>>>> that our >>>>> implementation supports the features advertised. >>>>> >>>>> 2) Every single new test created is to have an associated JIRA. >>>>> >>>>> We all remember the nightmare it was when the old AS4-6 suite >>>>> would fall >>>>> down. We'd comb through each test, at times trying to determine >>>>> its >>>>> purpose. By linking to JIRA we get history of intent, which acts >>>>> as a >>>>> nice record even in the case that the test isn't so >>>>> well-documented. >>>>> I'd argue that tests are a bigger asset than our code, and we >>>>> should be >>>>> thinking about these in terms of long-term maintenance to outlive >>>>> any >>>>> specific impl. >>>>> >>>>> 3) Documentation >>>>> >>>>> Alongside the JIRA reference, a quick note about we're looking to >>>>> accomplish is something I find very helpful. I don't personally >>>>> buy the >>>>> argument that code is self-documenting if written well. It gets >>>>> refactored and stale over time. >>>>> >>>>> 4) Run-mode profiles >>>>> >>>>> Arquillian provides a wonderful abstraction such that we can get >>>>> coverage for AS in both remote managed *and* embedded modes >>>>> without >>>>> changing the test itself. To certify that everything is working >>>>> as >>>>> advertised no matter the runtime, we should be able to run the >>>>> same >>>>> suite in standalone, domain, and embedded modes (generally >>>>> speaking). >>>>> >>>>> 5) Porting of AS6 Tests >>>>> >>>>> There's no discounting the value this coverage has given us, >>>>> though I >>>>> question the purpose of a lot of these tests. I think a great >>>>> majority >>>>> of these need to come into the new codebase, refactored to align >>>>> if needed. >>>>> >>>>> ---------------- >>>>> >>>>> [Current State] >>>>> >>>>> Here[1] is an example of what I believe to be a simple, >>>>> well-written >>>>> test, with the exception that the tested Servlet and EJB are in >>>>> the same >>>>> test source folder. >>>>> >>>>> The current "testsuite" aggregator contains modules which mix our >>>>> end-user certification stuff alongside internals, so I think >>>>> these >>>>> should be separated out. >>>>> >>>>> A lot of this is set up in some fashion already, but I would like >>>>> to see us: >>>>> >>>>> 1) Agree upon a strict scope for each type of testsuite along the >>>>> lines >>>>> of my points above, once we reach agreement >>>>> 2) Upgrade to ARQ 1.0.0.Alpha5 (which implies ShrinkWrap >>>>> 1.0.0-alpha-12), just released tonight. Currently AS is on a >>>>> forked >>>>> release of ARQ for OSGi purposes, and these changes, if >>>>> necessary, need >>>>> to get upstream so we can do upgrades. >>>>> >>>>> It's clear that AS7 has made full-steam-ahead progress since last >>>>> summer, and with a little organization our testsuite can give us >>>>> a great >>>>> view of where we stand, from an end-user's perspective, with >>>>> minimal >>>>> investment. >>>>> >>>>> S, >>>>> ALR >>>>> >>>>> >>>>> [1] >>>>> https://github.com/jbossas/jboss-as/blob/master/testsuite/integration/src... >>>>> _______________________________________________ >>>>> jboss-as7-dev mailing list >>>>> jboss-as7-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>> >>>> _______________________________________________ >>>> jboss-as7-dev mailing list >>>> jboss-as7-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>> _______________________________________________ >>> jboss-as7-dev mailing list >>> jboss-as7-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

Andrew, To bring the thread up a level, I propose the following requirements: - no hardcoding of configuration elements in the test code - currently many tests have localhost and java:/H2DS - integration tests should run remote, not embedded - provide an Abstract test class that can be extended so adding tests focus on test logic not test configuration

- can be run in standalone or domain (and cloud) - have reporting mechanism so test coverage can be obtained based on query - ex: what tests are available for lifecycle callbacks - (as Darran said) require a test be added with a fix for each JIRA - extensible for other stakeholders (QE) of the integration test suite Shelly McGowan JBoss, by Red Hat ----- Original Message ----- From: "Shelly McGowan" <smcgowan(a)redhat.com&gt; To: "Andrew Lee Rubinger" <andrew.rubinger(a)redhat.com&gt; Cc: "Aslak Knutsen" <aknutsen(a)redhat.com&gt;, jboss-as7-dev(a)lists.jboss.org Sent: Wednesday, March 23, 2011 7:36:58 AM Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 TestSuite > > @Test(groups = { "web_profile" } ) > public void testEJBLite throws Exception {} What does @Test.groups represent? Is that a proposal for our own extension? JUnit doesn't have this property in @org.junit.Test, so if it's our own thing we should likely rename to something else... http://kentbeck.github.com/junit/javadoc/latest/org/junit/Test.html S, ALR This, as shown in [3]: import org.testng.annotations.Test; http://testng.org/javadoc/org/testng/annotations/Test.html > > Shelly McGowan > JBoss, by Red Hat > > ----- Original Message ----- > From: "Aslak Knutsen&quot;&lt;aknutsen(a)redhat.com&gt; > To: "Andrew Lee Rubinger&quot;&lt;andrew.rubinger(a)redhat.com&gt; > Cc: jboss-as7-dev(a)lists.jboss.org > Sent: Tuesday, March 22, 2011 10:38:08 AM > Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 TestSuite > >> >> Why @IntegrationTest? By simply being in the AS Integration Test >> module, wouldn't that be self-explanatory? >> > > The use of @IntegrationTest in CDI TCK is basically to separate tests that can run in Weld SE (mocked EE v) vs the ones that need a full EE container. > > Not sure the AS Suite needs that kind of separation. > > -aslak- > >> On 03/20/2011 10:58 AM, Shelly McGowan wrote: >>> >>> >>> >>> Why don't we consider what was done for the CDI and BV TCKs. >>> Annotations[1] are used to define the test type. >>> If it is a spec assertion, the spec reference is clearly identified: >>> >>> @SpecAssertion(section = "2.1.1", id = "c") >>> >>> or you could annotate the Test with @IntegrationTest which could be >>> used to denote that AS-specific test. >>> >>> Test (not code) coverage reports can be generated to identify what >>> the coverage is for each component. The report >>> provides links to fisheye and the test code in svn. >>> >>> >>> [1]http://community.jboss.org/wiki/BeanValidationTCK#What_do_all_the_annot... >>> Other references: >>> [2]http://docs.jboss.org/hibernate/stable/beanvalidation/tck/reference/htm... >>> [3]http://anonsvn.jboss.org/repos/weld/cdi-tck/trunk/impl/src/main/java/or... >>> >>> >>> Shelly McGowan >>> JBoss, by Red Hat >>> >>> ----- Original Message ----- >>> From: "Andrew Lee Rubinger&quot;&lt;andrew.rubinger(a)redhat.com&gt; >>> To: jboss-as7-dev(a)lists.jboss.org >>> Sent: Saturday, March 19, 2011 5:34:30 PM >>> Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 >>> TestSuite >>> >>> It occurs to me I might not have been clear about the JIRA-per-test >>> thing. >>> >>> Say you're working on Feature X. You might commit the feature and >>> the >>> test alongside one another, citing the same JIRA, with the commit >>> message: >>> >>> "[JBAS-XXX] Implement Feature X and tests". >>> >>> That's great. No separate issues needed for the feature and the test >>> each. >>> >>> What's no good is going in and adding a bunch of tests for EJB, Web, >>> etc >>> while we pump up our coverage, alongside a commit message: >>> >>> "[JBAS-XXX] Add spec coverage to the testsuite" >>> >>> ...that tells us nothing about each test being made, just that we >>> dumped >>> 'em in there. Which is obvious, because they're there. >>> >>> S, >>> ALR >>> >>> On 03/19/2011 05:25 PM, Andrew Lee Rubinger wrote: >>>> Inline. >>>> >>>> On 03/19/2011 03:19 PM, Jason Greene wrote: >>>>> I really like the idea of our primary test suite covering API >>>>> interaction and your typical unit test which is tightly coupled to >>>>> internals. I also like your idea to separate spec Apis from JBoss >>>>> aAPIs. >>>>> >>>>> However, I disagree on the Jira per test requirement. This is only >>>>> because I worked with a similar model (all jiras require a test), >>>>> and you ended up with lots of duplicate tests, and you always had >>>>> to read Jira to understand what was being tested. Also, the tests >>>>> ended up being too specific, typically an exact replication of the >>>>> reporter's workflow. >>>> >>>> Consider this: If you had to read JIRA to understand what was being >>>> tested, that's a clear violation of point 3): Document the test. >>>> And >>>> failing good documentation, with no JIRA to turn to, you're flying >>>> blind. >>>> >>>> Also, how exactly does registering a JIRA encourage test >>>> duplication? >>>> Whether the tests are in an issue tracker or not seems to be >>>> unrelated >>>> to folks writing duplicate tests. >>>>> >>>>> IMO a better way to organize it would be by functional area, with >>>>> generous descriptions. All spec tests should reference spec >>>>> sections, for example. If we have a Jira that demonstrates lack of >>>>> coverage, we should translate the problem into meaningful and >>>>> reusable tests for the gaps we missed. Those tests should >>>>> primarily refer to the API contract and not the Jira. Although >>>>> commenting a list of jiras for extra info may be useful provided >>>>> the Jira doesn't confuse what the correct behavior should be. >>>> >>>> I'm a fan of organizing by functional areas and spec references. >>>> We'd >>>> done that in EJB3 with success: some packages were even named for >>>> the >>>> spec. Regardless, every commit had a JIRA association. >>>> >>>> In the end, so long as it's perfectly clear what we're testing and >>>> why, >>>> I'll have no objections. But I do believe that JIRA is the obvious >>>> answer to that question, and requires minimal effort to make a new >>>> issue. Especially since every commit is isolated to one concern and >>>> linked back to JIRA anyway, right? ;) >>>> >>>> S, >>>> ALR >>>> >>>>> >>>>> Sent from my iPad >>>>> >>>>> On Mar 18, 2011, at 2:21 AM, Andrew Lee >>>>> Rubinger&lt;andrew.rubinger(a)redhat.com&gt; wrote: >>>>> >>>>>> Looks like a lot of us have different ideas for what the AS7 >>>>>> Integration >>>>>> TestSuite should consist of, so I'll kickoff with what I believe >>>>>> is the >>>>>> first design proposal towards getting coverage focused on the >>>>>> end-user >>>>>> (not certifying our own internals). >>>>>> >>>>>> I suspect this breaks down into two categories, which may be >>>>>> modelled as >>>>>> separate modules under the existing "testsuite" aggregator >>>>>> parent: >>>>>> >>>>>> * Specification >>>>>> * AS-specific APIs >>>>>> >>>>>> This isn't difficult work, though I do think it's important we >>>>>> consider >>>>>> some hard rules. IMO we should be developing these suites as if >>>>>> we were >>>>>> application developers, not wearing our server dev hats. >>>>>> >>>>>> ---------------- >>>>>> >>>>>> [End Goal] >>>>>> >>>>>> 1) No compile-time dependencies in the module except for what's >>>>>> absolutely necessary. >>>>>> >>>>>> For the spec suite, this means: JDK and EE Spec APIs only in the >>>>>> compilation classpath. Testable asset sources and resources (ie. >>>>>> EJBs, >>>>>> Servlets, etc) would live under src/main/* to enforce that. Only >>>>>> the >>>>>> tests themselves would be located under "src/test/*". >>>>>> >>>>>> The AS-specific API suite may also add in our own APIs to the >>>>>> compilation classpath, but the line should end there. In "test" >>>>>> scope >>>>>> we can place all runtime dependencies. >>>>>> >>>>>> For the specification suite, AS-specific grammars like our own >>>>>> deployment descriptors are fine; these are in many ways >>>>>> equivalent to >>>>>> the TCK porting layer. We're not building a TCK; we're showing >>>>>> that our >>>>>> implementation supports the features advertised. >>>>>> >>>>>> 2) Every single new test created is to have an associated JIRA. >>>>>> >>>>>> We all remember the nightmare it was when the old AS4-6 suite >>>>>> would fall >>>>>> down. We'd comb through each test, at times trying to determine >>>>>> its >>>>>> purpose. By linking to JIRA we get history of intent, which acts >>>>>> as a >>>>>> nice record even in the case that the test isn't so >>>>>> well-documented. >>>>>> I'd argue that tests are a bigger asset than our code, and we >>>>>> should be >>>>>> thinking about these in terms of long-term maintenance to outlive >>>>>> any >>>>>> specific impl. >>>>>> >>>>>> 3) Documentation >>>>>> >>>>>> Alongside the JIRA reference, a quick note about we're looking to >>>>>> accomplish is something I find very helpful. I don't personally >>>>>> buy the >>>>>> argument that code is self-documenting if written well. It gets >>>>>> refactored and stale over time. >>>>>> >>>>>> 4) Run-mode profiles >>>>>> >>>>>> Arquillian provides a wonderful abstraction such that we can get >>>>>> coverage for AS in both remote managed *and* embedded modes >>>>>> without >>>>>> changing the test itself. To certify that everything is working >>>>>> as >>>>>> advertised no matter the runtime, we should be able to run the >>>>>> same >>>>>> suite in standalone, domain, and embedded modes (generally >>>>>> speaking). >>>>>> >>>>>> 5) Porting of AS6 Tests >>>>>> >>>>>> There's no discounting the value this coverage has given us, >>>>>> though I >>>>>> question the purpose of a lot of these tests. I think a great >>>>>> majority >>>>>> of these need to come into the new codebase, refactored to align >>>>>> if needed. >>>>>> >>>>>> ---------------- >>>>>> >>>>>> [Current State] >>>>>> >>>>>> Here[1] is an example of what I believe to be a simple, >>>>>> well-written >>>>>> test, with the exception that the tested Servlet and EJB are in >>>>>> the same >>>>>> test source folder. >>>>>> >>>>>> The current "testsuite" aggregator contains modules which mix our >>>>>> end-user certification stuff alongside internals, so I think >>>>>> these >>>>>> should be separated out. >>>>>> >>>>>> A lot of this is set up in some fashion already, but I would like >>>>>> to see us: >>>>>> >>>>>> 1) Agree upon a strict scope for each type of testsuite along the >>>>>> lines >>>>>> of my points above, once we reach agreement >>>>>> 2) Upgrade to ARQ 1.0.0.Alpha5 (which implies ShrinkWrap >>>>>> 1.0.0-alpha-12), just released tonight. Currently AS is on a >>>>>> forked >>>>>> release of ARQ for OSGi purposes, and these changes, if >>>>>> necessary, need >>>>>> to get upstream so we can do upgrades. >>>>>> >>>>>> It's clear that AS7 has made full-steam-ahead progress since last >>>>>> summer, and with a little organization our testsuite can give us >>>>>> a great >>>>>> view of where we stand, from an end-user's perspective, with >>>>>> minimal >>>>>> investment. >>>>>> >>>>>> S, >>>>>> ALR >>>>>> >>>>>> >>>>>> [1] >>>>>> https://github.com/jbossas/jboss-as/blob/master/testsuite/integration/src... >>>>>> _______________________________________________ >>>>>> jboss-as7-dev mailing list >>>>>> jboss-as7-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>>> >>>>> _______________________________________________ >>>>> jboss-as7-dev mailing list >>>>> jboss-as7-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>> _______________________________________________ >>>> jboss-as7-dev mailing list >>>> jboss-as7-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>> _______________________________________________ >>> jboss-as7-dev mailing list >>> jboss-as7-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

> > @Test(groups = { "web_profile" } ) > public void testEJBLite throws Exception {} What does @Test.groups represent? Is that a proposal for our own extension? JUnit doesn't have this property in @org.junit.Test, so if it's our own thing we should likely rename to something else... http://kentbeck.github.com/junit/javadoc/latest/org/junit/Test.html S, ALR This, as shown in [3]: import org.testng.annotations.Test; http://testng.org/javadoc/org/testng/annotations/Test.html > > Shelly McGowan > JBoss, by Red Hat > > ----- Original Message ----- > From: "Aslak Knutsen&quot;&lt;aknutsen(a)redhat.com&gt; > To: "Andrew Lee Rubinger&quot;&lt;andrew.rubinger(a)redhat.com&gt; > Cc: jboss-as7-dev(a)lists.jboss.org > Sent: Tuesday, March 22, 2011 10:38:08 AM > Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 TestSuite > >> >> Why @IntegrationTest? By simply being in the AS Integration Test >> module, wouldn't that be self-explanatory? >> > > The use of @IntegrationTest in CDI TCK is basically to separate tests that can run in Weld SE (mocked EE v) vs the ones that need a full EE container. > > Not sure the AS Suite needs that kind of separation. > > -aslak- > >> On 03/20/2011 10:58 AM, Shelly McGowan wrote: >>> >>> >>> >>> Why don't we consider what was done for the CDI and BV TCKs. >>> Annotations[1] are used to define the test type. >>> If it is a spec assertion, the spec reference is clearly identified: >>> >>> @SpecAssertion(section = "2.1.1", id = "c") >>> >>> or you could annotate the Test with @IntegrationTest which could be >>> used to denote that AS-specific test. >>> >>> Test (not code) coverage reports can be generated to identify what >>> the coverage is for each component. The report >>> provides links to fisheye and the test code in svn. >>> >>> >>> [1]http://community.jboss.org/wiki/BeanValidationTCK#What_do_all_the_annot... >>> Other references: >>> [2]http://docs.jboss.org/hibernate/stable/beanvalidation/tck/reference/htm... >>> [3]http://anonsvn.jboss.org/repos/weld/cdi-tck/trunk/impl/src/main/java/or... >>> >>> >>> Shelly McGowan >>> JBoss, by Red Hat >>> >>> ----- Original Message ----- >>> From: "Andrew Lee Rubinger&quot;&lt;andrew.rubinger(a)redhat.com&gt; >>> To: jboss-as7-dev(a)lists.jboss.org >>> Sent: Saturday, March 19, 2011 5:34:30 PM >>> Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 >>> TestSuite >>> >>> It occurs to me I might not have been clear about the JIRA-per-test >>> thing. >>> >>> Say you're working on Feature X. You might commit the feature and >>> the >>> test alongside one another, citing the same JIRA, with the commit >>> message: >>> >>> "[JBAS-XXX] Implement Feature X and tests". >>> >>> That's great. No separate issues needed for the feature and the test >>> each. >>> >>> What's no good is going in and adding a bunch of tests for EJB, Web, >>> etc >>> while we pump up our coverage, alongside a commit message: >>> >>> "[JBAS-XXX] Add spec coverage to the testsuite" >>> >>> ...that tells us nothing about each test being made, just that we >>> dumped >>> 'em in there. Which is obvious, because they're there. >>> >>> S, >>> ALR >>> >>> On 03/19/2011 05:25 PM, Andrew Lee Rubinger wrote: >>>> Inline. >>>> >>>> On 03/19/2011 03:19 PM, Jason Greene wrote: >>>>> I really like the idea of our primary test suite covering API >>>>> interaction and your typical unit test which is tightly coupled to >>>>> internals. I also like your idea to separate spec Apis from JBoss >>>>> aAPIs. >>>>> >>>>> However, I disagree on the Jira per test requirement. This is only >>>>> because I worked with a similar model (all jiras require a test), >>>>> and you ended up with lots of duplicate tests, and you always had >>>>> to read Jira to understand what was being tested. Also, the tests >>>>> ended up being too specific, typically an exact replication of the >>>>> reporter's workflow. >>>> >>>> Consider this: If you had to read JIRA to understand what was being >>>> tested, that's a clear violation of point 3): Document the test. >>>> And >>>> failing good documentation, with no JIRA to turn to, you're flying >>>> blind. >>>> >>>> Also, how exactly does registering a JIRA encourage test >>>> duplication? >>>> Whether the tests are in an issue tracker or not seems to be >>>> unrelated >>>> to folks writing duplicate tests. >>>>> >>>>> IMO a better way to organize it would be by functional area, with >>>>> generous descriptions. All spec tests should reference spec >>>>> sections, for example. If we have a Jira that demonstrates lack of >>>>> coverage, we should translate the problem into meaningful and >>>>> reusable tests for the gaps we missed. Those tests should >>>>> primarily refer to the API contract and not the Jira. Although >>>>> commenting a list of jiras for extra info may be useful provided >>>>> the Jira doesn't confuse what the correct behavior should be. >>>> >>>> I'm a fan of organizing by functional areas and spec references. >>>> We'd >>>> done that in EJB3 with success: some packages were even named for >>>> the >>>> spec. Regardless, every commit had a JIRA association. >>>> >>>> In the end, so long as it's perfectly clear what we're testing and >>>> why, >>>> I'll have no objections. But I do believe that JIRA is the obvious >>>> answer to that question, and requires minimal effort to make a new >>>> issue. Especially since every commit is isolated to one concern and >>>> linked back to JIRA anyway, right? ;) >>>> >>>> S, >>>> ALR >>>> >>>>> >>>>> Sent from my iPad >>>>> >>>>> On Mar 18, 2011, at 2:21 AM, Andrew Lee >>>>> Rubinger&lt;andrew.rubinger(a)redhat.com&gt; wrote: >>>>> >>>>>> Looks like a lot of us have different ideas for what the AS7 >>>>>> Integration >>>>>> TestSuite should consist of, so I'll kickoff with what I believe >>>>>> is the >>>>>> first design proposal towards getting coverage focused on the >>>>>> end-user >>>>>> (not certifying our own internals). >>>>>> >>>>>> I suspect this breaks down into two categories, which may be >>>>>> modelled as >>>>>> separate modules under the existing "testsuite" aggregator >>>>>> parent: >>>>>> >>>>>> * Specification >>>>>> * AS-specific APIs >>>>>> >>>>>> This isn't difficult work, though I do think it's important we >>>>>> consider >>>>>> some hard rules. IMO we should be developing these suites as if >>>>>> we were >>>>>> application developers, not wearing our server dev hats. >>>>>> >>>>>> ---------------- >>>>>> >>>>>> [End Goal] >>>>>> >>>>>> 1) No compile-time dependencies in the module except for what's >>>>>> absolutely necessary. >>>>>> >>>>>> For the spec suite, this means: JDK and EE Spec APIs only in the >>>>>> compilation classpath. Testable asset sources and resources (ie. >>>>>> EJBs, >>>>>> Servlets, etc) would live under src/main/* to enforce that. Only >>>>>> the >>>>>> tests themselves would be located under "src/test/*". >>>>>> >>>>>> The AS-specific API suite may also add in our own APIs to the >>>>>> compilation classpath, but the line should end there. In "test" >>>>>> scope >>>>>> we can place all runtime dependencies. >>>>>> >>>>>> For the specification suite, AS-specific grammars like our own >>>>>> deployment descriptors are fine; these are in many ways >>>>>> equivalent to >>>>>> the TCK porting layer. We're not building a TCK; we're showing >>>>>> that our >>>>>> implementation supports the features advertised. >>>>>> >>>>>> 2) Every single new test created is to have an associated JIRA. >>>>>> >>>>>> We all remember the nightmare it was when the old AS4-6 suite >>>>>> would fall >>>>>> down. We'd comb through each test, at times trying to determine >>>>>> its >>>>>> purpose. By linking to JIRA we get history of intent, which acts >>>>>> as a >>>>>> nice record even in the case that the test isn't so >>>>>> well-documented. >>>>>> I'd argue that tests are a bigger asset than our code, and we >>>>>> should be >>>>>> thinking about these in terms of long-term maintenance to outlive >>>>>> any >>>>>> specific impl. >>>>>> >>>>>> 3) Documentation >>>>>> >>>>>> Alongside the JIRA reference, a quick note about we're looking to >>>>>> accomplish is something I find very helpful. I don't personally >>>>>> buy the >>>>>> argument that code is self-documenting if written well. It gets >>>>>> refactored and stale over time. >>>>>> >>>>>> 4) Run-mode profiles >>>>>> >>>>>> Arquillian provides a wonderful abstraction such that we can get >>>>>> coverage for AS in both remote managed *and* embedded modes >>>>>> without >>>>>> changing the test itself. To certify that everything is working >>>>>> as >>>>>> advertised no matter the runtime, we should be able to run the >>>>>> same >>>>>> suite in standalone, domain, and embedded modes (generally >>>>>> speaking). >>>>>> >>>>>> 5) Porting of AS6 Tests >>>>>> >>>>>> There's no discounting the value this coverage has given us, >>>>>> though I >>>>>> question the purpose of a lot of these tests. I think a great >>>>>> majority >>>>>> of these need to come into the new codebase, refactored to align >>>>>> if needed. >>>>>> >>>>>> ---------------- >>>>>> >>>>>> [Current State] >>>>>> >>>>>> Here[1] is an example of what I believe to be a simple, >>>>>> well-written >>>>>> test, with the exception that the tested Servlet and EJB are in >>>>>> the same >>>>>> test source folder. >>>>>> >>>>>> The current "testsuite" aggregator contains modules which mix our >>>>>> end-user certification stuff alongside internals, so I think >>>>>> these >>>>>> should be separated out. >>>>>> >>>>>> A lot of this is set up in some fashion already, but I would like >>>>>> to see us: >>>>>> >>>>>> 1) Agree upon a strict scope for each type of testsuite along the >>>>>> lines >>>>>> of my points above, once we reach agreement >>>>>> 2) Upgrade to ARQ 1.0.0.Alpha5 (which implies ShrinkWrap >>>>>> 1.0.0-alpha-12), just released tonight. Currently AS is on a >>>>>> forked >>>>>> release of ARQ for OSGi purposes, and these changes, if >>>>>> necessary, need >>>>>> to get upstream so we can do upgrades. >>>>>> >>>>>> It's clear that AS7 has made full-steam-ahead progress since last >>>>>> summer, and with a little organization our testsuite can give us >>>>>> a great >>>>>> view of where we stand, from an end-user's perspective, with >>>>>> minimal >>>>>> investment. >>>>>> >>>>>> S, >>>>>> ALR >>>>>> >>>>>> >>>>>> [1] >>>>>> https://github.com/jbossas/jboss-as/blob/master/testsuite/integration/src... >>>>>> _______________________________________________ >>>>>> jboss-as7-dev mailing list >>>>>> jboss-as7-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>>> >>>>> _______________________________________________ >>>>> jboss-as7-dev mailing list >>>>> jboss-as7-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>> _______________________________________________ >>>> jboss-as7-dev mailing list >>>> jboss-as7-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>> _______________________________________________ >>> jboss-as7-dev mailing list >>> jboss-as7-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

I think you should define more carefully what you are proposing, then people might agree more... * if you write a regression test, it must have a linked JIRA OR it must reference a forum post/mailing list archive (you might have written a test to prove that there isn't a problem to a user, but silly to "waste" that test...) * if you write a test for a new feature, and the feature has a JIRA issue, then you should link it. New feature development doesn't always work well with an issue tracker IMO. You also need to think about linkage. Each test should have a comment /** * Testing blah... * * JIRA: JBAS-1234 */ and all bugs should also give the FQMN of the associated tests - this way you can go from one to the other very easily. We've used this scheme a lot with Weld and the CDI TCK and I find it works very well, especially 9 months later when you come back to an issue and can't remember what I did when.

On 23 Mar 2011, at 06:10, Andrew Lee Rubinger wrote: > I'm still confused by the general hesitation towards a JIRA association > for each test. To me this is almost a redundant policy; since when are > we committing *anything* without linking back to an issue? > > Comments inline. > > On 03/22/2011 12:11 PM, Shelly McGowan wrote: >> I realize the intent of @IntegrationTest annotation for the CDI TCK. It was just an example in response to the comments about: >> >> Jason: IMO a better way to organize it would be by functional area, with generous descriptions. All spec tests should reference spec >> sections, for example. >> >> ALR: I'm a fan of organizing by functional areas and spec references. >> >> therefore proposing to use annotations for such organization instead of JIRA. We can devise our own annotations that make sense. > > OK, what tooling do you envision would make use of this metadata? Or is > it just there as a documentation marker for us? > > To me, annotations are to be consumed/processed/enforced by software; > comments are to be read by developers. > >> Another consideration is because they are integration tests, it is common that more than one >> technology is used in the test cases. Having tests organized by directory structure does not always reflect that. >> >> integration/src/test/java/org/jboss/as/testsuite/integration/jms/JMSTestCase.java >> >> These tests could contain test cases that may also cover MDB, XA, JCA. So as in [3] below, we could do something like: >> >> @Test(groups = { "JMS", "XA", "MDB"}) >> public void testJms() throws Exception {} >> >> or >> >> @Test(groups = { "web_profile" } ) >> public void testEJBLite throws Exception {} > > What does @Test.groups represent? Is that a proposal for our own > extension? JUnit doesn't have this property in @org.junit.Test, so if > it's our own thing we should likely rename to something else... > > http://kentbeck.github.com/junit/javadoc/latest/org/junit/Test.html > > S, > ALR > >> >> Shelly McGowan >> JBoss, by Red Hat >> >> ----- Original Message ----- >> From: "Aslak Knutsen&quot;&lt;aknutsen(a)redhat.com&gt; >> To: "Andrew Lee Rubinger&quot;&lt;andrew.rubinger(a)redhat.com&gt; >> Cc: jboss-as7-dev(a)lists.jboss.org >> Sent: Tuesday, March 22, 2011 10:38:08 AM >> Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 TestSuite >> >>> >>> Why @IntegrationTest? By simply being in the AS Integration Test >>> module, wouldn't that be self-explanatory? >>> >> >> The use of @IntegrationTest in CDI TCK is basically to separate tests that can run in Weld SE (mocked EE v) vs the ones that need a full EE container. >> >> Not sure the AS Suite needs that kind of separation. >> >> -aslak- >> >>> On 03/20/2011 10:58 AM, Shelly McGowan wrote: >>>> >>>> >>>> >>>> Why don't we consider what was done for the CDI and BV TCKs. >>>> Annotations[1] are used to define the test type. >>>> If it is a spec assertion, the spec reference is clearly identified: >>>> >>>> @SpecAssertion(section = "2.1.1", id = "c") >>>> >>>> or you could annotate the Test with @IntegrationTest which could be >>>> used to denote that AS-specific test. >>>> >>>> Test (not code) coverage reports can be generated to identify what >>>> the coverage is for each component. The report >>>> provides links to fisheye and the test code in svn. >>>> >>>> >>>> [1]http://community.jboss.org/wiki/BeanValidationTCK#What_do_all_the_annot... >>>> Other references: >>>> [2]http://docs.jboss.org/hibernate/stable/beanvalidation/tck/reference/htm... >>>> [3]http://anonsvn.jboss.org/repos/weld/cdi-tck/trunk/impl/src/main/java/or... >>>> >>>> >>>> Shelly McGowan >>>> JBoss, by Red Hat >>>> >>>> ----- Original Message ----- >>>> From: "Andrew Lee Rubinger&quot;&lt;andrew.rubinger(a)redhat.com&gt; >>>> To: jboss-as7-dev(a)lists.jboss.org >>>> Sent: Saturday, March 19, 2011 5:34:30 PM >>>> Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 >>>> TestSuite >>>> >>>> It occurs to me I might not have been clear about the JIRA-per-test >>>> thing. >>>> >>>> Say you're working on Feature X. You might commit the feature and >>>> the >>>> test alongside one another, citing the same JIRA, with the commit >>>> message: >>>> >>>> "[JBAS-XXX] Implement Feature X and tests". >>>> >>>> That's great. No separate issues needed for the feature and the test >>>> each. >>>> >>>> What's no good is going in and adding a bunch of tests for EJB, Web, >>>> etc >>>> while we pump up our coverage, alongside a commit message: >>>> >>>> "[JBAS-XXX] Add spec coverage to the testsuite" >>>> >>>> ...that tells us nothing about each test being made, just that we >>>> dumped >>>> 'em in there. Which is obvious, because they're there. >>>> >>>> S, >>>> ALR >>>> >>>> On 03/19/2011 05:25 PM, Andrew Lee Rubinger wrote: >>>>> Inline. >>>>> >>>>> On 03/19/2011 03:19 PM, Jason Greene wrote: >>>>>> I really like the idea of our primary test suite covering API >>>>>> interaction and your typical unit test which is tightly coupled to >>>>>> internals. I also like your idea to separate spec Apis from JBoss >>>>>> aAPIs. >>>>>> >>>>>> However, I disagree on the Jira per test requirement. This is only >>>>>> because I worked with a similar model (all jiras require a test), >>>>>> and you ended up with lots of duplicate tests, and you always had >>>>>> to read Jira to understand what was being tested. Also, the tests >>>>>> ended up being too specific, typically an exact replication of the >>>>>> reporter's workflow. >>>>> >>>>> Consider this: If you had to read JIRA to understand what was being >>>>> tested, that's a clear violation of point 3): Document the test. >>>>> And >>>>> failing good documentation, with no JIRA to turn to, you're flying >>>>> blind. >>>>> >>>>> Also, how exactly does registering a JIRA encourage test >>>>> duplication? >>>>> Whether the tests are in an issue tracker or not seems to be >>>>> unrelated >>>>> to folks writing duplicate tests. >>>>>> >>>>>> IMO a better way to organize it would be by functional area, with >>>>>> generous descriptions. All spec tests should reference spec >>>>>> sections, for example. If we have a Jira that demonstrates lack of >>>>>> coverage, we should translate the problem into meaningful and >>>>>> reusable tests for the gaps we missed. Those tests should >>>>>> primarily refer to the API contract and not the Jira. Although >>>>>> commenting a list of jiras for extra info may be useful provided >>>>>> the Jira doesn't confuse what the correct behavior should be. >>>>> >>>>> I'm a fan of organizing by functional areas and spec references. >>>>> We'd >>>>> done that in EJB3 with success: some packages were even named for >>>>> the >>>>> spec. Regardless, every commit had a JIRA association. >>>>> >>>>> In the end, so long as it's perfectly clear what we're testing and >>>>> why, >>>>> I'll have no objections. But I do believe that JIRA is the obvious >>>>> answer to that question, and requires minimal effort to make a new >>>>> issue. Especially since every commit is isolated to one concern and >>>>> linked back to JIRA anyway, right? ;) >>>>> >>>>> S, >>>>> ALR >>>>> >>>>>> >>>>>> Sent from my iPad >>>>>> >>>>>> On Mar 18, 2011, at 2:21 AM, Andrew Lee >>>>>> Rubinger&lt;andrew.rubinger(a)redhat.com&gt; wrote: >>>>>> >>>>>>> Looks like a lot of us have different ideas for what the AS7 >>>>>>> Integration >>>>>>> TestSuite should consist of, so I'll kickoff with what I believe >>>>>>> is the >>>>>>> first design proposal towards getting coverage focused on the >>>>>>> end-user >>>>>>> (not certifying our own internals). >>>>>>> >>>>>>> I suspect this breaks down into two categories, which may be >>>>>>> modelled as >>>>>>> separate modules under the existing "testsuite" aggregator >>>>>>> parent: >>>>>>> >>>>>>> * Specification >>>>>>> * AS-specific APIs >>>>>>> >>>>>>> This isn't difficult work, though I do think it's important we >>>>>>> consider >>>>>>> some hard rules. IMO we should be developing these suites as if >>>>>>> we were >>>>>>> application developers, not wearing our server dev hats. >>>>>>> >>>>>>> ---------------- >>>>>>> >>>>>>> [End Goal] >>>>>>> >>>>>>> 1) No compile-time dependencies in the module except for what's >>>>>>> absolutely necessary. >>>>>>> >>>>>>> For the spec suite, this means: JDK and EE Spec APIs only in the >>>>>>> compilation classpath. Testable asset sources and resources (ie. >>>>>>> EJBs, >>>>>>> Servlets, etc) would live under src/main/* to enforce that. Only >>>>>>> the >>>>>>> tests themselves would be located under "src/test/*". >>>>>>> >>>>>>> The AS-specific API suite may also add in our own APIs to the >>>>>>> compilation classpath, but the line should end there. In "test" >>>>>>> scope >>>>>>> we can place all runtime dependencies. >>>>>>> >>>>>>> For the specification suite, AS-specific grammars like our own >>>>>>> deployment descriptors are fine; these are in many ways >>>>>>> equivalent to >>>>>>> the TCK porting layer. We're not building a TCK; we're showing >>>>>>> that our >>>>>>> implementation supports the features advertised. >>>>>>> >>>>>>> 2) Every single new test created is to have an associated JIRA. >>>>>>> >>>>>>> We all remember the nightmare it was when the old AS4-6 suite >>>>>>> would fall >>>>>>> down. We'd comb through each test, at times trying to determine >>>>>>> its >>>>>>> purpose. By linking to JIRA we get history of intent, which acts >>>>>>> as a >>>>>>> nice record even in the case that the test isn't so >>>>>>> well-documented. >>>>>>> I'd argue that tests are a bigger asset than our code, and we >>>>>>> should be >>>>>>> thinking about these in terms of long-term maintenance to outlive >>>>>>> any >>>>>>> specific impl. >>>>>>> >>>>>>> 3) Documentation >>>>>>> >>>>>>> Alongside the JIRA reference, a quick note about we're looking to >>>>>>> accomplish is something I find very helpful. I don't personally >>>>>>> buy the >>>>>>> argument that code is self-documenting if written well. It gets >>>>>>> refactored and stale over time. >>>>>>> >>>>>>> 4) Run-mode profiles >>>>>>> >>>>>>> Arquillian provides a wonderful abstraction such that we can get >>>>>>> coverage for AS in both remote managed *and* embedded modes >>>>>>> without >>>>>>> changing the test itself. To certify that everything is working >>>>>>> as >>>>>>> advertised no matter the runtime, we should be able to run the >>>>>>> same >>>>>>> suite in standalone, domain, and embedded modes (generally >>>>>>> speaking). >>>>>>> >>>>>>> 5) Porting of AS6 Tests >>>>>>> >>>>>>> There's no discounting the value this coverage has given us, >>>>>>> though I >>>>>>> question the purpose of a lot of these tests. I think a great >>>>>>> majority >>>>>>> of these need to come into the new codebase, refactored to align >>>>>>> if needed. >>>>>>> >>>>>>> ---------------- >>>>>>> >>>>>>> [Current State] >>>>>>> >>>>>>> Here[1] is an example of what I believe to be a simple, >>>>>>> well-written >>>>>>> test, with the exception that the tested Servlet and EJB are in >>>>>>> the same >>>>>>> test source folder. >>>>>>> >>>>>>> The current "testsuite" aggregator contains modules which mix our >>>>>>> end-user certification stuff alongside internals, so I think >>>>>>> these >>>>>>> should be separated out. >>>>>>> >>>>>>> A lot of this is set up in some fashion already, but I would like >>>>>>> to see us: >>>>>>> >>>>>>> 1) Agree upon a strict scope for each type of testsuite along the >>>>>>> lines >>>>>>> of my points above, once we reach agreement >>>>>>> 2) Upgrade to ARQ 1.0.0.Alpha5 (which implies ShrinkWrap >>>>>>> 1.0.0-alpha-12), just released tonight. Currently AS is on a >>>>>>> forked >>>>>>> release of ARQ for OSGi purposes, and these changes, if >>>>>>> necessary, need >>>>>>> to get upstream so we can do upgrades. >>>>>>> >>>>>>> It's clear that AS7 has made full-steam-ahead progress since last >>>>>>> summer, and with a little organization our testsuite can give us >>>>>>> a great >>>>>>> view of where we stand, from an end-user's perspective, with >>>>>>> minimal >>>>>>> investment. >>>>>>> >>>>>>> S, >>>>>>> ALR >>>>>>> >>>>>>> >>>>>>> [1] >>>>>>> https://github.com/jbossas/jboss-as/blob/master/testsuite/integration/src... >>>>>>> _______________________________________________ >>>>>>> jboss-as7-dev mailing list >>>>>>> jboss-as7-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>>>> >>>>>> _______________________________________________ >>>>>> jboss-as7-dev mailing list >>>>>> jboss-as7-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>>> _______________________________________________ >>>>> jboss-as7-dev mailing list >>>>> jboss-as7-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>> _______________________________________________ >>>> jboss-as7-dev mailing list >>>> jboss-as7-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>> _______________________________________________ >>> jboss-as7-dev mailing list >>> jboss-as7-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev