Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 TestSuite

Andrew, To bring the thread up a level, I propose the following requirements: - no hardcoding of configuration elements in the test code - currently many tests have localhost and java:/H2DS - integration tests should run remote, not embedded - provide an Abstract test class that can be extended so adding tests focus on test logic not test configuration - can be run in standalone or domain (and cloud)

- have reporting mechanism so test coverage can be obtained based on query - ex: what tests are available for lifecycle callbacks - (as Darran said) require a test be added with a fix for each JIRA - extensible for other stakeholders (QE) of the integration test suite Shelly McGowan JBoss, by Red Hat ----- Original Message ----- From: "Shelly McGowan"<smcgowan(a)redhat.com> To: "Andrew Lee Rubinger"<andrew.rubinger(a)redhat.com> Cc: "Aslak Knutsen"<aknutsen(a)redhat.com>, jboss-as7-dev(a)lists.jboss.org Sent: Wednesday, March 23, 2011 7:36:58 AM Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 TestSuite > > @Test(groups = { "web_profile" } ) > public void testEJBLite throws Exception {} What does @Test.groups represent? Is that a proposal for our own extension? JUnit doesn't have this property in @org.junit.Test, so if it's our own thing we should likely rename to something else... http://kentbeck.github.com/junit/javadoc/latest/org/junit/Test.html S, ALR This, as shown in [3]: import org.testng.annotations.Test; http://testng.org/javadoc/org/testng/annotations/Test.html > > Shelly McGowan > JBoss, by Red Hat > > ----- Original Message ----- > From: "Aslak Knutsen"<aknutsen(a)redhat.com> > To: "Andrew Lee Rubinger"<andrew.rubinger(a)redhat.com> > Cc: jboss-as7-dev(a)lists.jboss.org > Sent: Tuesday, March 22, 2011 10:38:08 AM > Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 TestSuite > >> >> Why @IntegrationTest? By simply being in the AS Integration Test >> module, wouldn't that be self-explanatory? >> > > The use of @IntegrationTest in CDI TCK is basically to separate tests that can run in Weld SE (mocked EE v) vs the ones that need a full EE container. > > Not sure the AS Suite needs that kind of separation. > > -aslak- > >> On 03/20/2011 10:58 AM, Shelly McGowan wrote: >>> >>> >>> >>> Why don't we consider what was done for the CDI and BV TCKs. >>> Annotations[1] are used to define the test type. >>> If it is a spec assertion, the spec reference is clearly identified: >>> >>> @SpecAssertion(section = "2.1.1", id = "c") >>> >>> or you could annotate the Test with @IntegrationTest which could be >>> used to denote that AS-specific test. >>> >>> Test (not code) coverage reports can be generated to identify what >>> the coverage is for each component. The report >>> provides links to fisheye and the test code in svn. >>> >>> >>> [1]http://community.jboss.org/wiki/BeanValidationTCK#What_do_all_the_annot... >>> Other references: >>> [2]http://docs.jboss.org/hibernate/stable/beanvalidation/tck/reference/htm... >>> [3]http://anonsvn.jboss.org/repos/weld/cdi-tck/trunk/impl/src/main/java/or... >>> >>> >>> Shelly McGowan >>> JBoss, by Red Hat >>> >>> ----- Original Message ----- >>> From: "Andrew Lee Rubinger"<andrew.rubinger(a)redhat.com> >>> To: jboss-as7-dev(a)lists.jboss.org >>> Sent: Saturday, March 19, 2011 5:34:30 PM >>> Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 >>> TestSuite >>> >>> It occurs to me I might not have been clear about the JIRA-per-test >>> thing. >>> >>> Say you're working on Feature X. You might commit the feature and >>> the >>> test alongside one another, citing the same JIRA, with the commit >>> message: >>> >>> "[JBAS-XXX] Implement Feature X and tests". >>> >>> That's great. No separate issues needed for the feature and the test >>> each. >>> >>> What's no good is going in and adding a bunch of tests for EJB, Web, >>> etc >>> while we pump up our coverage, alongside a commit message: >>> >>> "[JBAS-XXX] Add spec coverage to the testsuite" >>> >>> ...that tells us nothing about each test being made, just that we >>> dumped >>> 'em in there. Which is obvious, because they're there. >>> >>> S, >>> ALR >>> >>> On 03/19/2011 05:25 PM, Andrew Lee Rubinger wrote: >>>> Inline. >>>> >>>> On 03/19/2011 03:19 PM, Jason Greene wrote: >>>>> I really like the idea of our primary test suite covering API >>>>> interaction and your typical unit test which is tightly coupled to >>>>> internals. I also like your idea to separate spec Apis from JBoss >>>>> aAPIs. >>>>> >>>>> However, I disagree on the Jira per test requirement. This is only >>>>> because I worked with a similar model (all jiras require a test), >>>>> and you ended up with lots of duplicate tests, and you always had >>>>> to read Jira to understand what was being tested. Also, the tests >>>>> ended up being too specific, typically an exact replication of the >>>>> reporter's workflow. >>>> >>>> Consider this: If you had to read JIRA to understand what was being >>>> tested, that's a clear violation of point 3): Document the test. >>>> And >>>> failing good documentation, with no JIRA to turn to, you're flying >>>> blind. >>>> >>>> Also, how exactly does registering a JIRA encourage test >>>> duplication? >>>> Whether the tests are in an issue tracker or not seems to be >>>> unrelated >>>> to folks writing duplicate tests. >>>>> >>>>> IMO a better way to organize it would be by functional area, with >>>>> generous descriptions. All spec tests should reference spec >>>>> sections, for example. If we have a Jira that demonstrates lack of >>>>> coverage, we should translate the problem into meaningful and >>>>> reusable tests for the gaps we missed. Those tests should >>>>> primarily refer to the API contract and not the Jira. Although >>>>> commenting a list of jiras for extra info may be useful provided >>>>> the Jira doesn't confuse what the correct behavior should be. >>>> >>>> I'm a fan of organizing by functional areas and spec references. >>>> We'd >>>> done that in EJB3 with success: some packages were even named for >>>> the >>>> spec. Regardless, every commit had a JIRA association. >>>> >>>> In the end, so long as it's perfectly clear what we're testing and >>>> why, >>>> I'll have no objections. But I do believe that JIRA is the obvious >>>> answer to that question, and requires minimal effort to make a new >>>> issue. Especially since every commit is isolated to one concern and >>>> linked back to JIRA anyway, right? ;) >>>> >>>> S, >>>> ALR >>>> >>>>> >>>>> Sent from my iPad >>>>> >>>>> On Mar 18, 2011, at 2:21 AM, Andrew Lee >>>>> Rubinger<andrew.rubinger(a)redhat.com> wrote: >>>>> >>>>>> Looks like a lot of us have different ideas for what the AS7 >>>>>> Integration >>>>>> TestSuite should consist of, so I'll kickoff with what I believe >>>>>> is the >>>>>> first design proposal towards getting coverage focused on the >>>>>> end-user >>>>>> (not certifying our own internals). >>>>>> >>>>>> I suspect this breaks down into two categories, which may be >>>>>> modelled as >>>>>> separate modules under the existing "testsuite" aggregator >>>>>> parent: >>>>>> >>>>>> * Specification >>>>>> * AS-specific APIs >>>>>> >>>>>> This isn't difficult work, though I do think it's important we >>>>>> consider >>>>>> some hard rules. IMO we should be developing these suites as if >>>>>> we were >>>>>> application developers, not wearing our server dev hats. >>>>>> >>>>>> ---------------- >>>>>> >>>>>> [End Goal] >>>>>> >>>>>> 1) No compile-time dependencies in the module except for what's >>>>>> absolutely necessary. >>>>>> >>>>>> For the spec suite, this means: JDK and EE Spec APIs only in the >>>>>> compilation classpath. Testable asset sources and resources (ie. >>>>>> EJBs, >>>>>> Servlets, etc) would live under src/main/* to enforce that. Only >>>>>> the >>>>>> tests themselves would be located under "src/test/*". >>>>>> >>>>>> The AS-specific API suite may also add in our own APIs to the >>>>>> compilation classpath, but the line should end there. In "test" >>>>>> scope >>>>>> we can place all runtime dependencies. >>>>>> >>>>>> For the specification suite, AS-specific grammars like our own >>>>>> deployment descriptors are fine; these are in many ways >>>>>> equivalent to >>>>>> the TCK porting layer. We're not building a TCK; we're showing >>>>>> that our >>>>>> implementation supports the features advertised. >>>>>> >>>>>> 2) Every single new test created is to have an associated JIRA. >>>>>> >>>>>> We all remember the nightmare it was when the old AS4-6 suite >>>>>> would fall >>>>>> down. We'd comb through each test, at times trying to determine >>>>>> its >>>>>> purpose. By linking to JIRA we get history of intent, which acts >>>>>> as a >>>>>> nice record even in the case that the test isn't so >>>>>> well-documented. >>>>>> I'd argue that tests are a bigger asset than our code, and we >>>>>> should be >>>>>> thinking about these in terms of long-term maintenance to outlive >>>>>> any >>>>>> specific impl. >>>>>> >>>>>> 3) Documentation >>>>>> >>>>>> Alongside the JIRA reference, a quick note about we're looking to >>>>>> accomplish is something I find very helpful. I don't personally >>>>>> buy the >>>>>> argument that code is self-documenting if written well. It gets >>>>>> refactored and stale over time. >>>>>> >>>>>> 4) Run-mode profiles >>>>>> >>>>>> Arquillian provides a wonderful abstraction such that we can get >>>>>> coverage for AS in both remote managed *and* embedded modes >>>>>> without >>>>>> changing the test itself. To certify that everything is working >>>>>> as >>>>>> advertised no matter the runtime, we should be able to run the >>>>>> same >>>>>> suite in standalone, domain, and embedded modes (generally >>>>>> speaking). >>>>>> >>>>>> 5) Porting of AS6 Tests >>>>>> >>>>>> There's no discounting the value this coverage has given us, >>>>>> though I >>>>>> question the purpose of a lot of these tests. I think a great >>>>>> majority >>>>>> of these need to come into the new codebase, refactored to align >>>>>> if needed. >>>>>> >>>>>> ---------------- >>>>>> >>>>>> [Current State] >>>>>> >>>>>> Here[1] is an example of what I believe to be a simple, >>>>>> well-written >>>>>> test, with the exception that the tested Servlet and EJB are in >>>>>> the same >>>>>> test source folder. >>>>>> >>>>>> The current "testsuite" aggregator contains modules which mix our >>>>>> end-user certification stuff alongside internals, so I think >>>>>> these >>>>>> should be separated out. >>>>>> >>>>>> A lot of this is set up in some fashion already, but I would like >>>>>> to see us: >>>>>> >>>>>> 1) Agree upon a strict scope for each type of testsuite along the >>>>>> lines >>>>>> of my points above, once we reach agreement >>>>>> 2) Upgrade to ARQ 1.0.0.Alpha5 (which implies ShrinkWrap >>>>>> 1.0.0-alpha-12), just released tonight. Currently AS is on a >>>>>> forked >>>>>> release of ARQ for OSGi purposes, and these changes, if >>>>>> necessary, need >>>>>> to get upstream so we can do upgrades. >>>>>> >>>>>> It's clear that AS7 has made full-steam-ahead progress since last >>>>>> summer, and with a little organization our testsuite can give us >>>>>> a great >>>>>> view of where we stand, from an end-user's perspective, with >>>>>> minimal >>>>>> investment. >>>>>> >>>>>> S, >>>>>> ALR >>>>>> >>>>>> >>>>>> [1] >>>>>> https://github.com/jbossas/jboss-as/blob/master/testsuite/integration/src... >>>>>> _______________________________________________ >>>>>> jboss-as7-dev mailing list >>>>>> jboss-as7-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>>> >>>>> _______________________________________________ >>>>> jboss-as7-dev mailing list >>>>> jboss-as7-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>> _______________________________________________ >>>> jboss-as7-dev mailing list >>>> jboss-as7-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>> _______________________________________________ >>> jboss-as7-dev mailing list >>> jboss-as7-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev