Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 TestSuite

I think you should define more carefully what you are proposing, then people might agree more... * if you write a regression test, it must have a linked JIRA OR it must reference a forum post/mailing list archive (you might have written a test to prove that there isn't a problem to a user, but silly to "waste" that test...) * if you write a test for a new feature, and the feature has a JIRA issue, then you should link it. New feature development doesn't always work well with an issue tracker IMO. You also need to think about linkage. Each test should have a comment /** * Testing blah... * * JIRA: JBAS-1234 */ and all bugs should also give the FQMN of the associated tests - this way you can go from one to the other very easily. We've used this scheme a lot with Weld and the CDI TCK and I find it works very well, especially 9 months later when you come back to an issue and can't remember what I did when.

On 23 Mar 2011, at 06:10, Andrew Lee Rubinger wrote: > I'm still confused by the general hesitation towards a JIRA association > for each test. To me this is almost a redundant policy; since when are > we committing *anything* without linking back to an issue? > > Comments inline. > > On 03/22/2011 12:11 PM, Shelly McGowan wrote: >> I realize the intent of @IntegrationTest annotation for the CDI TCK. It was just an example in response to the comments about: >> >> Jason: IMO a better way to organize it would be by functional area, with generous descriptions. All spec tests should reference spec >> sections, for example. >> >> ALR: I'm a fan of organizing by functional areas and spec references. >> >> therefore proposing to use annotations for such organization instead of JIRA. We can devise our own annotations that make sense. > > OK, what tooling do you envision would make use of this metadata? Or is > it just there as a documentation marker for us? > > To me, annotations are to be consumed/processed/enforced by software; > comments are to be read by developers. > >> Another consideration is because they are integration tests, it is common that more than one >> technology is used in the test cases. Having tests organized by directory structure does not always reflect that. >> >> integration/src/test/java/org/jboss/as/testsuite/integration/jms/JMSTestCase.java >> >> These tests could contain test cases that may also cover MDB, XA, JCA. So as in [3] below, we could do something like: >> >> @Test(groups = { "JMS", "XA", "MDB"}) >> public void testJms() throws Exception {} >> >> or >> >> @Test(groups = { "web_profile" } ) >> public void testEJBLite throws Exception {} > > What does @Test.groups represent? Is that a proposal for our own > extension? JUnit doesn't have this property in @org.junit.Test, so if > it's our own thing we should likely rename to something else... > > http://kentbeck.github.com/junit/javadoc/latest/org/junit/Test.html > > S, > ALR > >> >> Shelly McGowan >> JBoss, by Red Hat >> >> ----- Original Message ----- >> From: "Aslak Knutsen"<aknutsen(a)redhat.com> >> To: "Andrew Lee Rubinger"<andrew.rubinger(a)redhat.com> >> Cc: jboss-as7-dev(a)lists.jboss.org >> Sent: Tuesday, March 22, 2011 10:38:08 AM >> Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 TestSuite >> >>> >>> Why @IntegrationTest? By simply being in the AS Integration Test >>> module, wouldn't that be self-explanatory? >>> >> >> The use of @IntegrationTest in CDI TCK is basically to separate tests that can run in Weld SE (mocked EE v) vs the ones that need a full EE container. >> >> Not sure the AS Suite needs that kind of separation. >> >> -aslak- >> >>> On 03/20/2011 10:58 AM, Shelly McGowan wrote: >>>> >>>> >>>> >>>> Why don't we consider what was done for the CDI and BV TCKs. >>>> Annotations[1] are used to define the test type. >>>> If it is a spec assertion, the spec reference is clearly identified: >>>> >>>> @SpecAssertion(section = "2.1.1", id = "c") >>>> >>>> or you could annotate the Test with @IntegrationTest which could be >>>> used to denote that AS-specific test. >>>> >>>> Test (not code) coverage reports can be generated to identify what >>>> the coverage is for each component. The report >>>> provides links to fisheye and the test code in svn. >>>> >>>> >>>> [1]http://community.jboss.org/wiki/BeanValidationTCK#What_do_all_the_annot... >>>> Other references: >>>> [2]http://docs.jboss.org/hibernate/stable/beanvalidation/tck/reference/htm... >>>> [3]http://anonsvn.jboss.org/repos/weld/cdi-tck/trunk/impl/src/main/java/or... >>>> >>>> >>>> Shelly McGowan >>>> JBoss, by Red Hat >>>> >>>> ----- Original Message ----- >>>> From: "Andrew Lee Rubinger"<andrew.rubinger(a)redhat.com> >>>> To: jboss-as7-dev(a)lists.jboss.org >>>> Sent: Saturday, March 19, 2011 5:34:30 PM >>>> Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 >>>> TestSuite >>>> >>>> It occurs to me I might not have been clear about the JIRA-per-test >>>> thing. >>>> >>>> Say you're working on Feature X. You might commit the feature and >>>> the >>>> test alongside one another, citing the same JIRA, with the commit >>>> message: >>>> >>>> "[JBAS-XXX] Implement Feature X and tests". >>>> >>>> That's great. No separate issues needed for the feature and the test >>>> each. >>>> >>>> What's no good is going in and adding a bunch of tests for EJB, Web, >>>> etc >>>> while we pump up our coverage, alongside a commit message: >>>> >>>> "[JBAS-XXX] Add spec coverage to the testsuite" >>>> >>>> ...that tells us nothing about each test being made, just that we >>>> dumped >>>> 'em in there. Which is obvious, because they're there. >>>> >>>> S, >>>> ALR >>>> >>>> On 03/19/2011 05:25 PM, Andrew Lee Rubinger wrote: >>>>> Inline. >>>>> >>>>> On 03/19/2011 03:19 PM, Jason Greene wrote: >>>>>> I really like the idea of our primary test suite covering API >>>>>> interaction and your typical unit test which is tightly coupled to >>>>>> internals. I also like your idea to separate spec Apis from JBoss >>>>>> aAPIs. >>>>>> >>>>>> However, I disagree on the Jira per test requirement. This is only >>>>>> because I worked with a similar model (all jiras require a test), >>>>>> and you ended up with lots of duplicate tests, and you always had >>>>>> to read Jira to understand what was being tested. Also, the tests >>>>>> ended up being too specific, typically an exact replication of the >>>>>> reporter's workflow. >>>>> >>>>> Consider this: If you had to read JIRA to understand what was being >>>>> tested, that's a clear violation of point 3): Document the test. >>>>> And >>>>> failing good documentation, with no JIRA to turn to, you're flying >>>>> blind. >>>>> >>>>> Also, how exactly does registering a JIRA encourage test >>>>> duplication? >>>>> Whether the tests are in an issue tracker or not seems to be >>>>> unrelated >>>>> to folks writing duplicate tests. >>>>>> >>>>>> IMO a better way to organize it would be by functional area, with >>>>>> generous descriptions. All spec tests should reference spec >>>>>> sections, for example. If we have a Jira that demonstrates lack of >>>>>> coverage, we should translate the problem into meaningful and >>>>>> reusable tests for the gaps we missed. Those tests should >>>>>> primarily refer to the API contract and not the Jira. Although >>>>>> commenting a list of jiras for extra info may be useful provided >>>>>> the Jira doesn't confuse what the correct behavior should be. >>>>> >>>>> I'm a fan of organizing by functional areas and spec references. >>>>> We'd >>>>> done that in EJB3 with success: some packages were even named for >>>>> the >>>>> spec. Regardless, every commit had a JIRA association. >>>>> >>>>> In the end, so long as it's perfectly clear what we're testing and >>>>> why, >>>>> I'll have no objections. But I do believe that JIRA is the obvious >>>>> answer to that question, and requires minimal effort to make a new >>>>> issue. Especially since every commit is isolated to one concern and >>>>> linked back to JIRA anyway, right? ;) >>>>> >>>>> S, >>>>> ALR >>>>> >>>>>> >>>>>> Sent from my iPad >>>>>> >>>>>> On Mar 18, 2011, at 2:21 AM, Andrew Lee >>>>>> Rubinger<andrew.rubinger(a)redhat.com> wrote: >>>>>> >>>>>>> Looks like a lot of us have different ideas for what the AS7 >>>>>>> Integration >>>>>>> TestSuite should consist of, so I'll kickoff with what I believe >>>>>>> is the >>>>>>> first design proposal towards getting coverage focused on the >>>>>>> end-user >>>>>>> (not certifying our own internals). >>>>>>> >>>>>>> I suspect this breaks down into two categories, which may be >>>>>>> modelled as >>>>>>> separate modules under the existing "testsuite" aggregator >>>>>>> parent: >>>>>>> >>>>>>> * Specification >>>>>>> * AS-specific APIs >>>>>>> >>>>>>> This isn't difficult work, though I do think it's important we >>>>>>> consider >>>>>>> some hard rules. IMO we should be developing these suites as if >>>>>>> we were >>>>>>> application developers, not wearing our server dev hats. >>>>>>> >>>>>>> ---------------- >>>>>>> >>>>>>> [End Goal] >>>>>>> >>>>>>> 1) No compile-time dependencies in the module except for what's >>>>>>> absolutely necessary. >>>>>>> >>>>>>> For the spec suite, this means: JDK and EE Spec APIs only in the >>>>>>> compilation classpath. Testable asset sources and resources (ie. >>>>>>> EJBs, >>>>>>> Servlets, etc) would live under src/main/* to enforce that. Only >>>>>>> the >>>>>>> tests themselves would be located under "src/test/*". >>>>>>> >>>>>>> The AS-specific API suite may also add in our own APIs to the >>>>>>> compilation classpath, but the line should end there. In "test" >>>>>>> scope >>>>>>> we can place all runtime dependencies. >>>>>>> >>>>>>> For the specification suite, AS-specific grammars like our own >>>>>>> deployment descriptors are fine; these are in many ways >>>>>>> equivalent to >>>>>>> the TCK porting layer. We're not building a TCK; we're showing >>>>>>> that our >>>>>>> implementation supports the features advertised. >>>>>>> >>>>>>> 2) Every single new test created is to have an associated JIRA. >>>>>>> >>>>>>> We all remember the nightmare it was when the old AS4-6 suite >>>>>>> would fall >>>>>>> down. We'd comb through each test, at times trying to determine >>>>>>> its >>>>>>> purpose. By linking to JIRA we get history of intent, which acts >>>>>>> as a >>>>>>> nice record even in the case that the test isn't so >>>>>>> well-documented. >>>>>>> I'd argue that tests are a bigger asset than our code, and we >>>>>>> should be >>>>>>> thinking about these in terms of long-term maintenance to outlive >>>>>>> any >>>>>>> specific impl. >>>>>>> >>>>>>> 3) Documentation >>>>>>> >>>>>>> Alongside the JIRA reference, a quick note about we're looking to >>>>>>> accomplish is something I find very helpful. I don't personally >>>>>>> buy the >>>>>>> argument that code is self-documenting if written well. It gets >>>>>>> refactored and stale over time. >>>>>>> >>>>>>> 4) Run-mode profiles >>>>>>> >>>>>>> Arquillian provides a wonderful abstraction such that we can get >>>>>>> coverage for AS in both remote managed *and* embedded modes >>>>>>> without >>>>>>> changing the test itself. To certify that everything is working >>>>>>> as >>>>>>> advertised no matter the runtime, we should be able to run the >>>>>>> same >>>>>>> suite in standalone, domain, and embedded modes (generally >>>>>>> speaking). >>>>>>> >>>>>>> 5) Porting of AS6 Tests >>>>>>> >>>>>>> There's no discounting the value this coverage has given us, >>>>>>> though I >>>>>>> question the purpose of a lot of these tests. I think a great >>>>>>> majority >>>>>>> of these need to come into the new codebase, refactored to align >>>>>>> if needed. >>>>>>> >>>>>>> ---------------- >>>>>>> >>>>>>> [Current State] >>>>>>> >>>>>>> Here[1] is an example of what I believe to be a simple, >>>>>>> well-written >>>>>>> test, with the exception that the tested Servlet and EJB are in >>>>>>> the same >>>>>>> test source folder. >>>>>>> >>>>>>> The current "testsuite" aggregator contains modules which mix our >>>>>>> end-user certification stuff alongside internals, so I think >>>>>>> these >>>>>>> should be separated out. >>>>>>> >>>>>>> A lot of this is set up in some fashion already, but I would like >>>>>>> to see us: >>>>>>> >>>>>>> 1) Agree upon a strict scope for each type of testsuite along the >>>>>>> lines >>>>>>> of my points above, once we reach agreement >>>>>>> 2) Upgrade to ARQ 1.0.0.Alpha5 (which implies ShrinkWrap >>>>>>> 1.0.0-alpha-12), just released tonight. Currently AS is on a >>>>>>> forked >>>>>>> release of ARQ for OSGi purposes, and these changes, if >>>>>>> necessary, need >>>>>>> to get upstream so we can do upgrades. >>>>>>> >>>>>>> It's clear that AS7 has made full-steam-ahead progress since last >>>>>>> summer, and with a little organization our testsuite can give us >>>>>>> a great >>>>>>> view of where we stand, from an end-user's perspective, with >>>>>>> minimal >>>>>>> investment. >>>>>>> >>>>>>> S, >>>>>>> ALR >>>>>>> >>>>>>> >>>>>>> [1] >>>>>>> https://github.com/jbossas/jboss-as/blob/master/testsuite/integration/src... >>>>>>> _______________________________________________ >>>>>>> jboss-as7-dev mailing list >>>>>>> jboss-as7-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>>>> >>>>>> _______________________________________________ >>>>>> jboss-as7-dev mailing list >>>>>> jboss-as7-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>>> _______________________________________________ >>>>> jboss-as7-dev mailing list >>>>> jboss-as7-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>> _______________________________________________ >>>> jboss-as7-dev mailing list >>>> jboss-as7-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>> _______________________________________________ >>> jboss-as7-dev mailing list >>> jboss-as7-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev