Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 TestSuite

Why @IntegrationTest? By simply being in the AS Integration Test module, wouldn't that be self-explanatory?

On 03/20/2011 10:58 AM, Shelly McGowan wrote: > > > > Why don't we consider what was done for the CDI and BV TCKs. > Annotations[1] are used to define the test type. > If it is a spec assertion, the spec reference is clearly identified: > > @SpecAssertion(section = "2.1.1", id = "c") > > or you could annotate the Test with @IntegrationTest which could be > used to denote that AS-specific test. > > Test (not code) coverage reports can be generated to identify what > the coverage is for each component. The report > provides links to fisheye and the test code in svn. > > > [1]http://community.jboss.org/wiki/BeanValidationTCK#What_do_all_the_annot... > Other references: > [2]http://docs.jboss.org/hibernate/stable/beanvalidation/tck/reference/htm... > [3]http://anonsvn.jboss.org/repos/weld/cdi-tck/trunk/impl/src/main/java/or... > > > Shelly McGowan > JBoss, by Red Hat > > ----- Original Message ----- > From: "Andrew Lee Rubinger"<andrew.rubinger(a)redhat.com> > To: jboss-as7-dev(a)lists.jboss.org > Sent: Saturday, March 19, 2011 5:34:30 PM > Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 > TestSuite > > It occurs to me I might not have been clear about the JIRA-per-test > thing. > > Say you're working on Feature X. You might commit the feature and > the > test alongside one another, citing the same JIRA, with the commit > message: > > "[JBAS-XXX] Implement Feature X and tests". > > That's great. No separate issues needed for the feature and the test > each. > > What's no good is going in and adding a bunch of tests for EJB, Web, > etc > while we pump up our coverage, alongside a commit message: > > "[JBAS-XXX] Add spec coverage to the testsuite" > > ...that tells us nothing about each test being made, just that we > dumped > 'em in there. Which is obvious, because they're there. > > S, > ALR > > On 03/19/2011 05:25 PM, Andrew Lee Rubinger wrote: >> Inline. >> >> On 03/19/2011 03:19 PM, Jason Greene wrote: >>> I really like the idea of our primary test suite covering API >>> interaction and your typical unit test which is tightly coupled to >>> internals. I also like your idea to separate spec Apis from JBoss >>> aAPIs. >>> >>> However, I disagree on the Jira per test requirement. This is only >>> because I worked with a similar model (all jiras require a test), >>> and you ended up with lots of duplicate tests, and you always had >>> to read Jira to understand what was being tested. Also, the tests >>> ended up being too specific, typically an exact replication of the >>> reporter's workflow. >> >> Consider this: If you had to read JIRA to understand what was being >> tested, that's a clear violation of point 3): Document the test. >> And >> failing good documentation, with no JIRA to turn to, you're flying >> blind. >> >> Also, how exactly does registering a JIRA encourage test >> duplication? >> Whether the tests are in an issue tracker or not seems to be >> unrelated >> to folks writing duplicate tests. >>> >>> IMO a better way to organize it would be by functional area, with >>> generous descriptions. All spec tests should reference spec >>> sections, for example. If we have a Jira that demonstrates lack of >>> coverage, we should translate the problem into meaningful and >>> reusable tests for the gaps we missed. Those tests should >>> primarily refer to the API contract and not the Jira. Although >>> commenting a list of jiras for extra info may be useful provided >>> the Jira doesn't confuse what the correct behavior should be. >> >> I'm a fan of organizing by functional areas and spec references. >> We'd >> done that in EJB3 with success: some packages were even named for >> the >> spec. Regardless, every commit had a JIRA association. >> >> In the end, so long as it's perfectly clear what we're testing and >> why, >> I'll have no objections. But I do believe that JIRA is the obvious >> answer to that question, and requires minimal effort to make a new >> issue. Especially since every commit is isolated to one concern and >> linked back to JIRA anyway, right? ;) >> >> S, >> ALR >> >>> >>> Sent from my iPad >>> >>> On Mar 18, 2011, at 2:21 AM, Andrew Lee >>> Rubinger<andrew.rubinger(a)redhat.com> wrote: >>> >>>> Looks like a lot of us have different ideas for what the AS7 >>>> Integration >>>> TestSuite should consist of, so I'll kickoff with what I believe >>>> is the >>>> first design proposal towards getting coverage focused on the >>>> end-user >>>> (not certifying our own internals). >>>> >>>> I suspect this breaks down into two categories, which may be >>>> modelled as >>>> separate modules under the existing "testsuite" aggregator >>>> parent: >>>> >>>> * Specification >>>> * AS-specific APIs >>>> >>>> This isn't difficult work, though I do think it's important we >>>> consider >>>> some hard rules. IMO we should be developing these suites as if >>>> we were >>>> application developers, not wearing our server dev hats. >>>> >>>> ---------------- >>>> >>>> [End Goal] >>>> >>>> 1) No compile-time dependencies in the module except for what's >>>> absolutely necessary. >>>> >>>> For the spec suite, this means: JDK and EE Spec APIs only in the >>>> compilation classpath. Testable asset sources and resources (ie. >>>> EJBs, >>>> Servlets, etc) would live under src/main/* to enforce that. Only >>>> the >>>> tests themselves would be located under "src/test/*". >>>> >>>> The AS-specific API suite may also add in our own APIs to the >>>> compilation classpath, but the line should end there. In "test" >>>> scope >>>> we can place all runtime dependencies. >>>> >>>> For the specification suite, AS-specific grammars like our own >>>> deployment descriptors are fine; these are in many ways >>>> equivalent to >>>> the TCK porting layer. We're not building a TCK; we're showing >>>> that our >>>> implementation supports the features advertised. >>>> >>>> 2) Every single new test created is to have an associated JIRA. >>>> >>>> We all remember the nightmare it was when the old AS4-6 suite >>>> would fall >>>> down. We'd comb through each test, at times trying to determine >>>> its >>>> purpose. By linking to JIRA we get history of intent, which acts >>>> as a >>>> nice record even in the case that the test isn't so >>>> well-documented. >>>> I'd argue that tests are a bigger asset than our code, and we >>>> should be >>>> thinking about these in terms of long-term maintenance to outlive >>>> any >>>> specific impl. >>>> >>>> 3) Documentation >>>> >>>> Alongside the JIRA reference, a quick note about we're looking to >>>> accomplish is something I find very helpful. I don't personally >>>> buy the >>>> argument that code is self-documenting if written well. It gets >>>> refactored and stale over time. >>>> >>>> 4) Run-mode profiles >>>> >>>> Arquillian provides a wonderful abstraction such that we can get >>>> coverage for AS in both remote managed *and* embedded modes >>>> without >>>> changing the test itself. To certify that everything is working >>>> as >>>> advertised no matter the runtime, we should be able to run the >>>> same >>>> suite in standalone, domain, and embedded modes (generally >>>> speaking). >>>> >>>> 5) Porting of AS6 Tests >>>> >>>> There's no discounting the value this coverage has given us, >>>> though I >>>> question the purpose of a lot of these tests. I think a great >>>> majority >>>> of these need to come into the new codebase, refactored to align >>>> if needed. >>>> >>>> ---------------- >>>> >>>> [Current State] >>>> >>>> Here[1] is an example of what I believe to be a simple, >>>> well-written >>>> test, with the exception that the tested Servlet and EJB are in >>>> the same >>>> test source folder. >>>> >>>> The current "testsuite" aggregator contains modules which mix our >>>> end-user certification stuff alongside internals, so I think >>>> these >>>> should be separated out. >>>> >>>> A lot of this is set up in some fashion already, but I would like >>>> to see us: >>>> >>>> 1) Agree upon a strict scope for each type of testsuite along the >>>> lines >>>> of my points above, once we reach agreement >>>> 2) Upgrade to ARQ 1.0.0.Alpha5 (which implies ShrinkWrap >>>> 1.0.0-alpha-12), just released tonight. Currently AS is on a >>>> forked >>>> release of ARQ for OSGi purposes, and these changes, if >>>> necessary, need >>>> to get upstream so we can do upgrades. >>>> >>>> It's clear that AS7 has made full-steam-ahead progress since last >>>> summer, and with a little organization our testsuite can give us >>>> a great >>>> view of where we stand, from an end-user's perspective, with >>>> minimal >>>> investment. >>>> >>>> S, >>>> ALR >>>> >>>> >>>> [1] >>>> https://github.com/jbossas/jboss-as/blob/master/testsuite/integration/src... >>>> _______________________________________________ >>>> jboss-as7-dev mailing list >>>> jboss-as7-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>> >>> _______________________________________________ >>> jboss-as7-dev mailing list >>> jboss-as7-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev