Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 TestSuite

@Test(groups = { "web_profile" } ) public void testEJBLite throws Exception {}

Shelly McGowan JBoss, by Red Hat ----- Original Message ----- From: "Aslak Knutsen"<aknutsen(a)redhat.com> To: "Andrew Lee Rubinger"<andrew.rubinger(a)redhat.com> Cc: jboss-as7-dev(a)lists.jboss.org Sent: Tuesday, March 22, 2011 10:38:08 AM Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 TestSuite > > Why @IntegrationTest? By simply being in the AS Integration Test > module, wouldn't that be self-explanatory? > The use of @IntegrationTest in CDI TCK is basically to separate tests that can run in Weld SE (mocked EE v) vs the ones that need a full EE container. Not sure the AS Suite needs that kind of separation. -aslak- > On 03/20/2011 10:58 AM, Shelly McGowan wrote: >> >> >> >> Why don't we consider what was done for the CDI and BV TCKs. >> Annotations[1] are used to define the test type. >> If it is a spec assertion, the spec reference is clearly identified: >> >> @SpecAssertion(section = "2.1.1", id = "c") >> >> or you could annotate the Test with @IntegrationTest which could be >> used to denote that AS-specific test. >> >> Test (not code) coverage reports can be generated to identify what >> the coverage is for each component. The report >> provides links to fisheye and the test code in svn. >> >> >> [1]http://community.jboss.org/wiki/BeanValidationTCK#What_do_all_the_annot... >> Other references: >> [2]http://docs.jboss.org/hibernate/stable/beanvalidation/tck/reference/htm... >> [3]http://anonsvn.jboss.org/repos/weld/cdi-tck/trunk/impl/src/main/java/or... >> >> >> Shelly McGowan >> JBoss, by Red Hat >> >> ----- Original Message ----- >> From: "Andrew Lee Rubinger"<andrew.rubinger(a)redhat.com> >> To: jboss-as7-dev(a)lists.jboss.org >> Sent: Saturday, March 19, 2011 5:34:30 PM >> Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 >> TestSuite >> >> It occurs to me I might not have been clear about the JIRA-per-test >> thing. >> >> Say you're working on Feature X. You might commit the feature and >> the >> test alongside one another, citing the same JIRA, with the commit >> message: >> >> "[JBAS-XXX] Implement Feature X and tests". >> >> That's great. No separate issues needed for the feature and the test >> each. >> >> What's no good is going in and adding a bunch of tests for EJB, Web, >> etc >> while we pump up our coverage, alongside a commit message: >> >> "[JBAS-XXX] Add spec coverage to the testsuite" >> >> ...that tells us nothing about each test being made, just that we >> dumped >> 'em in there. Which is obvious, because they're there. >> >> S, >> ALR >> >> On 03/19/2011 05:25 PM, Andrew Lee Rubinger wrote: >>> Inline. >>> >>> On 03/19/2011 03:19 PM, Jason Greene wrote: >>>> I really like the idea of our primary test suite covering API >>>> interaction and your typical unit test which is tightly coupled to >>>> internals. I also like your idea to separate spec Apis from JBoss >>>> aAPIs. >>>> >>>> However, I disagree on the Jira per test requirement. This is only >>>> because I worked with a similar model (all jiras require a test), >>>> and you ended up with lots of duplicate tests, and you always had >>>> to read Jira to understand what was being tested. Also, the tests >>>> ended up being too specific, typically an exact replication of the >>>> reporter's workflow. >>> >>> Consider this: If you had to read JIRA to understand what was being >>> tested, that's a clear violation of point 3): Document the test. >>> And >>> failing good documentation, with no JIRA to turn to, you're flying >>> blind. >>> >>> Also, how exactly does registering a JIRA encourage test >>> duplication? >>> Whether the tests are in an issue tracker or not seems to be >>> unrelated >>> to folks writing duplicate tests. >>>> >>>> IMO a better way to organize it would be by functional area, with >>>> generous descriptions. All spec tests should reference spec >>>> sections, for example. If we have a Jira that demonstrates lack of >>>> coverage, we should translate the problem into meaningful and >>>> reusable tests for the gaps we missed. Those tests should >>>> primarily refer to the API contract and not the Jira. Although >>>> commenting a list of jiras for extra info may be useful provided >>>> the Jira doesn't confuse what the correct behavior should be. >>> >>> I'm a fan of organizing by functional areas and spec references. >>> We'd >>> done that in EJB3 with success: some packages were even named for >>> the >>> spec. Regardless, every commit had a JIRA association. >>> >>> In the end, so long as it's perfectly clear what we're testing and >>> why, >>> I'll have no objections. But I do believe that JIRA is the obvious >>> answer to that question, and requires minimal effort to make a new >>> issue. Especially since every commit is isolated to one concern and >>> linked back to JIRA anyway, right? ;) >>> >>> S, >>> ALR >>> >>>> >>>> Sent from my iPad >>>> >>>> On Mar 18, 2011, at 2:21 AM, Andrew Lee >>>> Rubinger<andrew.rubinger(a)redhat.com> wrote: >>>> >>>>> Looks like a lot of us have different ideas for what the AS7 >>>>> Integration >>>>> TestSuite should consist of, so I'll kickoff with what I believe >>>>> is the >>>>> first design proposal towards getting coverage focused on the >>>>> end-user >>>>> (not certifying our own internals). >>>>> >>>>> I suspect this breaks down into two categories, which may be >>>>> modelled as >>>>> separate modules under the existing "testsuite" aggregator >>>>> parent: >>>>> >>>>> * Specification >>>>> * AS-specific APIs >>>>> >>>>> This isn't difficult work, though I do think it's important we >>>>> consider >>>>> some hard rules. IMO we should be developing these suites as if >>>>> we were >>>>> application developers, not wearing our server dev hats. >>>>> >>>>> ---------------- >>>>> >>>>> [End Goal] >>>>> >>>>> 1) No compile-time dependencies in the module except for what's >>>>> absolutely necessary. >>>>> >>>>> For the spec suite, this means: JDK and EE Spec APIs only in the >>>>> compilation classpath. Testable asset sources and resources (ie. >>>>> EJBs, >>>>> Servlets, etc) would live under src/main/* to enforce that. Only >>>>> the >>>>> tests themselves would be located under "src/test/*". >>>>> >>>>> The AS-specific API suite may also add in our own APIs to the >>>>> compilation classpath, but the line should end there. In "test" >>>>> scope >>>>> we can place all runtime dependencies. >>>>> >>>>> For the specification suite, AS-specific grammars like our own >>>>> deployment descriptors are fine; these are in many ways >>>>> equivalent to >>>>> the TCK porting layer. We're not building a TCK; we're showing >>>>> that our >>>>> implementation supports the features advertised. >>>>> >>>>> 2) Every single new test created is to have an associated JIRA. >>>>> >>>>> We all remember the nightmare it was when the old AS4-6 suite >>>>> would fall >>>>> down. We'd comb through each test, at times trying to determine >>>>> its >>>>> purpose. By linking to JIRA we get history of intent, which acts >>>>> as a >>>>> nice record even in the case that the test isn't so >>>>> well-documented. >>>>> I'd argue that tests are a bigger asset than our code, and we >>>>> should be >>>>> thinking about these in terms of long-term maintenance to outlive >>>>> any >>>>> specific impl. >>>>> >>>>> 3) Documentation >>>>> >>>>> Alongside the JIRA reference, a quick note about we're looking to >>>>> accomplish is something I find very helpful. I don't personally >>>>> buy the >>>>> argument that code is self-documenting if written well. It gets >>>>> refactored and stale over time. >>>>> >>>>> 4) Run-mode profiles >>>>> >>>>> Arquillian provides a wonderful abstraction such that we can get >>>>> coverage for AS in both remote managed *and* embedded modes >>>>> without >>>>> changing the test itself. To certify that everything is working >>>>> as >>>>> advertised no matter the runtime, we should be able to run the >>>>> same >>>>> suite in standalone, domain, and embedded modes (generally >>>>> speaking). >>>>> >>>>> 5) Porting of AS6 Tests >>>>> >>>>> There's no discounting the value this coverage has given us, >>>>> though I >>>>> question the purpose of a lot of these tests. I think a great >>>>> majority >>>>> of these need to come into the new codebase, refactored to align >>>>> if needed. >>>>> >>>>> ---------------- >>>>> >>>>> [Current State] >>>>> >>>>> Here[1] is an example of what I believe to be a simple, >>>>> well-written >>>>> test, with the exception that the tested Servlet and EJB are in >>>>> the same >>>>> test source folder. >>>>> >>>>> The current "testsuite" aggregator contains modules which mix our >>>>> end-user certification stuff alongside internals, so I think >>>>> these >>>>> should be separated out. >>>>> >>>>> A lot of this is set up in some fashion already, but I would like >>>>> to see us: >>>>> >>>>> 1) Agree upon a strict scope for each type of testsuite along the >>>>> lines >>>>> of my points above, once we reach agreement >>>>> 2) Upgrade to ARQ 1.0.0.Alpha5 (which implies ShrinkWrap >>>>> 1.0.0-alpha-12), just released tonight. Currently AS is on a >>>>> forked >>>>> release of ARQ for OSGi purposes, and these changes, if >>>>> necessary, need >>>>> to get upstream so we can do upgrades. >>>>> >>>>> It's clear that AS7 has made full-steam-ahead progress since last >>>>> summer, and with a little organization our testsuite can give us >>>>> a great >>>>> view of where we stand, from an end-user's perspective, with >>>>> minimal >>>>> investment. >>>>> >>>>> S, >>>>> ALR >>>>> >>>>> >>>>> [1] >>>>> https://github.com/jbossas/jboss-as/blob/master/testsuite/integration/src... >>>>> _______________________________________________ >>>>> jboss-as7-dev mailing list >>>>> jboss-as7-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>> >>>> _______________________________________________ >>>> jboss-as7-dev mailing list >>>> jboss-as7-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>> _______________________________________________ >>> jboss-as7-dev mailing list >>> jboss-as7-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev