[jboss-as7-dev] Requirements and Design Proposal: AS7 TestSuite

Tue Mar 22 10:38:08 EDT 2011

> 
> Why @IntegrationTest? By simply being in the AS Integration Test
> module, wouldn't that be self-explanatory?
> 

The use of @IntegrationTest in CDI TCK is basically to separate tests that can run in Weld SE (mocked EE v) vs the ones that need a full EE container.

Not sure the AS Suite needs that kind of separation.

-aslak-

> On 03/20/2011 10:58 AM, Shelly McGowan wrote:
> >
> >
> >
> > Why don't we consider what was done for the CDI and BV TCKs.
> > Annotations[1] are used to define the test type.
> > If it is a spec assertion, the spec reference is clearly identified:
> >
> > @SpecAssertion(section = "2.1.1", id = "c")
> >
> > or you could annotate the Test with @IntegrationTest which could be
> > used to denote that AS-specific test.
> >
> > Test (not code) coverage reports can be generated to identify what
> > the coverage is for each component. The report
> >   provides links to fisheye and the test code in svn.
> >
> >
> > [1]http://community.jboss.org/wiki/BeanValidationTCK#What_do_all_the_annotations_mean
> > Other references:
> > http://docs.jboss.org/hibernate/stable/beanvalidation/tck/reference/html_single/#d0e594
> > http://anonsvn.jboss.org/repos/weld/cdi-tck/trunk/impl/src/main/java/org/jboss/jsr299/tck/tests/context/conversation/ClientConversationContextTest.java
> >
> >
> > Shelly McGowan
> > JBoss, by Red Hat
> >
> > ----- Original Message -----
> > From: "Andrew Lee Rubinger"<andrew.rubinger at redhat.com>
> > To: jboss-as7-dev at lists.jboss.org
> > Sent: Saturday, March 19, 2011 5:34:30 PM
> > Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7
> > TestSuite
> >
> > It occurs to me I might not have been clear about the JIRA-per-test
> > thing.
> >
> > Say you're working on Feature X. You might commit the feature and
> > the
> > test alongside one another, citing the same JIRA, with the commit
> > message:
> >
> > "[JBAS-XXX] Implement Feature X and tests".
> >
> > That's great. No separate issues needed for the feature and the test
> > each.
> >
> > What's no good is going in and adding a bunch of tests for EJB, Web,
> > etc
> > while we pump up our coverage, alongside a commit message:
> >
> > "[JBAS-XXX] Add spec coverage to the testsuite"
> >
> > ...that tells us nothing about each test being made, just that we
> > dumped
> > 'em in there. Which is obvious, because they're there.
> >
> > S,
> > ALR
> >
> > On 03/19/2011 05:25 PM, Andrew Lee Rubinger wrote:
> >> Inline.
> >>
> >> On 03/19/2011 03:19 PM, Jason Greene wrote:
> >>> I really like the idea of our primary test suite covering API
> >>> interaction and your typical unit test which is tightly coupled to
> >>> internals. I also like your idea to separate spec Apis from JBoss
> >>> aAPIs.
> >>>
> >>> However, I disagree on the Jira per test requirement. This is only
> >>> because I worked with a similar model (all jiras require a test),
> >>> and you ended up with lots of duplicate tests, and you always had
> >>> to read Jira to understand what was being tested. Also, the tests
> >>> ended up being too specific, typically an exact replication of the
> >>> reporter's workflow.
> >>
> >> Consider this: If you had to read JIRA to understand what was being
> >> tested, that's a clear violation of point 3): Document the test.
> >> And
> >> failing good documentation, with no JIRA to turn to, you're flying
> >> blind.
> >>
> >> Also, how exactly does registering a JIRA encourage test
> >> duplication?
> >> Whether the tests are in an issue tracker or not seems to be
> >> unrelated
> >> to folks writing duplicate tests.
> >>>
> >>> IMO a better way to organize it would be by functional area, with
> >>> generous descriptions. All spec tests should reference spec
> >>> sections, for example. If we have a Jira that demonstrates lack of
> >>> coverage, we should translate the problem into meaningful and
> >>> reusable tests for the gaps we missed. Those tests should
> >>> primarily refer to the API contract and not the Jira. Although
> >>> commenting a list of jiras for extra info may be useful provided
> >>> the Jira doesn't confuse what the correct behavior should be.
> >>
> >> I'm a fan of organizing by functional areas and spec references.
> >> We'd
> >> done that in EJB3 with success: some packages were even named for
> >> the
> >> spec. Regardless, every commit had a JIRA association.
> >>
> >> In the end, so long as it's perfectly clear what we're testing and
> >> why,
> >> I'll have no objections. But I do believe that JIRA is the obvious
> >> answer to that question, and requires minimal effort to make a new
> >> issue. Especially since every commit is isolated to one concern and
> >> linked back to JIRA anyway, right? ;)
> >>
> >> S,
> >> ALR
> >>
> >>>
> >>> Sent from my iPad
> >>>
> >>> On Mar 18, 2011, at 2:21 AM, Andrew Lee
> >>> Rubinger<andrew.rubinger at redhat.com> wrote:
> >>>
> >>>> Looks like a lot of us have different ideas for what the AS7
> >>>> Integration
> >>>> TestSuite should consist of, so I'll kickoff with what I believe
> >>>> is the
> >>>> first design proposal towards getting coverage focused on the
> >>>> end-user
> >>>> (not certifying our own internals).
> >>>>
> >>>> I suspect this breaks down into two categories, which may be
> >>>> modelled as
> >>>> separate modules under the existing "testsuite" aggregator
> >>>> parent:
> >>>>
> >>>> * Specification
> >>>> * AS-specific APIs
> >>>>
> >>>> This isn't difficult work, though I do think it's important we
> >>>> consider
> >>>> some hard rules. IMO we should be developing these suites as if
> >>>> we were
> >>>> application developers, not wearing our server dev hats.
> >>>>
> >>>> ----------------
> >>>>
> >>>> [End Goal]
> >>>>
> >>>> 1) No compile-time dependencies in the module except for what's
> >>>> absolutely necessary.
> >>>>
> >>>> For the spec suite, this means: JDK and EE Spec APIs only in the
> >>>> compilation classpath. Testable asset sources and resources (ie.
> >>>> EJBs,
> >>>> Servlets, etc) would live under src/main/* to enforce that. Only
> >>>> the
> >>>> tests themselves would be located under "src/test/*".
> >>>>
> >>>> The AS-specific API suite may also add in our own APIs to the
> >>>> compilation classpath, but the line should end there. In "test"
> >>>> scope
> >>>> we can place all runtime dependencies.
> >>>>
> >>>> For the specification suite, AS-specific grammars like our own
> >>>> deployment descriptors are fine; these are in many ways
> >>>> equivalent to
> >>>> the TCK porting layer. We're not building a TCK; we're showing
> >>>> that our
> >>>> implementation supports the features advertised.
> >>>>
> >>>> 2) Every single new test created is to have an associated JIRA.
> >>>>
> >>>> We all remember the nightmare it was when the old AS4-6 suite
> >>>> would fall
> >>>> down. We'd comb through each test, at times trying to determine
> >>>> its
> >>>> purpose. By linking to JIRA we get history of intent, which acts
> >>>> as a
> >>>> nice record even in the case that the test isn't so
> >>>> well-documented.
> >>>> I'd argue that tests are a bigger asset than our code, and we
> >>>> should be
> >>>> thinking about these in terms of long-term maintenance to outlive
> >>>> any
> >>>> specific impl.
> >>>>
> >>>> 3) Documentation
> >>>>
> >>>> Alongside the JIRA reference, a quick note about we're looking to
> >>>> accomplish is something I find very helpful. I don't personally
> >>>> buy the
> >>>> argument that code is self-documenting if written well. It gets
> >>>> refactored and stale over time.
> >>>>
> >>>> 4) Run-mode profiles
> >>>>
> >>>> Arquillian provides a wonderful abstraction such that we can get
> >>>> coverage for AS in both remote managed *and* embedded modes
> >>>> without
> >>>> changing the test itself. To certify that everything is working
> >>>> as
> >>>> advertised no matter the runtime, we should be able to run the
> >>>> same
> >>>> suite in standalone, domain, and embedded modes (generally
> >>>> speaking).
> >>>>
> >>>> 5) Porting of AS6 Tests
> >>>>
> >>>> There's no discounting the value this coverage has given us,
> >>>> though I
> >>>> question the purpose of a lot of these tests. I think a great
> >>>> majority
> >>>> of these need to come into the new codebase, refactored to align
> >>>> if needed.
> >>>>
> >>>> ----------------
> >>>>
> >>>> [Current State]
> >>>>
> >>>> Here[1] is an example of what I believe to be a simple,
> >>>> well-written
> >>>> test, with the exception that the tested Servlet and EJB are in
> >>>> the same
> >>>> test source folder.
> >>>>
> >>>> The current "testsuite" aggregator contains modules which mix our
> >>>> end-user certification stuff alongside internals, so I think
> >>>> these
> >>>> should be separated out.
> >>>>
> >>>> A lot of this is set up in some fashion already, but I would like
> >>>> to see us:
> >>>>
> >>>> 1) Agree upon a strict scope for each type of testsuite along the
> >>>> lines
> >>>> of my points above, once we reach agreement
> >>>> 2) Upgrade to ARQ 1.0.0.Alpha5 (which implies ShrinkWrap
> >>>> 1.0.0-alpha-12), just released tonight. Currently AS is on a
> >>>> forked
> >>>> release of ARQ for OSGi purposes, and these changes, if
> >>>> necessary, need
> >>>> to get upstream so we can do upgrades.
> >>>>
> >>>> It's clear that AS7 has made full-steam-ahead progress since last
> >>>> summer, and with a little organization our testsuite can give us
> >>>> a great
> >>>> view of where we stand, from an end-user's perspective, with
> >>>> minimal
> >>>> investment.
> >>>>
> >>>> S,
> >>>> ALR
> >>>>
> >>>>
> >>>> [1]
> >>>> https://github.com/jbossas/jboss-as/blob/master/testsuite/integration/src/test/java/org/jboss/as/testsuite/integration/webejb/ServletInjectionTestCase.java
> >>>> _______________________________________________
> >>>> jboss-as7-dev mailing list
> >>>> jboss-as7-dev at lists.jboss.org
> >>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
> >>>
> >>> _______________________________________________
> >>> jboss-as7-dev mailing list
> >>> jboss-as7-dev at lists.jboss.org
> >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
> >> _______________________________________________
> >> jboss-as7-dev mailing list
> >> jboss-as7-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
> > _______________________________________________
> > jboss-as7-dev mailing list
> > jboss-as7-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev