[jboss-as7-dev] Requirements and Design Proposal: AS7 TestSuite

Wed Mar 23 10:56:33 EDT 2011

Andrew,

To bring the thread up a level, I propose the following requirements:

- no hardcoding of configuration elements in the test code
  - currently many tests have localhost and java:/H2DS

- integration tests should run remote, not embedded

- provide an Abstract test class that can be extended so adding tests focus on test logic not test configuration

- can be run in standalone or domain (and cloud)

- have reporting mechanism so test coverage can be obtained based on query
  - ex:  what tests are available for lifecycle callbacks

- (as Darran said) require a test be added with a fix for each JIRA

- extensible for other stakeholders (QE) of the integration test suite

Shelly McGowan
JBoss, by Red Hat

----- Original Message -----
From: "Shelly McGowan" <smcgowan at redhat.com>
To: "Andrew Lee Rubinger" <andrew.rubinger at redhat.com>
Cc: "Aslak Knutsen" <aknutsen at redhat.com>, jboss-as7-dev at lists.jboss.org
Sent: Wednesday, March 23, 2011 7:36:58 AM
Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 TestSuite

>
> @Test(groups = { "web_profile" } )
>     public void testEJBLite throws Exception {}

What does @Test.groups represent?  Is that a proposal for our own 
extension?  JUnit doesn't have this property in @org.junit.Test, so if 
it's our own thing we should likely rename to something else...

http://kentbeck.github.com/junit/javadoc/latest/org/junit/Test.html

S,
ALR

This, as shown in [3]:

import org.testng.annotations.Test;

http://testng.org/javadoc/org/testng/annotations/Test.html

>
> Shelly McGowan
> JBoss, by Red Hat
>
> ----- Original Message -----
> From: "Aslak Knutsen"<aknutsen at redhat.com>
> To: "Andrew Lee Rubinger"<andrew.rubinger at redhat.com>
> Cc: jboss-as7-dev at lists.jboss.org
> Sent: Tuesday, March 22, 2011 10:38:08 AM
> Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 TestSuite
>
>>
>> Why @IntegrationTest? By simply being in the AS Integration Test
>> module, wouldn't that be self-explanatory?
>>
>
> The use of @IntegrationTest in CDI TCK is basically to separate tests that can run in Weld SE (mocked EE v) vs the ones that need a full EE container.
>
> Not sure the AS Suite needs that kind of separation.
>
> -aslak-
>
>> On 03/20/2011 10:58 AM, Shelly McGowan wrote:
>>>
>>>
>>>
>>> Why don't we consider what was done for the CDI and BV TCKs.
>>> Annotations[1] are used to define the test type.
>>> If it is a spec assertion, the spec reference is clearly identified:
>>>
>>> @SpecAssertion(section = "2.1.1", id = "c")
>>>
>>> or you could annotate the Test with @IntegrationTest which could be
>>> used to denote that AS-specific test.
>>>
>>> Test (not code) coverage reports can be generated to identify what
>>> the coverage is for each component. The report
>>>    provides links to fisheye and the test code in svn.
>>>
>>>
>>> [1]http://community.jboss.org/wiki/BeanValidationTCK#What_do_all_the_annotations_mean
>>> Other references:
>>> [2]http://docs.jboss.org/hibernate/stable/beanvalidation/tck/reference/html_single/#d0e594
>>> [3]http://anonsvn.jboss.org/repos/weld/cdi-tck/trunk/impl/src/main/java/org/jboss/jsr299/tck/tests/context/conversation/ClientConversationContextTest.java
>>>
>>>
>>> Shelly McGowan
>>> JBoss, by Red Hat
>>>
>>> ----- Original Message -----
>>> From: "Andrew Lee Rubinger"<andrew.rubinger at redhat.com>
>>> To: jboss-as7-dev at lists.jboss.org
>>> Sent: Saturday, March 19, 2011 5:34:30 PM
>>> Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7
>>> TestSuite
>>>
>>> It occurs to me I might not have been clear about the JIRA-per-test
>>> thing.
>>>
>>> Say you're working on Feature X. You might commit the feature and
>>> the
>>> test alongside one another, citing the same JIRA, with the commit
>>> message:
>>>
>>> "[JBAS-XXX] Implement Feature X and tests".
>>>
>>> That's great. No separate issues needed for the feature and the test
>>> each.
>>>
>>> What's no good is going in and adding a bunch of tests for EJB, Web,
>>> etc
>>> while we pump up our coverage, alongside a commit message:
>>>
>>> "[JBAS-XXX] Add spec coverage to the testsuite"
>>>
>>> ...that tells us nothing about each test being made, just that we
>>> dumped
>>> 'em in there. Which is obvious, because they're there.
>>>
>>> S,
>>> ALR
>>>
>>> On 03/19/2011 05:25 PM, Andrew Lee Rubinger wrote:
>>>> Inline.
>>>>
>>>> On 03/19/2011 03:19 PM, Jason Greene wrote:
>>>>> I really like the idea of our primary test suite covering API
>>>>> interaction and your typical unit test which is tightly coupled to
>>>>> internals. I also like your idea to separate spec Apis from JBoss
>>>>> aAPIs.
>>>>>
>>>>> However, I disagree on the Jira per test requirement. This is only
>>>>> because I worked with a similar model (all jiras require a test),
>>>>> and you ended up with lots of duplicate tests, and you always had
>>>>> to read Jira to understand what was being tested. Also, the tests
>>>>> ended up being too specific, typically an exact replication of the
>>>>> reporter's workflow.
>>>>
>>>> Consider this: If you had to read JIRA to understand what was being
>>>> tested, that's a clear violation of point 3): Document the test.
>>>> And
>>>> failing good documentation, with no JIRA to turn to, you're flying
>>>> blind.
>>>>
>>>> Also, how exactly does registering a JIRA encourage test
>>>> duplication?
>>>> Whether the tests are in an issue tracker or not seems to be
>>>> unrelated
>>>> to folks writing duplicate tests.
>>>>>
>>>>> IMO a better way to organize it would be by functional area, with
>>>>> generous descriptions. All spec tests should reference spec
>>>>> sections, for example. If we have a Jira that demonstrates lack of
>>>>> coverage, we should translate the problem into meaningful and
>>>>> reusable tests for the gaps we missed. Those tests should
>>>>> primarily refer to the API contract and not the Jira. Although
>>>>> commenting a list of jiras for extra info may be useful provided
>>>>> the Jira doesn't confuse what the correct behavior should be.
>>>>
>>>> I'm a fan of organizing by functional areas and spec references.
>>>> We'd
>>>> done that in EJB3 with success: some packages were even named for
>>>> the
>>>> spec. Regardless, every commit had a JIRA association.
>>>>
>>>> In the end, so long as it's perfectly clear what we're testing and
>>>> why,
>>>> I'll have no objections. But I do believe that JIRA is the obvious
>>>> answer to that question, and requires minimal effort to make a new
>>>> issue. Especially since every commit is isolated to one concern and
>>>> linked back to JIRA anyway, right? ;)
>>>>
>>>> S,
>>>> ALR
>>>>
>>>>>
>>>>> Sent from my iPad
>>>>>
>>>>> On Mar 18, 2011, at 2:21 AM, Andrew Lee
>>>>> Rubinger<andrew.rubinger at redhat.com>  wrote:
>>>>>
>>>>>> Looks like a lot of us have different ideas for what the AS7
>>>>>> Integration
>>>>>> TestSuite should consist of, so I'll kickoff with what I believe
>>>>>> is the
>>>>>> first design proposal towards getting coverage focused on the
>>>>>> end-user
>>>>>> (not certifying our own internals).
>>>>>>
>>>>>> I suspect this breaks down into two categories, which may be
>>>>>> modelled as
>>>>>> separate modules under the existing "testsuite" aggregator
>>>>>> parent:
>>>>>>
>>>>>> * Specification
>>>>>> * AS-specific APIs
>>>>>>
>>>>>> This isn't difficult work, though I do think it's important we
>>>>>> consider
>>>>>> some hard rules. IMO we should be developing these suites as if
>>>>>> we were
>>>>>> application developers, not wearing our server dev hats.
>>>>>>
>>>>>> ----------------
>>>>>>
>>>>>> [End Goal]
>>>>>>
>>>>>> 1) No compile-time dependencies in the module except for what's
>>>>>> absolutely necessary.
>>>>>>
>>>>>> For the spec suite, this means: JDK and EE Spec APIs only in the
>>>>>> compilation classpath. Testable asset sources and resources (ie.
>>>>>> EJBs,
>>>>>> Servlets, etc) would live under src/main/* to enforce that. Only
>>>>>> the
>>>>>> tests themselves would be located under "src/test/*".
>>>>>>
>>>>>> The AS-specific API suite may also add in our own APIs to the
>>>>>> compilation classpath, but the line should end there. In "test"
>>>>>> scope
>>>>>> we can place all runtime dependencies.
>>>>>>
>>>>>> For the specification suite, AS-specific grammars like our own
>>>>>> deployment descriptors are fine; these are in many ways
>>>>>> equivalent to
>>>>>> the TCK porting layer. We're not building a TCK; we're showing
>>>>>> that our
>>>>>> implementation supports the features advertised.
>>>>>>
>>>>>> 2) Every single new test created is to have an associated JIRA.
>>>>>>
>>>>>> We all remember the nightmare it was when the old AS4-6 suite
>>>>>> would fall
>>>>>> down. We'd comb through each test, at times trying to determine
>>>>>> its
>>>>>> purpose. By linking to JIRA we get history of intent, which acts
>>>>>> as a
>>>>>> nice record even in the case that the test isn't so
>>>>>> well-documented.
>>>>>> I'd argue that tests are a bigger asset than our code, and we
>>>>>> should be
>>>>>> thinking about these in terms of long-term maintenance to outlive
>>>>>> any
>>>>>> specific impl.
>>>>>>
>>>>>> 3) Documentation
>>>>>>
>>>>>> Alongside the JIRA reference, a quick note about we're looking to
>>>>>> accomplish is something I find very helpful. I don't personally
>>>>>> buy the
>>>>>> argument that code is self-documenting if written well. It gets
>>>>>> refactored and stale over time.
>>>>>>
>>>>>> 4) Run-mode profiles
>>>>>>
>>>>>> Arquillian provides a wonderful abstraction such that we can get
>>>>>> coverage for AS in both remote managed *and* embedded modes
>>>>>> without
>>>>>> changing the test itself. To certify that everything is working
>>>>>> as
>>>>>> advertised no matter the runtime, we should be able to run the
>>>>>> same
>>>>>> suite in standalone, domain, and embedded modes (generally
>>>>>> speaking).
>>>>>>
>>>>>> 5) Porting of AS6 Tests
>>>>>>
>>>>>> There's no discounting the value this coverage has given us,
>>>>>> though I
>>>>>> question the purpose of a lot of these tests. I think a great
>>>>>> majority
>>>>>> of these need to come into the new codebase, refactored to align
>>>>>> if needed.
>>>>>>
>>>>>> ----------------
>>>>>>
>>>>>> [Current State]
>>>>>>
>>>>>> Here[1] is an example of what I believe to be a simple,
>>>>>> well-written
>>>>>> test, with the exception that the tested Servlet and EJB are in
>>>>>> the same
>>>>>> test source folder.
>>>>>>
>>>>>> The current "testsuite" aggregator contains modules which mix our
>>>>>> end-user certification stuff alongside internals, so I think
>>>>>> these
>>>>>> should be separated out.
>>>>>>
>>>>>> A lot of this is set up in some fashion already, but I would like
>>>>>> to see us:
>>>>>>
>>>>>> 1) Agree upon a strict scope for each type of testsuite along the
>>>>>> lines
>>>>>> of my points above, once we reach agreement
>>>>>> 2) Upgrade to ARQ 1.0.0.Alpha5 (which implies ShrinkWrap
>>>>>> 1.0.0-alpha-12), just released tonight. Currently AS is on a
>>>>>> forked
>>>>>> release of ARQ for OSGi purposes, and these changes, if
>>>>>> necessary, need
>>>>>> to get upstream so we can do upgrades.
>>>>>>
>>>>>> It's clear that AS7 has made full-steam-ahead progress since last
>>>>>> summer, and with a little organization our testsuite can give us
>>>>>> a great
>>>>>> view of where we stand, from an end-user's perspective, with
>>>>>> minimal
>>>>>> investment.
>>>>>>
>>>>>> S,
>>>>>> ALR
>>>>>>
>>>>>>
>>>>>> [1]
>>>>>> https://github.com/jbossas/jboss-as/blob/master/testsuite/integration/src/test/java/org/jboss/as/testsuite/integration/webejb/ServletInjectionTestCase.java
>>>>>> _______________________________________________
>>>>>> jboss-as7-dev mailing list
>>>>>> jboss-as7-dev at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>
>>>>> _______________________________________________
>>>>> jboss-as7-dev mailing list
>>>>> jboss-as7-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>> _______________________________________________
>>>> jboss-as7-dev mailing list
>>>> jboss-as7-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> jboss-as7-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>> _______________________________________________
>> jboss-as7-dev mailing list
>> jboss-as7-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev