[jboss-as7-dev] Requirements and Design Proposal: AS7 TestSuite

Pete Muir pmuir at redhat.com
Wed Mar 23 08:52:44 EDT 2011 

I think you should define more carefully what you are proposing, then people might agree more...

* if you write a regression test, it must have a linked JIRA OR it must reference a forum post/mailing list archive (you might have written a test to prove that there isn't a problem to a user, but silly to "waste" that test...)
* if you write a test for a new feature, and the feature has a JIRA issue, then you should link it. New feature development doesn't always work well with an issue tracker IMO.

You also need to think about linkage. Each test should have a comment

/**
 * Testing blah...
 *
 * JIRA: JBAS-1234
 */

and all bugs should also give the FQMN of the associated tests - this way you can go from one to the other very easily.

We've used this scheme a lot with Weld and the CDI TCK and I find it works very well, especially 9 months later when you come back to an issue and can't remember what I did when.


On 23 Mar 2011, at 06:10, Andrew Lee Rubinger wrote:

> I'm still confused by the general hesitation towards a JIRA association 
> for each test.  To me this is almost a redundant policy; since when are 
> we committing *anything* without linking back to an issue?
> 
> Comments inline.
> 
> On 03/22/2011 12:11 PM, Shelly McGowan wrote:
>> I realize the intent of @IntegrationTest annotation for the CDI TCK.  It was just an example in response to the comments about:
>> 
>> Jason: IMO a better way to organize it would be by functional area, with generous descriptions. All spec tests should reference spec
>> sections, for example.
>> 
>> ALR: I'm a fan of organizing by functional areas and spec references.
>> 
>> therefore proposing to use annotations for such organization instead of JIRA.  We can devise our own annotations that make sense.
> 
> OK, what tooling do you envision would make use of this metadata?  Or is 
> it just there as a documentation marker for us?
> 
> To me, annotations are to be consumed/processed/enforced by software; 
> comments are to be read by developers.
> 
>> Another consideration is because they are integration tests, it is common that more than one
>> technology is used in the test cases.    Having tests organized by directory structure does not always reflect that.
>> 
>> integration/src/test/java/org/jboss/as/testsuite/integration/jms/JMSTestCase.java
>> 
>> These tests could contain test cases that may also cover MDB, XA, JCA.  So as in [3] below, we could do something like:
>> 
>> @Test(groups = { "JMS", "XA", "MDB"})
>>   public void testJms() throws Exception {}
>> 
>> or
>> 
>> @Test(groups = { "web_profile" } )
>>   public void testEJBLite throws Exception {}
> 
> What does @Test.groups represent?  Is that a proposal for our own 
> extension?  JUnit doesn't have this property in @org.junit.Test, so if 
> it's our own thing we should likely rename to something else...
> 
> http://kentbeck.github.com/junit/javadoc/latest/org/junit/Test.html
> 
> S,
> ALR
> 
>> 
>> Shelly McGowan
>> JBoss, by Red Hat
>> 
>> ----- Original Message -----
>> From: "Aslak Knutsen"<aknutsen at redhat.com>
>> To: "Andrew Lee Rubinger"<andrew.rubinger at redhat.com>
>> Cc: jboss-as7-dev at lists.jboss.org
>> Sent: Tuesday, March 22, 2011 10:38:08 AM
>> Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7 TestSuite
>> 
>>> 
>>> Why @IntegrationTest? By simply being in the AS Integration Test
>>> module, wouldn't that be self-explanatory?
>>> 
>> 
>> The use of @IntegrationTest in CDI TCK is basically to separate tests that can run in Weld SE (mocked EE v) vs the ones that need a full EE container.
>> 
>> Not sure the AS Suite needs that kind of separation.
>> 
>> -aslak-
>> 
>>> On 03/20/2011 10:58 AM, Shelly McGowan wrote:
>>>> 
>>>> 
>>>> 
>>>> Why don't we consider what was done for the CDI and BV TCKs.
>>>> Annotations[1] are used to define the test type.
>>>> If it is a spec assertion, the spec reference is clearly identified:
>>>> 
>>>> @SpecAssertion(section = "2.1.1", id = "c")
>>>> 
>>>> or you could annotate the Test with @IntegrationTest which could be
>>>> used to denote that AS-specific test.
>>>> 
>>>> Test (not code) coverage reports can be generated to identify what
>>>> the coverage is for each component. The report
>>>>  provides links to fisheye and the test code in svn.
>>>> 
>>>> 
>>>> [1]http://community.jboss.org/wiki/BeanValidationTCK#What_do_all_the_annotations_mean
>>>> Other references:
>>>> [2]http://docs.jboss.org/hibernate/stable/beanvalidation/tck/reference/html_single/#d0e594
>>>> [3]http://anonsvn.jboss.org/repos/weld/cdi-tck/trunk/impl/src/main/java/org/jboss/jsr299/tck/tests/context/conversation/ClientConversationContextTest.java
>>>> 
>>>> 
>>>> Shelly McGowan
>>>> JBoss, by Red Hat
>>>> 
>>>> ----- Original Message -----
>>>> From: "Andrew Lee Rubinger"<andrew.rubinger at redhat.com>
>>>> To: jboss-as7-dev at lists.jboss.org
>>>> Sent: Saturday, March 19, 2011 5:34:30 PM
>>>> Subject: Re: [jboss-as7-dev] Requirements and Design Proposal: AS7
>>>> TestSuite
>>>> 
>>>> It occurs to me I might not have been clear about the JIRA-per-test
>>>> thing.
>>>> 
>>>> Say you're working on Feature X. You might commit the feature and
>>>> the
>>>> test alongside one another, citing the same JIRA, with the commit
>>>> message:
>>>> 
>>>> "[JBAS-XXX] Implement Feature X and tests".
>>>> 
>>>> That's great. No separate issues needed for the feature and the test
>>>> each.
>>>> 
>>>> What's no good is going in and adding a bunch of tests for EJB, Web,
>>>> etc
>>>> while we pump up our coverage, alongside a commit message:
>>>> 
>>>> "[JBAS-XXX] Add spec coverage to the testsuite"
>>>> 
>>>> ...that tells us nothing about each test being made, just that we
>>>> dumped
>>>> 'em in there. Which is obvious, because they're there.
>>>> 
>>>> S,
>>>> ALR
>>>> 
>>>> On 03/19/2011 05:25 PM, Andrew Lee Rubinger wrote:
>>>>> Inline.
>>>>> 
>>>>> On 03/19/2011 03:19 PM, Jason Greene wrote:
>>>>>> I really like the idea of our primary test suite covering API
>>>>>> interaction and your typical unit test which is tightly coupled to
>>>>>> internals. I also like your idea to separate spec Apis from JBoss
>>>>>> aAPIs.
>>>>>> 
>>>>>> However, I disagree on the Jira per test requirement. This is only
>>>>>> because I worked with a similar model (all jiras require a test),
>>>>>> and you ended up with lots of duplicate tests, and you always had
>>>>>> to read Jira to understand what was being tested. Also, the tests
>>>>>> ended up being too specific, typically an exact replication of the
>>>>>> reporter's workflow.
>>>>> 
>>>>> Consider this: If you had to read JIRA to understand what was being
>>>>> tested, that's a clear violation of point 3): Document the test.
>>>>> And
>>>>> failing good documentation, with no JIRA to turn to, you're flying
>>>>> blind.
>>>>> 
>>>>> Also, how exactly does registering a JIRA encourage test
>>>>> duplication?
>>>>> Whether the tests are in an issue tracker or not seems to be
>>>>> unrelated
>>>>> to folks writing duplicate tests.
>>>>>> 
>>>>>> IMO a better way to organize it would be by functional area, with
>>>>>> generous descriptions. All spec tests should reference spec
>>>>>> sections, for example. If we have a Jira that demonstrates lack of
>>>>>> coverage, we should translate the problem into meaningful and
>>>>>> reusable tests for the gaps we missed. Those tests should
>>>>>> primarily refer to the API contract and not the Jira. Although
>>>>>> commenting a list of jiras for extra info may be useful provided
>>>>>> the Jira doesn't confuse what the correct behavior should be.
>>>>> 
>>>>> I'm a fan of organizing by functional areas and spec references.
>>>>> We'd
>>>>> done that in EJB3 with success: some packages were even named for
>>>>> the
>>>>> spec. Regardless, every commit had a JIRA association.
>>>>> 
>>>>> In the end, so long as it's perfectly clear what we're testing and
>>>>> why,
>>>>> I'll have no objections. But I do believe that JIRA is the obvious
>>>>> answer to that question, and requires minimal effort to make a new
>>>>> issue. Especially since every commit is isolated to one concern and
>>>>> linked back to JIRA anyway, right? ;)
>>>>> 
>>>>> S,
>>>>> ALR
>>>>> 
>>>>>> 
>>>>>> Sent from my iPad
>>>>>> 
>>>>>> On Mar 18, 2011, at 2:21 AM, Andrew Lee
>>>>>> Rubinger<andrew.rubinger at redhat.com>  wrote:
>>>>>> 
>>>>>>> Looks like a lot of us have different ideas for what the AS7
>>>>>>> Integration
>>>>>>> TestSuite should consist of, so I'll kickoff with what I believe
>>>>>>> is the
>>>>>>> first design proposal towards getting coverage focused on the
>>>>>>> end-user
>>>>>>> (not certifying our own internals).
>>>>>>> 
>>>>>>> I suspect this breaks down into two categories, which may be
>>>>>>> modelled as
>>>>>>> separate modules under the existing "testsuite" aggregator
>>>>>>> parent:
>>>>>>> 
>>>>>>> * Specification
>>>>>>> * AS-specific APIs
>>>>>>> 
>>>>>>> This isn't difficult work, though I do think it's important we
>>>>>>> consider
>>>>>>> some hard rules. IMO we should be developing these suites as if
>>>>>>> we were
>>>>>>> application developers, not wearing our server dev hats.
>>>>>>> 
>>>>>>> ----------------
>>>>>>> 
>>>>>>> [End Goal]
>>>>>>> 
>>>>>>> 1) No compile-time dependencies in the module except for what's
>>>>>>> absolutely necessary.
>>>>>>> 
>>>>>>> For the spec suite, this means: JDK and EE Spec APIs only in the
>>>>>>> compilation classpath. Testable asset sources and resources (ie.
>>>>>>> EJBs,
>>>>>>> Servlets, etc) would live under src/main/* to enforce that. Only
>>>>>>> the
>>>>>>> tests themselves would be located under "src/test/*".
>>>>>>> 
>>>>>>> The AS-specific API suite may also add in our own APIs to the
>>>>>>> compilation classpath, but the line should end there. In "test"
>>>>>>> scope
>>>>>>> we can place all runtime dependencies.
>>>>>>> 
>>>>>>> For the specification suite, AS-specific grammars like our own
>>>>>>> deployment descriptors are fine; these are in many ways
>>>>>>> equivalent to
>>>>>>> the TCK porting layer. We're not building a TCK; we're showing
>>>>>>> that our
>>>>>>> implementation supports the features advertised.
>>>>>>> 
>>>>>>> 2) Every single new test created is to have an associated JIRA.
>>>>>>> 
>>>>>>> We all remember the nightmare it was when the old AS4-6 suite
>>>>>>> would fall
>>>>>>> down. We'd comb through each test, at times trying to determine
>>>>>>> its
>>>>>>> purpose. By linking to JIRA we get history of intent, which acts
>>>>>>> as a
>>>>>>> nice record even in the case that the test isn't so
>>>>>>> well-documented.
>>>>>>> I'd argue that tests are a bigger asset than our code, and we
>>>>>>> should be
>>>>>>> thinking about these in terms of long-term maintenance to outlive
>>>>>>> any
>>>>>>> specific impl.
>>>>>>> 
>>>>>>> 3) Documentation
>>>>>>> 
>>>>>>> Alongside the JIRA reference, a quick note about we're looking to
>>>>>>> accomplish is something I find very helpful. I don't personally
>>>>>>> buy the
>>>>>>> argument that code is self-documenting if written well. It gets
>>>>>>> refactored and stale over time.
>>>>>>> 
>>>>>>> 4) Run-mode profiles
>>>>>>> 
>>>>>>> Arquillian provides a wonderful abstraction such that we can get
>>>>>>> coverage for AS in both remote managed *and* embedded modes
>>>>>>> without
>>>>>>> changing the test itself. To certify that everything is working
>>>>>>> as
>>>>>>> advertised no matter the runtime, we should be able to run the
>>>>>>> same
>>>>>>> suite in standalone, domain, and embedded modes (generally
>>>>>>> speaking).
>>>>>>> 
>>>>>>> 5) Porting of AS6 Tests
>>>>>>> 
>>>>>>> There's no discounting the value this coverage has given us,
>>>>>>> though I
>>>>>>> question the purpose of a lot of these tests. I think a great
>>>>>>> majority
>>>>>>> of these need to come into the new codebase, refactored to align
>>>>>>> if needed.
>>>>>>> 
>>>>>>> ----------------
>>>>>>> 
>>>>>>> [Current State]
>>>>>>> 
>>>>>>> Here[1] is an example of what I believe to be a simple,
>>>>>>> well-written
>>>>>>> test, with the exception that the tested Servlet and EJB are in
>>>>>>> the same
>>>>>>> test source folder.
>>>>>>> 
>>>>>>> The current "testsuite" aggregator contains modules which mix our
>>>>>>> end-user certification stuff alongside internals, so I think
>>>>>>> these
>>>>>>> should be separated out.
>>>>>>> 
>>>>>>> A lot of this is set up in some fashion already, but I would like
>>>>>>> to see us:
>>>>>>> 
>>>>>>> 1) Agree upon a strict scope for each type of testsuite along the
>>>>>>> lines
>>>>>>> of my points above, once we reach agreement
>>>>>>> 2) Upgrade to ARQ 1.0.0.Alpha5 (which implies ShrinkWrap
>>>>>>> 1.0.0-alpha-12), just released tonight. Currently AS is on a
>>>>>>> forked
>>>>>>> release of ARQ for OSGi purposes, and these changes, if
>>>>>>> necessary, need
>>>>>>> to get upstream so we can do upgrades.
>>>>>>> 
>>>>>>> It's clear that AS7 has made full-steam-ahead progress since last
>>>>>>> summer, and with a little organization our testsuite can give us
>>>>>>> a great
>>>>>>> view of where we stand, from an end-user's perspective, with
>>>>>>> minimal
>>>>>>> investment.
>>>>>>> 
>>>>>>> S,
>>>>>>> ALR
>>>>>>> 
>>>>>>> 
>>>>>>> [1]
>>>>>>> https://github.com/jbossas/jboss-as/blob/master/testsuite/integration/src/test/java/org/jboss/as/testsuite/integration/webejb/ServletInjectionTestCase.java
>>>>>>> _______________________________________________
>>>>>>> jboss-as7-dev mailing list
>>>>>>> jboss-as7-dev at lists.jboss.org
>>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>> 
>>>>>> _______________________________________________
>>>>>> jboss-as7-dev mailing list
>>>>>> jboss-as7-dev at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>> _______________________________________________
>>>>> jboss-as7-dev mailing list
>>>>> jboss-as7-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>> _______________________________________________
>>>> jboss-as7-dev mailing list
>>>> jboss-as7-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> jboss-as7-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>> _______________________________________________
>> jboss-as7-dev mailing list
>> jboss-as7-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

More information about the jboss-as7-dev mailing list