[webbeans-dev] TCK - grouping broken etc. tests

[Pete Muir]

Hi all,

I just spent some time working through the tests marked as ri-broken  
tests in the TCK. I've fixed those I can easily, and filed WBRI JIRAs  
for the rest. It made me realize we need some guidelines on how to  
handle tests which don't work:

http://www.seamframework.org/WebBeans/GroupingTests

* If the test isn't working properly (e.g. you haven't finished it,  
you need to check it in to get advice from others, you find a test and  
can see it is broken, either due to changes in the spec, or due to a  
bad test in the first place, you need to add it to the "broken" group  
- @Test (groups={"broken"}). Please also add a note saying what is  
wrong with the test, and if you've spent some time with it, note where  
you got to with understanding/fixing it. Broken tests are highlighted  
in the audit in yellow. If you are working on a section, always check  
to see if you can't fix the broken tests. Alternatively, if you've  
written a test which will work standalone, but won't work in the  
container, add it to the incontainer-broken group

* If you think the test is correct, but it won't run against the RI  
you need to:
    - add it to the ri-broken group - @Test groups={"broken"})
    - alternatively, if the test will pass standalone, but won't pass  
in JBoss AS (e.g. we've had this where serialization is involved),  
then add it to the incontainer-ri-broken group
    - check and see if there is already a matching WBRI issue, if  
there isn't create one
    - add a note to the issue with the FQN of the test method (use  
copy qualified name in eclipse on the method)
    - add a comment to the test indicating the WBRI issue number

* If you find a test that is correct, but could be more complete, add  
it to the rewrite group @Test(groups={"rewrite"})

* Do not use the stub group any more

* Feel free to tag tests using groups to mark what they relate to.  
Eventually, I would like to go through and make a consistent effort at  
this with a well defined list of tags, but this is very low priority.

* We will also use this approach to allow testing on an embeddable EJB  
3.1 container. If you are tagging a test that will definitely run on  
this container (for example, a JSF test won't run, a servlet test  
won't run) then please add the ejb-embeddable tag. Again, this is low  
priority for now

Pete