[webbeans-dev] Granularity of assertions in TCK

[Pete Muir]

Hi

Since the TCK audit has been rebuilt recently, we've lost a lot of the  
granularity of the report. As we go through checking it we need to  
make sure to reintroduce that granularity.

Here are a few guidelines (of course, sometimes they don't hold):

* A testcase should verify one thing
    - in other words don't check two distinct thing in the same testcase
    - it may well be the case that the spec asserts the same thing  
multiple times (in different places), then it is fine to reuse the  
testcase
    - it may well be the case that a single check proves multiple  
things (for example the fact that a bean resolves could be checking  
type resolution rules, binding resolution rules etc.)
    - use common sense - if a test seems to be checking > 1thing,  
split it up

* An assertion should be fine grained
    - if an assertion is being checked in multiple testcases (as  
defined above), it's often an indication that the assertion isn't fine  
grained enough
    - sometimes multiple testcases are needed, but you should think  
hard about whether you can split the assertion up (use tools like the  
highlighter (_highlight_) and strikethrough(~strikethrough~) to split  
the assertion up)
    - if an assertion has something like "X and Y must through a  
definition error if P, Q or R" then you should consider splitting this  
into 2 * 3 = 6 assertions so that a test can be assigned to each one

if you need to split an assertions, takes it current ID and add an  
extra letter on the end.

If you are wondering why these guidelines - simply that it makes  
getting an overview of the TCK easily - we can scan through and check:

* how well covered a section is (whether or not it uses dense text to  
define something)
* compare how dense a section is
* view easily how covered a section is
* easily check that a test is relevant to an assertion.