[hibernate-commits] Hibernate SVN: r17578 - in beanvalidation/tck/trunk: src/main and 5 other directories.

Author: hardy.ferentschik Date: 2009-09-29 12:34:58 -0400 (Tue, 29 Sep 2009) New Revision: 17578 Added: beanvalidation/tck/trunk/src/main/docbook/ beanvalidation/tck/trunk/src/main/docbook/en-US/ beanvalidation/tck/trunk/src/main/docbook/en-US/Author_Group.xml beanvalidation/tck/trunk/src/main/docbook/en-US/Book_Info.xml beanvalidation/tck/trunk/src/main/docbook/en-US/Book_Preface.xml beanvalidation/tck/trunk/src/main/docbook/en-US/appeals-process.xml beanvalidation/tck/trunk/src/main/docbook/en-US/configuration.xml beanvalidation/tck/trunk/src/main/docbook/en-US/eclipse-debugging.xml beanvalidation/tck/trunk/src/main/docbook/en-US/eclipse-running.xml beanvalidation/tck/trunk/src/main/docbook/en-US/executing.xml beanvalidation/tck/trunk/src/main/docbook/en-US/harness/ beanvalidation/tck/trunk/src/main/docbook/en-US/harness/Author_Group.xml beanvalidation/tck/trunk/src/main/docbook/en-US/harness/Book_Info.xml beanvalidation/tck/trunk/src/main/docbook/en-US/harness/Book_Preface.xml beanvalidation/tck/trunk/src/main/docbook/en-US/harness/configuration.xml beanvalidation/tck/trunk/src/main/docbook/en-US/harness/executing.xml beanvalidation/tck/trunk/src/main/docbook/en-US/harness/images/ beanvalidation/tck/trunk/src/main/docbook/en-US/harness/images/in-container-execution.png beanvalidation/tck/trunk/src/main/docbook/en-US/harness/introduction.xml beanvalidation/tck/trunk/src/main/docbook/en-US/harness/master.xml beanvalidation/tck/trunk/src/main/docbook/en-US/images/ beanvalidation/tck/trunk/src/main/docbook/en-US/images/testng-emailable-report.png beanvalidation/tck/trunk/src/main/docbook/en-US/images/testng-plugin-report.png beanvalidation/tck/trunk/src/main/docbook/en-US/images/testng-suite-detail-report.png beanvalidation/tck/trunk/src/main/docbook/en-US/images/testng-summary-report.png beanvalidation/tck/trunk/src/main/docbook/en-US/installation.xml beanvalidation/tck/trunk/src/main/docbook/en-US/introduction.xml beanvalidation/tck/trunk/src/main/docbook/en-US/master.xml beanvalidation/tck/trunk/src/main/docbook/en-US/part-background.xml beanvalidation/tck/trunk/src/main/docbook/en-US/part-execution.xml beanvalidation/tck/trunk/src/main/docbook/en-US/part-test-harness.xml beanvalidation/tck/trunk/src/main/docbook/en-US/reporting.xml Modified: beanvalidation/tck/trunk/ beanvalidation/tck/trunk/pom.xml Log: Initial import of TCK doc. Taken from JSR-299 Property changes on: beanvalidation/tck/trunk ___________________________________________________________________ Name: svn:ignore + target Modified: beanvalidation/tck/trunk/pom.xml =================================================================== --- beanvalidation/tck/trunk/pom.xml 2009-09-29 11:56:52 UTC (rev 17577) +++ beanvalidation/tck/trunk/pom.xml 2009-09-29 16:34:58 UTC (rev 17578) @@ -73,7 +73,72 @@ </extension> </extensions> <plugins> + <plugin> + <groupId>org.jboss.maven.plugins</groupId> + <artifactId>maven-jdocbook-plugin</artifactId> + <version>2.2.0</version> + <extensions>true</extensions> + <dependencies> + <dependency> + <groupId>org.hibernate</groupId> + <artifactId>hibernate-jdocbook-style</artifactId> + <version>2.0.0</version> + <type>jdocbook-style</type> + </dependency> + </dependencies> + <configuration> + <sourceDocumentName>master.xml</sourceDocumentName> + <sourceDirectory>${basedir}/src/main/docbook</sourceDirectory> + <masterTranslation>en-US</masterTranslation> + <imageResource> + <directory>${basedir}/src/main/docbook/en-US/images</directory> + <directory>${basedir}/src/main/docbook/en-US/harness</directory> + </imageResource> + <formats> + <format> + <formatName>pdf</formatName> + <stylesheetResource>classpath:/xslt/org/hibernate/jdocbook/xslt/pdf.xsl</stylesheetResource> + <finalName>beanvalidation-tck.pdf</finalName> + </format> + <format> + <formatName>html_single</formatName> + <stylesheetResource>classpath:/xslt/org/hibernate/jdocbook/xslt/xhtml-single.xsl + </stylesheetResource> + <finalName>index.html</finalName> + </format> + <format> + <formatName>html</formatName> + <stylesheetResource>classpath:/xslt/org/hibernate/jdocbook/xslt/xhtml.xsl + </stylesheetResource> + <finalName>index.html</finalName> + </format> + </formats> + <options> + <xincludeSupported>true</xincludeSupported> + <xmlTransformerType>saxon</xmlTransformerType> + <!-- needed for uri-resolvers; can be ommitted if using 'current' uri scheme --> + <!-- could also locate the docbook dependency and inspect its version... --> + <docbookVersion>1.72.0</docbookVersion> + <localeSeparator>-</localeSeparator> + </options> + </configuration> + <executions> + <execution> + <id>make-doc</id> + <phase>site</phase> + <goals> + <goal>resources</goal> + <goal>generate</goal> + </goals> + </execution> + </executions> + </plugin> <plugin> + <groupId>org.jboss.maven.plugins</groupId> + <artifactId>maven-jdocbook-style-plugin</artifactId> + <version>2.0.0</version> + </plugin> + <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>build-helper-maven-plugin</artifactId> <version>1.2</version> Added: beanvalidation/tck/trunk/src/main/docbook/en-US/Author_Group.xml =================================================================== --- beanvalidation/tck/trunk/src/main/docbook/en-US/Author_Group.xml (rev 0) +++ beanvalidation/tck/trunk/src/main/docbook/en-US/Author_Group.xml 2009-09-29 16:34:58 UTC (rev 17578) @@ -0,0 +1,32 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE authorgroup PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [ ]> +<authorgroup> + <author> + <firstname>Gavin</firstname> + <surname>King</surname> + <affiliation> + <jobtitle>JSR-299: Contexts and Dependency Injection (CDI) for Java EE + specification lead</jobtitle> + <orgname>Red Hat Inc.</orgname> + </affiliation> + </author> + <author> + <firstname>Pete</firstname> + <surname>Muir</surname> + <affiliation> + <jobtitle>CDI TCK lead</jobtitle> + <orgname>Red Hat Inc.</orgname> + </affiliation> + </author> + <author> + <firstname>Dan</firstname> + <surname>Allen</surname> + <affiliation> + <jobtitle>CDI TCK developer</jobtitle> + <orgname>Red Hat Inc.</orgname> + </affiliation> + </author> +<!-- +vim: ts=3:sw=3:tw=80:set expandtab +--> +</authorgroup> Added: beanvalidation/tck/trunk/src/main/docbook/en-US/Book_Info.xml =================================================================== --- beanvalidation/tck/trunk/src/main/docbook/en-US/Book_Info.xml (rev 0) +++ beanvalidation/tck/trunk/src/main/docbook/en-US/Book_Info.xml 2009-09-29 16:34:58 UTC (rev 17578) @@ -0,0 +1,29 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE bookinfo PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" +"http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> +<bookinfo> + <releaseinfo><literallayout>Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License.</literallayout></releaseinfo> + + <title>Technology Compatibility Kit Reference Guide for JSR-303: Bean + Validation</title> + + <subtitle>Specification Lead: Red Hat Inc.</subtitle> + + <copyright> + <year>2009 </year> + + <holder>Red Hat Middleware, LCC.</holder> + </copyright> + + <!-- +vim: ts=3:sw=3:tw=80:set expandtab +--> +</bookinfo> Added: beanvalidation/tck/trunk/src/main/docbook/en-US/Book_Preface.xml =================================================================== --- beanvalidation/tck/trunk/src/main/docbook/en-US/Book_Preface.xml (rev 0) +++ beanvalidation/tck/trunk/src/main/docbook/en-US/Book_Preface.xml 2009-09-29 16:34:58 UTC (rev 17578) @@ -0,0 +1,122 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE preface PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" +"http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> +<preface id="book-preface"> + <title>Preface</title> + + <para>This guide describes how to download, install, configure, and run the + Technology Compatibility Kit (TCK) used to verify the compatibility of an + implementation of the JSR-303: Bean Validation specification.</para> + + <para>The Bean Validation TCK is built atop the JBoss Test Harness, a + portable and configurable automated test suite for authoring unit and + integration tests in a Java EE environment. The TCK 1.0.0 uses the JBoss + Test Harness version 1.0.0 to execute the test suite.</para> + + <para>The Bean Validation TCK is provided under the <ulink + url="http://www.apache.org/licenses/LICENSE-2.0">Apache Public License + 2.0</ulink>.</para> + + <section id="target-audience"> + <title>Who Should Use This Book</title> + + <para>This guide is for implementors of the Bean Validation specification + to assist in running the test suite that verifies the compatibility of + their implementation.</para> + </section> + + <section id="before-reading"> + <title>Before You Read This Book</title> + + <para>The Bean Validation TCK is based on the Bean Validation + specification 1.0 (JSR-303). Information about the specification, + including links to the specification documents, can be found on the <ulink + url="http://jcp.org/en/jsr/detail?id=299">JSR-303 JCP page</ulink>.</para> + + <para>Before running the tests in the Bean Validation TCK, read and become + familiar with the JBoss Test Harness Reference Guide (pending), which + describes how the test harness functions.</para> + </section> + + <section id="book-organization"> + <title>How This Book Is Organized</title> + + <para>If you are running the Bean Validation TCK for the first time, read + <xref linkend="introduction" /> and <xref + linkend="test-harness-introduction" /> completely for the necessary + background information about the TCK and the JBoss Test Harness, + respectively. Once you have reviewed that material, perform the steps + outlined in the remaining chapters.</para> + + <itemizedlist> + <listitem> + <para><xref linkend="introduction" /> gives an overview of the + principles that apply generally to all Technology Compatibility Kits + (TCKs), outlines the appeals process and describes the Bean Validation + TCK architecture and components. It also includes a broad overview of + how the TCK is executed and lists the platforms on which the TCK has + been tested and verified.</para> + </listitem> + + <listitem> + <para><xref linkend="appeals-process" /> explains the process to be + followed by an implementor should they wish to challenge any test in + the TCK.</para> + </listitem> + + <listitem> + <para><xref linkend="installation" /> explains where to obtain the + required software for the Bean Validation TCK and how to install it. + It covers both the primary TCK components as well as tools useful for + troubleshooting tests.</para> + </listitem> + + <listitem> + <para><xref linkend="configuration" /> details the configuration of + the JBoss Test Harness, how to create a TCK runner for the TCK test + suite and the mechanics of how an in-container test is + conducted.</para> + </listitem> + + <listitem> + <para><xref linkend="reporting" /> explains the test reports that are + generated by the TCK test suite and introduces the TCK audit report as + a tool for measuring the completeness of the TCK in testing the + JSR-303 specification and in understanding how testcases relate to the + specification.</para> + </listitem> + + <listitem> + <para><xref linkend="executing" /> documents how the TCK test suite is + executed. It covers both modes supported by the TCK, standalone and + in-container, and shows how to dump the generated test artifacts to + disk.</para> + </listitem> + + <listitem> + <para><xref linkend="eclipse-running" /> shows how to run individual + tests in Eclipse and advises the best way to setup your Eclipse + workspace for running the tests.</para> + </listitem> + + <listitem> + <para><xref linkend="eclipse-debugging" /> builds on <xref + linkend="eclipse-running" /> by detailing how to debug individual + tests in Eclipse.</para> + </listitem> + + <listitem> + <para><xref linkend="test-harness" /> includes excerpts from the JBoss + Test Harness Reference Guide. How to configure the JBoss Test Harness + as it relates to the Bean Validation TCK is presented in <xref + linkend="configuration" />. However, to aid in debugging or + configuring the TCK in your environment, you may want to read in more + detail how to use the JBoss Test Harness.</para> + </listitem> + </itemizedlist> + </section> + + <!-- +vim: ts=3:sw=3:tw=80:set expandtab +--> +</preface> Added: beanvalidation/tck/trunk/src/main/docbook/en-US/appeals-process.xml =================================================================== --- beanvalidation/tck/trunk/src/main/docbook/en-US/appeals-process.xml (rev 0) +++ beanvalidation/tck/trunk/src/main/docbook/en-US/appeals-process.xml 2009-09-29 16:34:58 UTC (rev 17578) @@ -0,0 +1,102 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" +"http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> +<chapter id="appeals-process"> + <title>Appeals Process</title> + + <para>While the Bean Validation TCK is rigourous about enforcing an + implementation's conformance to the JSR-303 specification, it's reasonable + to assume that an implementor may discover new and/or better ways to + validate the assertions. This chapter covers the appeals process, defined by + the Specification Lead, Red Hat Middleware LLC., which allows implementors + of the JSR-303 specification to challenge one or more tests defined by the + Bean Validation TCK.</para> + + <para>The appeals process identifies who can make challenges to the TCK, + what challenges to the TCK may be submitted, how these challenges are + submitted, how and by whom challenges are addressed and how accepted + challenges to the TCK are managed.</para> + + <para>Following the recent adoption of transparency in the JCP, implementors + are encouraged to make their appeals public, which this process facilitates. + The JCP community should recognize that issue reports are a central aspect + of any good software and it's only natural to point out shortcomings and + strive to make improvements. Despite this good faith, not all implementors + will be comfortable with a public appeals process. Instructions about how to + make a private appeal are therefore provided.</para> + + <section> + <title>Who can make challenges to the TCK?</title> + + <para>Any implementor may submit an appeal to challenge one or more tests + in the Bean Validation TCK. In fact, members of the JSR-303 Expert Group + (EG) encourage this level of participation.</para> + </section> + + <section> + <title>What challenges to the TCK may be submitted?</title> + + <para>Any test case (e.g., <literal>@Artifact</literal> class, + <literal>@Test</literal> method), test case configuration (e.g., + validation.xml), test entities, annotations and other resources may be + challenged by an appeal.</para> + + <para>What is generally not challengable are the assertions made by the + specification. The specification document is controlled by a separate + process and challenges to it should be handled through the JSR-303 EG by + sending an e-mail to <ulink + url="mailto:jsr299-comments@jcp.org">jsr303-comments@jcp.org</ulink>.</para> + </section> + + <section> + <title>How these challenges are submitted?</title> + + <para>To submit a challenge, a new issue should be created in the <ulink + url="https://jira.jboss.org/jira/browse/WBTCK">HV project</ulink> of the + JBoss JIRA using the Issue Type: Bug. The appellant should complete the + Summary, Component (TCK Appeal), Environment and Description Field only. + Any communication regarding the issue should be pursed in the comments of + the filed issue for accurate record.</para> + + <para>To submit an issue in the JBoss JIRA, you must have a (free) + JBoss.com member account. You can create a member account using the <ulink + url="http://www.jboss.org/index.html?op=checkage&amp;module=user... + registration</ulink>.</para> + + <para>If you wish to make a private challenge, you should follow the above + procedure, setting the Security Level to Private. Only the issue reporter, + TCK Project Lead and designates will be able to view the issue.</para> + </section> + + <section> + <title>How and by whom challenges are addressed?</title> + + <para>The challenges will be addressed in a timely fashion by the Bean + Validation TCK Project Lead, as designated by Specification Lead, Red Hat + Middleware LLC. or his/her designate. The appellant can also monitor the + process by following the issue report filed in the <ulink + url="https://jira.jboss.org/jira/browse/WBTCK">HV project</ulink> of the + JBoss JIRA.</para> + + <para>The current TCK Project Lead is listed on the <ulink type="" + url="https://jira.jboss.org/jira/browse/WBTCK" userlevel="">HV Project + Summary Page</ulink> on the JBoss JIRA.</para> + </section> + + <section> + <title>How accepted challenges to the TCK are managed?</title> + + <para>Accepted challenges will be acknowledged via the filed issue's + comment section. Communication between the Bean Validation TCK Project + Lead and the appellant will take place via the issue comments. The issue's + status will be set to "Resolved" when the TCK project lead believes the + issue to be resolved. The appellant should, within 30 days, either close + the issue if they agree, or reopen the issue if they do not believe the + issue to be resolved.</para> + + <para>Resolved issue not addressed for 30 days will be closed by the TCK + Project Lead. If the TCK Project Lead and appellant are unable to agree on + the issue resolution, it will be referred to the JSR-303 specification + lead or his/her designate.</para> + </section> +</chapter> Added: beanvalidation/tck/trunk/src/main/docbook/en-US/configuration.xml =================================================================== --- beanvalidation/tck/trunk/src/main/docbook/en-US/configuration.xml (rev 0) +++ beanvalidation/tck/trunk/src/main/docbook/en-US/configuration.xml 2009-09-29 16:34:58 UTC (rev 17578) @@ -0,0 +1,157 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" +"http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> +<chapter id="configuration"> + <title>Configuration</title> + + <para>This chapter lays out how to configure the TCK Harness by specifying + the SPI implementation classes, defining the target container connection + information, and various other switches. You then learn how to setup a TCK + runner project that executes the the TCK test suite, putting these settings + into practice. Finally, a detailed account of how the JBoss Test Harness + negotiates the execution of the tests in the container is given.</para> + + <para>This chapter does not discuss in detail how to use the TCK in + standalone mode. The JBoss Test Harness guide provides more on running in + standalone mode.</para> + + <section id="tck-harness-properties"> + <title>TCK Harness Properties</title> + + <para>The JBoss Test Harness allows the test suite to be launched in a + pluggable fashion. In order to execute the TCK, the JBoss Test Harness + must be configured by specifying implementations of the test launcher and + container APIs.</para> + + <para>System properties and/or the resource + <code>META-INF/jboss-test-harness.properties</code>, a Java properties + file, are used to configure the JBoss Test Harness. You can read more + about configuring the JBoss Test Harness in <xref + linkend="test-harness-properties" />.</para> + + <para>You should set the following properties:</para> + + <table frame="all"> + <title>Required JBoss Test Harness Configuration Properties</title> + + <tgroup cols="2"> + <colspec colname="property" colnum="1" colwidth="5*" /> + + <colspec colname="description" colnum="2" colwidth="2*" /> + + <thead> + <row> + <entry>Property = Required/Example Value</entry> + + <entry>Description</entry> + </row> + </thead> + + <tbody> + <row> + <entry><literal>org.jboss.testharness.libraryDirectory=/path/to/extra/libraries + </literal></entry> + + <entry>Directory containing extra JARs you want placed in artifact + library directory such as the porting package + implementation.</entry> + </row> + + <row> + <entry><literal>org.jboss.testharness.standalone=false</literal></entry> + + <entry>You must run the tests in-container to pass the TCK</entry> + </row> + + <row> + <entry><literal>org.jboss.testharness.runIntegrationTests=true + </literal></entry> + + <entry>You must run the integration tests to pass the TCK</entry> + </row> + + <row> + <entry><literal>org.jboss.testharness.spi.Containers=com.acme.AcmeContainer + </literal></entry> + + <entry>The container implementation for deploying and executing + in-container tests. See <xref + linkend="deployment-api-contributions" /></entry> + </row> + + <row> + <entry><literal>org.jboss.testharness.api.TestLauncher=org.jboss.testharness.impl.runner.servlet.ServletTestLauncher + </literal></entry> + + <entry>You should use the <literal>ServletTestLauncher</literal> + for Java EE 6 and Java EE 6 Web Profile.</entry> + </row> + </tbody> + </tgroup> + </table> + + <para>To run the full TCK you must additionally implement + <literal>org.jboss.testharness.spi.Containers</literal>, which handles + deploying the test artifact to the container. An implementations of this + API is already available for JBoss AS 5.1. Therefore, you only need to + implement this part of the porting package if you wish to use another + container.</para> + + <note id="deployment-api-contributions"> + <para>Red Hat Middleware LLC encourages CDI implementators to contribute + JBoss Test Harness Deployment API implementations for other containers + under the ASL license. Please contact the Bean Validation TCK + lead.</para> + </note> + </section> + + <section> + <title>Configuring TestNG to execute the TCK</title> + + <para>The JBoss Test Harness is built atop TestNG, and it's TestNG that is + responsible for selecting the tests to execute, the order of execution, + and reporting the results. Detailed TestNG documentation can be found at + <ulink + url="http://testng.org/doc/documentation-main.html">testng.o... + + <para>The <literal>tck-tests.xml</literal> artifact provided in the TCK + distribution must be run by TestNG 5.9 (described by the TestNG + documenation as "with a <literal>testng.xml</literal> file") unmodified + for an implementation to pass the TCK. This file also allows tests to be + excluded from a run:</para> + + <programlisting>&lt;suite name="JSR-299 TCK" verbose="2"&gt; + &lt;test name="JSR-299 TCK"&gt; + ... + &lt;classes&gt; + &lt;class name="org.jboss.jsr299.tck.tests.context.application.ApplicationContextTest"&gt; + &lt;methods&gt; + &lt;exclude name="testApplicationScopeActiveDuringServiceMethod"/&gt; + &lt;/methods&gt; + &lt;/class&gt; + &lt;/classes&gt; + ... + &lt;/test&gt; +&lt;/suite&gt;</programlisting> + + <para>TestNG provides extensive reporting information. Depending on the + build tool or IDE you use, the reporting will take a different format. + Please consult the TestNG documentation and the tool documentation for + more information.</para> + </section> + + <section> + <title>Configuring your build environment to execute the TCK</title> + + <para>It's beyond the scope of this guide to describe in how to set up + your build environment to run the TCK. The JBoss Test Harness guide + describes how Web Beans uses Maven 2 to execute the Bean Validation TCK. + See <xref linkend="test-suite-runner" />. The TestNG documentation + provides extensive information on launching TestNG using the Java, Ant, + Eclipse or IntellJ IDEA.</para> + </section> + + <!-- +vim: ts=3:sw=3:tw=80:set expandtab +--> +</chapter> Added: beanvalidation/tck/trunk/src/main/docbook/en-US/eclipse-debugging.xml =================================================================== --- beanvalidation/tck/trunk/src/main/docbook/en-US/eclipse-debugging.xml (rev 0) +++ beanvalidation/tck/trunk/src/main/docbook/en-US/eclipse-debugging.xml 2009-09-29 16:34:58 UTC (rev 17578) @@ -0,0 +1,146 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [ ]> +<chapter id="eclipse-debugging"> + <title>Debugging Tests in Eclipse</title> + <para> + This chapter explains how to debug standalone and integration tests from + the TCK test suite in Eclipse. You should be able to use the lessons + learned here to debug tests in an alternate IDE as well. + </para> + <section> + <title>Debugging a standalone test</title> + <para> + There is almost no difference in how you debug a standalone test + from how you run it. With the test class open in the Eclipse editor, + simply right click in the editor view and select Debug As &gt; TestNG + Test. Eclipse will stop at any breakpoints you set just like it would + with any other local debug process. + </para> + <para> + If you plan to step into a class in a Web Beans implementation (or any + other dependent library), you must ensure that the source is properly + associated with the library. Below are the steps to follow to associate + the source of Web Beans with the TestNG debug configuration: + </para> + <orderedlist> + <listitem> + <para> + Select the Run &gt; Debug Configurations... menu from the main + menubar + </para> + </listitem> + <listitem> + <para> + Select the name of the test class in the TestNG category + </para> + </listitem> + <listitem> + <para> + Select the Source tab + </para> + </listitem> + <listitem> + <para> + Click the Add... button on the right + </para> + </listitem> + <listitem> + <para> + Select Java Project + </para> + </listitem> + <listitem> + <para> + Check the project the contains the class you want to debug + (e.g., webbeans-core) + </para> + </listitem> + <listitem> + <para> + Click OK on the Project Selection window + </para> + </listitem> + <listitem> + <para> + Click Close on the Debug Configurations window + </para> + </listitem> + </orderedlist> + <para> + You'll have to complete those steps for any test class you are + debugging, though you only have to do it once (the debug configuration + hangs around indefinitely). + </para> + <para> + Again, running a test in standalone isn't enough to pass the TCK and + cannot be used to run or debug an integration test. Let's look at how + to debug a test running in the context of the container. + </para> + </section> + <section> + <title>Debugging an integration test</title> + <para> + In order to debug an integration test, or any test run using in-container + mode, the test must be configured to run in-container, as described in + <xref linkend="eclipse-in-container" />, + and you must attach the IDE debugger to the container. That puts the + debugger on both sides of the fence, so to speak. + </para> + <para> + Since setting up a test to run in-container has already been + covered, we'll look at how to attach the IDE debugger to the container, + and then move on launching the test in debug mode. + </para> + <section> + <title>Attaching the IDE debugger to the container</title> + <para> + There are two ways to attach the IDE debugger to the container. + You can either start the container in debug mode from within the + IDE, or you can attach the debugger over a socket connection to a + standalone container running with JPDA enabled. + </para> + <para> + The Eclipse Server Tools, a subproject of the Eclipse Web Tools + Project (WTP), has support for launching most major application + servers, including JBoss AS 5. However, if you are using JBoss AS, + you should consider using JBoss Tools instead, which offers tighter + integration with JBoss technologies. See either the + <ulink url="http://www.eclipse.org/webtools/server/server.php">Server Tools documentation</ulink> + or the + <ulink url="http://docs.jboss.org/tools/3.0.1.GA/en/as/html/index.html"... Tools documentation</ulink> + for how to setup a container and start it in debug mode. + </para> + <para> + See + <ulink + url="http://maverikpro.wordpress.com/2007/11/26/remote-debug-a-web-a... blog entry</ulink> + to learn how to start JBoss AS with JPDA enabled and how to get the + Eclipse debugger to connect to the remote process. + </para> + </section> + <section> + <title>Launching the test in the debugger</title> + <para> + Once Eclipse is debugging the container, you can set a breakpoint in + the test and debug it just like a standalone test. Let's give it a + try. + </para> + <para> + Open a test annotated with <literal>@IntegrationTest</literal> in + the Eclipse editor, right click in the editor view, and select Debug + As &gt; TestNG Test. This time when the IDE hits the breakpoint, it + halts the JVM thread of the container rather than the thread that + launched the test. + </para> + <para> + Remember that if you need to debug into dependent libraries, the + source code for those libraries will need to be registered with the + TestNG debug configuration as described in the first section in this + chapter. + </para> + </section> + </section> +<!-- +vim: ts=3:sw=3:tw=80:set expandtab +--> +</chapter> Added: beanvalidation/tck/trunk/src/main/docbook/en-US/eclipse-running.xml =================================================================== --- beanvalidation/tck/trunk/src/main/docbook/en-US/eclipse-running.xml (rev 0) +++ beanvalidation/tck/trunk/src/main/docbook/en-US/eclipse-running.xml 2009-09-29 16:34:58 UTC (rev 17578) @@ -0,0 +1,447 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [ ]> +<chapter id="eclipse-running"> + <title>Running Tests in Eclipse</title> + <para> + This chapter explains how to run individual tests using the Eclipse + TestNG plugin. It covers running non-integration tests in standalone mode + and integration tests (as well as non-integration tests) in in-container + mode. You should be able to use the lessons learned here to debug tests in + an alternate IDE as well. + </para> + <section> + <title>Leveraging Eclipse's plugin ecosystem</title> + <para> + Using an existing test harness (TestNG) allows the tests to be executed + and debugged in an Integrated Development Environment (IDE) using + available plugins. Using an IDE is also the easiest way to execute a + test class in isolation. + + </para> + <para> + The TCK can be executed in any IDE for which there is a TestNG plugin + available. Running a test from the CDI TCK test suite using the Eclipse + TestNG plugin is almost as simple as running any other TestNG test. You + can also use the plugin to debug a test, which is described in the next + chapter. + </para> + <para> + Before running a test from the TCK test suite in Eclipse, you must have + the Eclipse <ulink url="http://testng.org">TestNG plugin</ulink> and + either the m2eclipse plugin or an Eclipse project generated use the + Maven 2 Eclipse plugin (<literal>maven-eclipse-plugin</literal>). Refer + to <xref linkend="eclipse-plugins" /> for more information on these + plugins. + </para> + + <para> + With the m2eclipse plugin installed, Eclipse should recognize the + CDI TCK projects as valid Eclipse projects (or any Web Beans project + for that matter). Import them into the Eclipse workspace at this time. + You should also import the Web Beans projects if you want to debug into + that code, which is covered later. + </para> + <tip> + <para> + If you choose to use the Maven 2 Eclipse plugin + (<literal>maven-eclipse-plugin</literal>), you should execute the + plugin in both the tck and webbeans projects: + </para> + <programlisting><![CDATA[cd tck +mvn clean eclipse:clean eclipse:eclipse -DdownloadSources -DdownloadJavadocs +cd ../webbeans +mvn clean eclipse:clean eclipse:eclipse -DdownloadSources -DdownloadJavadocs]]></programlisting> + </tip> + </section> + <section> + <title>Readying the Eclipse workspace</title> + <para> + When setting up your Ecilpse workspace, we recommended creating three + workings sets: + </para> + <orderedlist> + <listitem> + <para> + <emphasis role="bold">Web Beans</emphasis> - Groups the CDI API + and the CDI RI (i.e., Web Beans) projects + </para> + </listitem> + <listitem> + <para> + <emphasis role="bold">CDI TCK</emphasis> - Groups the CDI TCK + API and the test suite projects + </para> + </listitem> + <listitem> + <para> + <emphasis role="bold">Web Beans JBoss TCK Runner</emphasis> - + Groups the porting package implementation and TCK runner projects + that are configured to certify Web Beans deployed on JBoss AS 5.1 + </para> + </listitem> + </orderedlist> + <para> + The dependencies between the projects will either be established + automatically by the m2eclipse plugin, based on the dependency + information in the pom.xml files, or as generated by the <literal>mvn + eclipse:eclipse</literal> command. + </para> + <para> + Your workspace should appear as follows: + </para> + <programlisting><![CDATA[Web Beans + jsr299-api + webbeans-api + webbeans-core + webbeans-core-test + webbeans-logging + webbeans-parent + webbeans-spi + webbeans-version-matrix +CDI TCK + jsr299-tck-api + jsr299-tck-impl + jsr299-tck-parent +Web Beans JBoss TCK Runner + webbeans-jboss-tck-runner + webbeans-porting-package]]></programlisting> + <para> + The tests in the TCK test suite are located in the jsr299-tck-impl + project. You'll be working within this project in Eclipse when you are + developing tests. However, as you learned earlier, there are no references + to a CDI implementation in the TCK. So how can you execute an + individual test in Eclipse? The secret is that you need to establish a + link in Eclipse (not in Maven) between the jsr299-tck-impl project and + your TCK runner project, which in this case is + webbeans-jboss-tck-runner (the project in the jboss-tck-runner + directory). + </para> + <para> + Here are the steps to establish the link: + </para> + <orderedlist> + <listitem> + <para> + Right click on the jsr299-tck-impl project + </para> + </listitem> + <listitem> + <para> + Select Build Path &gt; Configure Build Path... + </para> + </listitem> + <listitem> + <para> + Click on the Projects tab + </para> + </listitem> + <listitem> + <para> + Click the Add... button on the right + </para> + </listitem> + <listitem> + <para> + Check the TCK runner project (e.g., webbeans-jboss-tck-runner) + </para> + </listitem> + <listitem> + <para> + Click the OK button on the Required Project Selection dialog + window + </para> + </listitem> + <listitem> + <para> + Click the OK button on the Java Build Path window + </para> + </listitem> + </orderedlist> + <para> + Of course, the webbeans-jboss-tck-runner also depends on the + jsr299-tck-impl at runtime (so it can actually find the tests to + execute). But Eclipse doesn't distinguish between build-time and + runtime dependencies. As a result, we've created a circular dependency + between the projects. In all likelihood, Eclipse will struggle (if not + fail) to compile one or more projects. How can we break this cycle? + </para> + <para> + As it turns out, the TCK runner doesn't need to access the tests to + build. It only needs its classes, configurations and other dependenices + at runtime (when the TestNG plugin executes). Therefore, we can remove + the TCK runner's dependence on the jsr299-tck-impl project by following + these steps: + </para> + <orderedlist> + <listitem> + <para> + Right click on the webbeans-jboss-tck-runner project + </para> + </listitem> + <listitem> + <para> + Select Build Path &gt; Configure Build Path... + </para> + </listitem> + <listitem> + <para> + Click on the Projects tab + </para> + </listitem> + <listitem> + <para> + Select the TCK tests project (jsr299-tck-impl) + </para> + </listitem> + <listitem> + <para> + Click the Remove button on the right + </para> + </listitem> + <listitem> + <para> + Click the OK button on the Java Build Path window + </para> + </listitem> + </orderedlist> + <para> + You are now ready to execute an individual test class (or artifact). + Let's start with a test artifact capable of running in standalone mode. + </para> + </section> + <section id="eclipse-standalone"> + <title>Running a test in standalone mode</title> + <para> + A standalone test artifact is a class which extends + <literal>AbstractJSR299Test</literal>, is annotated with + <literal>@Artifact</literal>, but is <emphasis + role="italic">not</emphasis> annotated with + <literal>@IntegrationTest</literal>. Select a standalone test artifact + and open it in the Eclipse editor. Now right click in the editor view + and select Run As &gt; TestNG Test. The TestNG view should pop out and + you should see all the tests in that artifact pass (if all goes well). + </para> + <note> + <para> + If the TCK complains that there is a property missing, close all + the projects, open them again, and rebuild. The m2eclipse plugin can + be finicky getting everything built correctly the first time. + </para> + </note> + <para> + So far you have executed a test in standalone mode. That's not + sufficient to pass the TCK. The test must be executed using + in-container mode. Using in-container mode is also the only way to + execute a test annotated with <literal>@IntegrationTest</literal> + (that's what the annotation means) as it requires container resources. + </para> + <para> + Let's see what has to be done to execute an integration test. This + will result in the artifact being deployed to the container, which is + JBoss AS 5.1 if you are using the JBoss TCK runner. + </para> + </section> + <section id="eclipse-in-container"> + <title>Running integration tests</title> + <para> + As you have learned, the JBoss test harness determines how to behave + based on the values of numerous system properties or properties defined + in META-INF/jboss-test-harness.properties classpath resources. If the + property named + <literal>org.jboss.testharness.standalone</literal> + is not defined, the harness assumes that the test is to be run in + standalone mode. In order to run the tests in the container, you need + to add a properties file to the classpath that sets the standalone + property to false and provides values for other properties required to + run an in-container test. + </para> + <para> + The JBoss TCK runner project conveniently provides the properties file + src/test/debug-resources/META-INF/jboss-test-harness.properties that + contains all of the necessary properties for in-container testing in + Eclipse. Assuming you followed the directory structure recommended in + <xref linkend="tck-environment"/>, you are good to go. Otherwise, you + may have to tune the + <literal>org.jboss.testharness.container.extraConfigurationDir</literal> and + <literal>org.jboss.testharness.libraryDirectory</literal> properties to + point to the relative location of the related projects. The properties + should be defined as follows: + </para> + <itemizedlist> + <listitem> + <para> + <literal>org.jboss.testharness.container.extraConfigurationDir + </literal> + - the relative path from the jboss-tck-impl project to a + directory that contains a build.properties or + local.build.properties file defining the location of a JBoss AS + 5.1 installation in the + <literal>jboss.home</literal> + property + </para> + </listitem> + <listitem> + <para> + <literal>org.jboss.testharness.libraryDirectory</literal> + - the relative path from the jboss-tck-impl project to the + target/dependency/lib directory in the TCK runner project. + </para> + </listitem> + </itemizedlist> + <para> + The other properties in that file are defined as follows: + </para> + <programlisting>org.jboss.testharness.standalone=false +orjboss.testharness.container.forceRestart=false +orjboss.testharness.runIntegrationTests=true</programlisting> + <para> + You're now ready to execute an integration test. Select an integration + test (a class that extends <literal>AbstractJSR299Test</literal> and is + annotated with both <literal>@Artifact</literal> and + <literal>@IntegrationTest</literal>) and open it in your Eclipse + editor. Follow these steps to execute the class with the TestNG plugin: + </para> + <orderedlist> + <listitem> + <para> + Right click in the editor view and select Run As &gt; TestNG Test + </para> + </listitem> + <listitem> + <para> + Observe the test fail because of missing dependencies + </para> + </listitem> + <listitem> + <para> + Select the Run &gt; Run Configurations... menu from the main + menubar + </para> + </listitem> + <listitem> + <para> + Select the name of the test class under the TestNG category + </para> + </listitem> + <listitem> + <para> + Select the Classpath tab + </para> + </listitem> + <listitem> + <para> + Select User Entries in the tree + </para> + </listitem> + <listitem> + <para> + Click the Advanced... button on the right + </para> + </listitem> + <listitem> + <para> + Select Add Folders and click the OK button + </para> + </listitem> + <listitem> + <para> + Select the webbeans-jboss-tck-runner/src/test/debug-resources + folder + </para> + </listitem> + <listitem> + <para> + Click the OK button on the Folder Selection dialog window + </para> + </listitem> + <listitem> + <para> + Click on the webbeans-jboss-tck-runner entry + </para> + </listitem> + <listitem> + <para> + Move the webbeans-jboss-tck-runner to the first entry using + the Up button + </para> + </listitem> + <listitem> + <para> + Click the Run button on the Run Configurations dialog window + </para> + </listitem> + </orderedlist> + <para> + When you run the test this time, it should pass. If you get a + failure that the container (e.g., JBoss AS 5.1) must be run with + assertions enabled, you need to stop the container and start it with + the -ea flag (or just leave it stopped and the test will start it + appropriately). + </para> + <para> + You can simply right click and select Run As &gt; TestNG Test for + all subsequent runs for the reason cited earlier, the run configuration + for a class is retained indefinitely. + </para> + <para> + Alternatively, you can configure TestNG to execute all tests + in-container by default by adding the properties file in the + debug-resources folder to the project's classpath as follows: + </para> + <orderedlist> + <listitem> + <para> + Right click on the jsr299-tck-impl project + </para> + </listitem> + <listitem> + <para> + Select Build Path &gt; Configure Build Path... + </para> + </listitem> + <listitem> + <para> + Click on the Libraries tab + </para> + </listitem> + <listitem> + <para> + Click the Add Class Folder... button on the right + </para> + </listitem> + <listitem> + <para> + Check the webbeans-jboss-tck-runner/src/test/debug-resources + folder + </para> + </listitem> + <listitem> + <para> + Click the OK button on the Class Folder Selection dialog + window + </para> + </listitem> + <listitem> + <para> + Click the OK button on the Java Build Path window + </para> + </listitem> + </orderedlist> + <para> + Now you don't have to do any special configuration per test class. + </para> + <para> + You can stop the individual tests from running in-container by + reversing the steps above to remove the debug-resources folder from the + Eclipse classpath. + </para> + <para> + You have now mastered running the CDI TCK against Web Beans using + both Maven 2 and within Eclipse. Now you're likely interested in how to + debug a test so that you can efficiently investigate test failures. + </para> + </section> +<!-- +vim: ts=3:sw=3:tw=80:set expandtab +--> +</chapter> Added: beanvalidation/tck/trunk/src/main/docbook/en-US/executing.xml =================================================================== --- beanvalidation/tck/trunk/src/main/docbook/en-US/executing.xml (rev 0) +++ beanvalidation/tck/trunk/src/main/docbook/en-US/executing.xml 2009-09-29 16:34:58 UTC (rev 17578) @@ -0,0 +1,124 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" +"http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> +<chapter id="executing"> + <title>Executing the Test Suite</title> + + <para>This chapter explains how to run the TCK on Hibernate Validator as + well as your own implementation. The Bean Validation TCK uses the Maven 2 + TestNG plugin and the JBoss Test Harness to execute the test suite. Learning + to execute the test suite from Maven 2 is prerequisite knowlege for running + the tests in an IDE, such as Eclipse.</para> + + <section> + <title>The Test Suite Runner</title> + + <para>The test suite is executed by the Maven 2 TestNG plugin during the + test phase of the Maven 2 life cycle. The execution happens within a TCK + runner project (as opposed to the TCK project itself). Hibernate Validator + includes a TCK runner project that executes the Bean Validation TCK on + Hibernate Validator running inside JBoss AS 5.1. To execute the Bean + Validation TCK on your own Bean Validation implementation, you could + modify the TCK runner project included with Hibernate Validator to use + your Bean Validation implementation as described in <xref + linkend="configuration" />.</para> + </section> + + <section> + <title>Running the Tests In Standalone Mode</title> + + <para>To execute the TCK test suite against Web Beans, first switch to the + jboss-tck-runner directory in the extracted Web Beans distribution:</para> + + <programlisting>cd ri/hibernate-validator-tck-runner</programlisting> + + <note> + <para>These instructions assume you have extracted the CDI-related + software according to the recommendation given in <xref + linkend="tck-environment" />.</para> + </note> + + <para>Then execute the Maven 2 life cycle through the test phase:</para> + + <programlisting>mvn test</programlisting> + + <para>Without any command-line flags, the test suite is run in standalone + mode, which means that any test class with the + <literal&gt;(a)org.jboss.testharness.impl.packaging.IntegrationTest </literal> + annotation is skipped. This mode uses the + <literal>StandaloneContainers</literal> SPI to invoke the test artifact + within a mock Java EE life cycle and capture the results of the test. + However, passing the suite in this mode is not sufficient to pass the TCK + as a whole. The suite must be passed while executing using the + in-container mode.</para> + </section> + + <section> + <title>Running the Tests In the Container</title> + + <para>To execute the test suite using in-container mode with the JBoss TCK + runner, you first have to setup JBoss AS as described in the <xref + linkend="tck-in-jboss-as" /> callout.</para> + + <para>Then, execute the TCK runner with Maven 2 as follows:</para> + + <programlisting>mvn test -Dincontainer</programlisting> + + <para>The presence of the <literal>incontainer</literal> property + activates a Maven 2 profile that assigns the + <literal>org.jboss.testharness.standalone</literal> system property to + <literal>false</literal> and the + <literal>org.jboss.testharness.runIntegrationTests</literal> system + property to <literal>true</literal>, hence activating the in-container + test mode. This time, all the test artifacts in the test suite are + executed.</para> + + <para>The in-container profile will also start and stop the application + server automatically, a feature which the profile activates by setting the + <literal>org.jboss.testharness.container.forceRestart</literal> to + <literal>true</literal>.</para> + + <para>The in-container mode uses the <literal>Containers</literal> SPI to + deploy the test artifact to the container and execute the test in a true + Java EE life cycle. The JBoss TCK runner has a dependency on the library + that provides an implementation of this interface for JBoss AS 5.1.</para> + + <para>Since in-container tests are executed in a remote JVM, the results + of the test must be communicated back to the runner over a + container-supported protocol. The JBoss Test Harness provides + servlet-based communication over HTTP as described in <xref + linkend="incontainer-communication" />.</para> + </section> + + <section> + <title>Dumping the Test Artifacts</title> + + <para>As you have learned, when the test suite is executing using + in-container mode, each test class is packaged as a deployable artifact + and deployed to the container. The test is then executed within the + context of the deployed application. This leaves room for errors in + packaging. When investigating a test failure, you may find it helpful to + inspect the artifact after it's generated. The TCK can accommodate this + type of inspection by "dumping" the generated artifact to disk.</para> + + <para>The feature just described is activated in the jboss-tck-runner + project by appending the <literal>dumpArtifacts</literal> command line + property to the end of the command that invokes the Maven 2 test + phase.</para> + + <programlisting>mvn test-compile -DdumpArtifacts</programlisting> + + <para>The directory where the artifacts get written is configured using + the <literal>org.jboss.testharness.outputDirectory</literal> property. The + <literal>dumpArtifacts</literal> profile in the jboss-tck-runner project + sets this value to the relative directory path + <literal>target/jsr303-artifacts</literal>.</para> + + <para>You can read more about this feature in <xref + linkend="dumping-test-artifacts" />.</para> + </section> + + <!-- +vim: ts=3:sw=3:tw=80:set expandtab +--> +</chapter> Added: beanvalidation/tck/trunk/src/main/docbook/en-US/harness/Author_Group.xml =================================================================== --- beanvalidation/tck/trunk/src/main/docbook/en-US/harness/Author_Group.xml (rev 0) +++ beanvalidation/tck/trunk/src/main/docbook/en-US/harness/Author_Group.xml 2009-09-29 16:34:58 UTC (rev 17578) @@ -0,0 +1,22 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE authorgroup PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [ ]> + + +<authorgroup> + <author> + <firstname>Pete</firstname> + <surname>Muir</surname> + <affiliation> + <jobtitle>JBoss Test Harness lead</jobtitle> + <orgname>Red Hat Middleware LLC</orgname> + </affiliation> + </author> + <author> + <firstname>Dan</firstname> + <surname>Allen</surname> + <affiliation> + <jobtitle>JBoss Test Harness developer</jobtitle> + <orgname>Red Hat Middleware LLC</orgname> + </affiliation> + </author> +</authorgroup> Added: beanvalidation/tck/trunk/src/main/docbook/en-US/harness/Book_Info.xml =================================================================== --- beanvalidation/tck/trunk/src/main/docbook/en-US/harness/Book_Info.xml (rev 0) +++ beanvalidation/tck/trunk/src/main/docbook/en-US/harness/Book_Info.xml 2009-09-29 16:34:58 UTC (rev 17578) @@ -0,0 +1,15 @@ +<?xml version='1.0' encoding='UTF-8'?> +<!DOCTYPE bookinfo PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [ ]> +<bookinfo> + <copyright> + <year>2009</year> + <holder> + Red Hat Middleware LLC, and individual contributors listed as + authors. + </holder> + </copyright> + + <title>JBoss Test Harness Refernce Guide</title> + <subtitle></subtitle> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Author_Group.xml" /> +</bookinfo> Added: beanvalidation/tck/trunk/src/main/docbook/en-US/harness/Book_Preface.xml =================================================================== --- beanvalidation/tck/trunk/src/main/docbook/en-US/harness/Book_Preface.xml (rev 0) +++ beanvalidation/tck/trunk/src/main/docbook/en-US/harness/Book_Preface.xml 2009-09-29 16:34:58 UTC (rev 17578) @@ -0,0 +1,15 @@ +<?xml version='1.0' encoding='UTF-8'?> +<!DOCTYPE preface PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [ ]> +<preface id="book-preface"> + <title>Preface</title> + + <section id="book-organization"> + + <title>How This Book Is Organized</title> + + <para> + Chapter 1 provides an introduction and explains the motiviations for + and goals of the test harness. + </para> + </section> +</preface> Added: beanvalidation/tck/trunk/src/main/docbook/en-US/harness/configuration.xml =================================================================== --- beanvalidation/tck/trunk/src/main/docbook/en-US/harness/configuration.xml (rev 0) +++ beanvalidation/tck/trunk/src/main/docbook/en-US/harness/configuration.xml 2009-09-29 16:34:58 UTC (rev 17578) @@ -0,0 +1,224 @@ +<?xml version='1.0' encoding='UTF-8'?> +<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [ ]> +<chapter id="test-harness-configuration"> + <title>Configuration</title> + <para> + This chapter lays out how to configure the JBoss Test Harness by + specifying the API implementation classes, defining the target container + connection information, and various other switches. Finally, a detailed + account of how the JBoss Test Harness negotiates the execution of the + tests in the container is given. + </para> + <section id="test-harness-properties"> + <title>JBoss Test Harness Properties</title> + <para> + The JBoss Test Harness allows the test suite to be launched in a + pluggable fashion. In order to execute a test suite, the JBoss Test + Harness must be configured by specifying implementations of the test + launcher and container APIs. + </para> + <para> + System properties and/or the resource + <code>META-INF/jboss-test-harness.properties</code>, + a Java properties file, are used to configure the JBoss Test Harness. + The bootstrap configuration builder looks to the property + <literal>org.jboss.testharness.api.ConfigurationBuilder</literal>, + the first property listed in table 3.1, for the fully qualified class + name (FQCN) of a concrete configuration builder implementation to get + started. This implementation loads the remaining configuration settings + and produces a JBoss Test Harness configuration. + </para> + <para> + For you convenience, the default configuration builder implementation + <literal>org.jboss.testharness.impl.PropertiesBasedConfigurationBuilder + </literal> is provided, which collates all the JBoss Test Harness + configuration settings from system and Java properties. It does so by + aggregating the system properties with the properties defined in the + META-INF/jboss-test-harness.properties resource in any classpath entry + under a single properties map, allowing you to partition the + configuration settings as needed. + </para> + <para> + A complete list of configuration properties for the JBoss Test + Harness + has been itemized in + <xref linkend="test-harness-properties-list" />, + accompanied by the default value (if any) and a description for each + property. + </para> + <table frame="all" id="test-harness-properties-list" title="Test Harness Configuration Properties"> + <title>JBoss Test Harness Configuration Properties</title> + <tgroup cols="2"> + <colspec colnum="1" colname="property" colwidth="5*" /> + <colspec colnum="2" colname="description" colwidth="2*" /> + <thead> + <row> + <entry>Property = Default Value</entry> + <entry>Description</entry> + </row> + </thead> + <tbody> + <row> + <entry> + <literal>org.jboss.testharness.api.ConfigurationBuilder= + </literal> + <literal>org.jboss.testharness.impl.PropertiesBasedConfigurationBuilder + </literal> + </entry> + <entry> + The configuration bootstrap class for the JBoss Test + Harness. + </entry> + </row> + <row> + <entry> + <literal>org.jboss.testharness.testPackage=</literal> + </entry> + <entry> + The top-level Java package containing classes to be + tested. Used to determine which artifacts to dump to disk + only; not used during running of a suite. + </entry> + </row> + <row> + <entry> + <literal>org.jboss.testharness.libraryDirectory=</literal> + </entry> + <entry> + Directory containing extra JARs which should be deployed + in + the artifact (for example in + <literal>WEB-INF/lib</literal>. + </entry> + </row> + <row> + <entry> + <literal>org.jboss.testharness.standalone=true</literal> + </entry> + <entry> + Tests are run using standalone mode if true or using + in-container mode if false. + </entry> + </row> + <row> + <entry> + <literal>org.jboss.testharness.runIntegrationTests=false + </literal> + </entry> + <entry> + If true, integration tests are run. In-container mode + must be activated. + </entry> + </row> + <row> + <entry> + <literal>org.jboss.testharness.spi.Containers=</literal> + </entry> + <entry> + The deployment implementation for setting up and + tearing down the container and deploying and undeploying + in-container tests. + </entry> + </row> + <row> + <entry> + <literal>org.jboss.testharness.host=localhost:8080 + </literal> + </entry> + <entry> + The host and port on which the container is running. + </entry> + </row> + <row> + <entry> + <literal>org.jboss.testharness.connectDelay=5000</literal> + </entry> + <entry> + The timeout (ms) when attempting to connect to the + container (e.g. via http). + </entry> + </row> + <row> + <entry> + <literal>org.jboss.testharness.api.TestLauncher=</literal> + </entry> + <entry> + The in-container test launcher, the built in + <literal>org.jboss.testharness.impl.runner.servlet.ServletTestLauncher + </literal> + is provided and suitable for any Servlet environment. + </entry> + </row> + <row> + <entry> + <literal>org.jboss.testharness.container.\</literal> + <literal>deploymentExceptionTransformer=</literal> + </entry> + <entry> + Provides an interception feature for deployment + exceptions, + allowing them to be inspected and altered before + reporting + to the test harness for validation by the test + case. + </entry> + </row> + <row> + <entry> + <literal>org.jboss.testharness.container.forceRestart=false + </literal> + </entry> + <entry> + Whether the container should be restarted before the + tests are executed. + </entry> + </row> + <row> + <entry> + <literal>org.jboss.testharness.container.extraConfigurationDir= + </literal> + </entry> + <entry> + A directory containing a + <literal>build.properties</literal> + or + <literal>local.build.properties</literal> + files that define additional properties. Can be used to + provide runtime specific properties. + </entry> + </row> + <row> + <entry> + <literal>org.jboss.testharness.spi.StandaloneContainers= + </literal> + </entry> + <entry> + The container implementation for executing standalone + tests. + </entry> + </row> + <row> + <entry> + <literal>dumpArtifacts=false</literal> + </entry> + <entry> + Whether the test artifacts should be written to disk + for inspection. + </entry> + </row> + <row> + <entry> + <literal>org.jboss.testharness.outputDirectory=</literal> + <literal>%java.io.tmpdir%/jsr-299-tck/</literal> + </entry> + <entry> + Directory where test artifacts will be written to disk, if + <literal>dumpArtifacts</literal> + is true. + </entry> + </row> + </tbody> + </tgroup> + </table> + </section> +</chapter> Added: beanvalidation/tck/trunk/src/main/docbook/en-US/harness/executing.xml =================================================================== --- beanvalidation/tck/trunk/src/main/docbook/en-US/harness/executing.xml (rev 0) +++ beanvalidation/tck/trunk/src/main/docbook/en-US/harness/executing.xml 2009-09-29 16:34:58 UTC (rev 17578) @@ -0,0 +1,228 @@ +<?xml version='1.0' encoding='UTF-8'?> +<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [ ]> +<chapter id="executing-test-harness"> + <title>Executing a Test Suite</title> + <para> + This chapter explains how to execute and debug a test suite built using + the JBoss Test Harness. + </para> + <section id="test-suite-runner"> + <title>Building a test suite runner using Maven 2</title> + <para> + The test suite runner project is the magic that makes everything come + together and allows you to execute the test suite. If you fully + understand how the JBoss Test Harness functions, and have a good grasp + on Maven 2, then it's not to difficult to understand how the test suite + runner project works. Regardless of your background, this guide covers + what you need to know to get up and running by studying the test suite + runner used to run the CDI TCK against the CDI RI, Web Beans. + </para> + <para> + The TCK runner for the Web Beans can be found in the jboss-tck-runner + directory in the Web Beans distribution. The dependencies of the TCK + runner project for Web Beans are listed in + <xref linkend="tck-runner-dependencies" />. + </para> + <table frame="all" id="tck-runner-dependencies" title="CDI TCK Runner dependencies"> + <title>Web Beans JBoss TCK Runner Dependencies</title> + <tgroup cols="3"> + <colspec colname="c1" /> + <colspec colname="c2" /> + <colspec colname="c3" /> + <thead> + <row> + <entry>Group ID</entry> + <entry>Artifact ID</entry> + <entry>Version</entry> + </row> + </thead> + <tbody> + <row> + <entry>org.jboss.webbeans</entry> + <entry>jsr299-api</entry> + <entry>1.0.0-SNAPSHOT</entry> + </row> + <row> + <entry>org.jboss.jsr299.tck</entry> + <entry>jsr299-tck-api</entry> + <entry>1.0.0-SNAPSHOT</entry> + </row> + <row> + <entry>org.jboss.jsr299.tck</entry> + <entry>jsr299-tck-impl</entry> + <entry>1.0.0-SNAPSHOT</entry> + </row> + <row> + <entry>org.jboss.webbeans</entry> + <entry>webbeans-core</entry> + <entry>1.0.0-SNAPSHOT</entry> + </row> + <row> + <entry>org.jboss.webbeans</entry> + <entry>webbeans-porting-package</entry> + <entry>1.0.0-SNAPSHOT</entry> + </row> + <row> + <entry>org.testng</entry> + <entry>testng (classifier: jdk15)</entry> + <entry>5.8</entry> + </row> + <row> + <entry>org.jboss.test-harness</entry> + <entry>jboss-test-harness-jboss-as-51</entry> + <entry>1.0.0.BETA3</entry> + </row> + </tbody> + </tgroup> + </table> + <para> + You can find all of these artifacts in the <ulink + url="http://repository.jboss.org/maven2">JBoss Maven + repository</ulink>. + </para> + <para> + You should substituate the webbeans-core and webbeans-porting-package + artifacts from table 2.2.3 with your own artifacts. You'll also need to + replace the jboss-test-harness-jboss-as-51 artifact if you are not + testing your implementation on JBoss AS 5.1. The + jboss-test-harness-jboss-as-51 artifact contains implementations of the + <literal>Containers</literal> + SPI for the JBoss Test Harness for JBoss AS 5.1. + </para> + <note> + <para> + When running the test suite in the in-container mode, the tests + will run against libraries installed into the container. In this + project, Web Beans is only declared as a Maven dependency for when + the TCK test suite is being executed in standalone mode. + </para> + </note> + + <programlisting role="xml"><![CDATA[<plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-dependency-plugin</artifactId> + <executions> + <execution> + <id>copy</id> + <phase>process-resources</phase> + <goals> + <goal>copy</goal> + </goals> + <configuration> + <stripVersion>true</stripVersion> + <artifactItems> + <artifactItem> + <groupId>org.jboss.jsr299.tck</groupId> + <artifactId>jsr299-tck-impl</artifactId> + <type>xml</type> + <classifier>suite</classifier> + <overWrite>true</overWrite> + </artifactItem> + <artifactItem> + <groupId>org.jboss.webbeans</groupId> + <artifactId> + webbeans-porting-package + </artifactId> + <overWrite>true</overWrite> + <outputDirectory> + ${project.build.directory}/dependency/lib + </outputDirectory> + </artifactItem> + <artifactItem> + <groupId>org.jboss.webbeans</groupId> + <artifactId>webbeans-core-test</artifactId> + <overWrite>true</overWrite> + <outputDirectory> + ${project.build.directory}/dependency/lib + </outputDirectory> + </artifactItem> + <artifactItem> + <groupId>javax.el</groupId> + <artifactId>el-ri</artifactId> + <overWrite>true</overWrite> + <outputDirectory> + ${project.build.directory}/dependency/lib + </outputDirectory> + </artifactItem> + </artifactItems> + </configuration> + </execution> + </executions> +</plugin>]]></programlisting> + <para> + The target folder for the copies of the dependencies (i.e., the JAR + files) is declared as the JBoss Test Harness library directory; this + results in these libraries being added to the test artifact using the + following property assignment: + </para> + <programlisting>org.jboss.testharness.libraryDirectory=target/dependency/lib</programlisting> + <para> + We also copy the test suite configuration from the + local Maven + repository (groupId=org.jboss.jsr299.tck, + artifactId=jsr299-tck-impl, + classifier=suite, type=xml, + version=1.0.0-SNAPSHOT) to a local + repository as the TestNG Maven + plugin expects a local file. + </para> + <para> + The TCK is executed using the Maven TestNG plugin. Maven 2 profiles are + used to control the properties that are set at the time of the + execution. For instance, the + <literal>incontainer</literal> + profile enables integration tests and disables standalone mode, + changing the default settings. + </para> + <para> + The jboss-tck-runner project also defines the JBoss Test Harness + extra configuration directory using the following property: + </para> + <programlisting>org.jboss.testharness.container.extraConfigurationDir=../jboss-as</programlisting> + <para> + The JBoss Test Harness looks in this directory for either a + build.properties or local.build.properties file that declares + additional configuration properties. In particular, the JBoss AS + Containers implementation looks here to find the + <literal>jboss.home</literal> property so it knows where the scripts + are located to start and stop JBoss AS. + </para> + </section> + + <section id="dumping-test-artifacts"> + <title>Dumping the Test Artifacts to Disk</title> + <para> + As you have learned, when the test suite is executing using + in-container mode, each test class is packaged as a deployable artifact + and deployed to the container. The test is then executed within the + context of the deployed application. This leaves room for errors in + packaging. When investigating a test failure, it's helpful to be able + to inspect the artifact after it is generated. The JBoss Test Harness + can accommodate this type of inspection by "dumping" the generated + artifact to disk. + </para> + <para> + If you want to write the artifacts to disk, and avoid executing the + test suite, you can simply execute the main method of the class + <literal>org.jboss.testharness.api.TCK</literal>. + For example you could use a Maven profile that is activated when the + <literal>dumpArtifacts</literal> + command line property is defined: + </para> + <programlisting>mvn test-compile -DdumpArtifacts</programlisting> + <para> + The output directory where the artifacts are written is defined by the + property + <literal>org.jboss.testharness.outputDirectory</literal>. + </para> + <para> + Once the artifact is written to disk, you have an option of manually + deploying it to the container. You can execute the tests in the artfact + by requesting the context path of the application in the browser. If + you want to execute an individual test method, specify the method name + in the + <literal>methodName</literal> + request parameter (e.g., ?methodName=testMethodName). + </para> + </section> +</chapter> Added: beanvalidation/tck/trunk/src/main/docbook/en-US/harness/images/in-container-execution.png =================================================================== (Binary files differ) Property changes on: beanvalidation/tck/trunk/src/main/docbook/en-US/harness/images/in-container-execution.png ___________________________________________________________________ Name: svn:mime-type + application/octet-stream Added: beanvalidation/tck/trunk/src/main/docbook/en-US/harness/introduction.xml =================================================================== --- beanvalidation/tck/trunk/src/main/docbook/en-US/harness/introduction.xml (rev 0) +++ beanvalidation/tck/trunk/src/main/docbook/en-US/harness/introduction.xml 2009-09-29 16:34:58 UTC (rev 17578) @@ -0,0 +1,286 @@ +<?xml version='1.0' encoding='UTF-8'?> +<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [ ]> +<chapter id="test-harness-introduction"> + <title>Introduction (JBoss Test Harness)</title> + + <para> + This chapter explains the purpose of the test harness and describes + its key features. + </para> + <para> + The JBoss Test Harness is a testing framework based on TestNG that + provides a series of extensions that allow runtime packaging and + deployment of Java EE artifacts (EAR or WAR) for in-container testing. + It's important to note that the JBoss Test Harness has no relation with, + or dependency on, the JBoss Application Server (JBoss AS). + </para> + <note> + <para> + You'll often see the term + <emphasis role="italic">in-container</emphasis> + used in this reference guide. This term refers to running the test + suite in any of the aforementioned environments, whilst + <emphasis role="italic">standalone</emphasis> + refers to running the tests outside the container via an + implementation-specific standalone bootstrap. The standalone mode only + runs those tests which the CDI RI can run without deployment in a Java + EE container. + </para> + </note> + <para> + The last thing Java developers want is yet another testing framework to + make their life more complicated. That's why the JBoss Test Harness is + built entirely upon TestNG. TestNG is one of two prominent test + frameworks for Java (the other being JUnit). Furthermore, what developers + want is a good integration with their Integrated Development Environment + (IDE). These days, if a tool doesn't have an IDE plugin, then it won't + get the attention it deserves. TestNG plugins are available for all major + IDEs and build tools (Ant and Maven 2). Again, a motivating factor for + extending TestNG. + </para> + <para> + Because it leverages the existing TestNG ecosystem, there is no need for + a special test launcher for the JBoss Test Harness. You simply use the + IDE or build tool of your choice (so long as it has TestNG support). You + also get reporting and debugging for free (various reporting plugins are + provided for TestNG). + </para> + <para> + You can read more about TestNG at + <ulink url="http://testng.org/doc/documentation-main.html">testng.o...;. + </para> + <para> + The JBoss Test Harness supports the following features: + </para> + <itemizedlist> + <listitem> + <para> + Test activation via any method supported by the TestNG + configuration descriptor (package, group, class) + </para> + </listitem> + <listitem> + <para> + Exclusion of in-container tests in standalone mode + </para> + </listitem> + <listitem> + <para> + Exclusion of individual tests labeled as under investigation + </para> + </listitem> + <listitem> + <para> + Integration with any TestNG plugin (Eclipse, IntelliJ, Ant, Maven) + </para> + </listitem> + <listitem> + <para> + Automated reporting capability as provided by TestNG + </para> + </listitem> + <listitem> + <para> + Standalone and in-container test mode + </para> + </listitem> + <listitem> + <para> + Container pluggability + </para> + </listitem> + <listitem> + <para> + Declarative packaging of additional resources and classes in + artifact + </para> + </listitem> + <listitem> + <para> + Declarative deployment exception trapping + </para> + </listitem> + <listitem> + <para> + Artifact dumping for failure and packaging analysis + </para> + </listitem> + </itemizedlist> + <para> + A test is designated by a method annotated with + <literal&gt;(a)org.testng.annotations.Test&lt;/literal&gt; + in a class which extends + <literal>org.jboss.testharness.AbstractTest</literal> + and is annotated with + <literal&gt;(a)org.jboss.testharness.impl.packaging.Artifact&lt;/literal&gt;. + </para> + <note> + <para> + Test suites may often choose to extend <literal>AbstractTest</literal> + and require tests to extend that base class. In fact, both the CDI TCK + and the Bean Validation TCK provide base classes that extend + <literal>AbstractTest</literal> to provide functionality specific to + the needs of the TCK. + </para> + </note> + <para> + The + <literal>@Test</literal> + annotation is provided by TestNG, the + <literal>@Artifact</literal> + annotation is provided by the JBoss Test Harness and the + <literal>AbstractTest</literal> + is part of the JBoss Test Harness. There is a one-to-one mapping between a + TestNG test class and an artifact. The packaging type is defined by the + <literal&gt;(a)org.jboss.testharness.impl.packaging.Packaging&lt;/literal&gt; + annotation on the test class, defaulting to a WAR if not specified. + </para> + <para> + Prior to executing the tests for a given class, the JBoss Test Harness + packages the class as a deployable artifact (EAR or WAR), along with any + extra resources specified, and deploys the artifact to the container. The + harness provides test execution and result reporting via HTTP + communication to a simple Servlet using a thin layer over the TestNG test + launcher. The test harness can also catch and enforce expected deployment + exceptions. This setup and tear down activity is provided by the super + class + <literal>org.jboss.testharness.AbstractTest</literal>, + which all test classes must extend (directly or indirectly). + </para> + <para> + If the annotation + <literal&gt;(a)org.jboss.testharness.impl.packaging.IntegrationTest + </literal> + is not present on the test class, then it means the test class can be + executed in standalone mode. In standalone mode, the deployable artifact + is assembled on the local classpath and the tests execute in the same JVM + as the launcher, just as though it were a regular TestNG test case. The + standalone mode is provided for convenience and efficiency, allowing you + the speed of mock-based testing and the confidence of an in-container test, + using the same test objects and tests. + </para> + + <section id="in-container-communication"> + <title>Negotiating the execution of an in-container test</title> + <para> + The basic procedure of an in-container test is as follows. The JBoss + Test Harness produces a deployable artifact from an + <literal>@Artifact</literal> + test class and any declared dependent classes, descriptors or other + resources. Then it deploys the artifact to the container using the + <literal>Containers</literal> + SPI, negotiates with the container to execute the test and return the + result and, finally, undeploys the artifact. TestNG collects the results + of all the tests run in the typical way and produces a report. + </para> + <graphic fileref="images/in-container-execution.png" align="center" /> + <para> + The question is, how does the JBoss Test Harness negotiate with the + container to execute the test when TestNG is being invoked locally? + Technially the mechanism is pluggable, but JBoss Test Harness provides + a default implementation that uses HTTP communication that you will + likely use. Here's how the default implementation works. + </para> + <para> + The artifact generator bundles and registers (in the web.xml + descriptor) an + <literal>HttpServlet</literal>, + <literal>org.jboss.testharness.impl.runner.servlet.ServletTestRunner</literal>, + that responds to test execution GET requests. TestNG running on the + client side delegates to a test launcher (more on that in a moment) + which originates these text execution requests to transfer control to + the container JVM. The name of the test method to be executed is + specified in a request query parameter named + <literal>methodName</literal>. + </para> + <para> + When the test execution request is received, the servlet delegates to + an instance of + <literal>org.jboss.testharness.impl.runner.TestRunner</literal>, + passing it the name of the test method. + <literal>TestRunner</literal> + reads the name of the test class from the resource + META-INF/jboss-test-harness.properties, which is bundled in the + artifact by the artifact generator. It then combines the class name + and the method name to produce a TestNG test suite and runs the suite + (within the context of the container). + </para> + <para> + TestNG returns the results of the run as an + <literal>ITestResult</literal> + object. + <literal>ServletTestRunner</literal> + translates this object into a + <literal>org.jboss.testharness.api.TestResult</literal> + and passes it back to the test launcher on the client side by encoding + the translated object into the response. The object gets encoded as + either html or a serialized object, depending on the value of the + <literal>outputMode</literal> + request parameter that was passed to the servlet. Once the result has + been transfered to the client-side TestNG, TestNG wraps up the run of + the test as though it had been executed in the same JVM. + </para> + <para> + There's one piece missing. How does TestNG on the client side know to + submit a request to the + <literal>ServletTestRunner</literal> + servlet to get TestNG to execute the test in the container JVM? That's + the role of the test launcher. + </para> + <para> + The test launcher is the API that allows test suite to launch the test + in a pluggable fashion. + <literal>AbstractTest</literal>, + the super class of + <literal>AbtractJSR299Test</literal>, + implements + <literal>IHookable</literal>, + a TestNG interface which allows the execution of the test method to + be intercepted. Using that mechanism, <literal>AbstractTest</literal> delegates execution + of the test method (a method annotated with + <literal>@Test</literal> + in an + <literal>@Artifact</literal> + class) to an implementation of + <literal>org.jboss.testharness.api.TestLauncher</literal> + if the tests are being executed in-container. As you might anticipate, + the implementation is specified using a property with the same name as + the interface in a META-INF/jboss-test-launcher.properties resource. + The JBoss Test Harness provides a default implementation, + <literal>org.jboss.testharness.impl.runner.servlet.ServletTestLauncher</literal>, + that hooks into the HTTP communication infrastructure described + above. It invokes the + <literal>ServletTestRunner</literal> + servlet for each method annotated with + <literal>@Test</literal> + in the + <literal>@Artifact</literal> + that is not otherwise disabled. + </para> + <para> + If you wish to implement the runner yourself, you must return a + <literal>TestResult</literal> as a result of executing the method in + the container. You must also ensure that any exception which occurs + during deployment is wrapped as a + <literal>org.jboss.testharness.api.DeploymentException</literal>, and + that any communication problem is rethrown as an + <literal>IOException</literal>. The deployment exception may be + transformed by an implementation of the + <literal>org.jboss.testharness.api.DeploymentExceptionTransformer + </literal> interface, which is specified using the + <literal>org.jboss.testharness.container.deploymentExceptionTransformer + </literal> property. The default implementation passes on the original + exception unchanged. The implementation for JBoss AS used with the CDI + TCK, on the other hand, deciphers the exception thrown by the JBoss + deployer and converts it to one of the catagory exceptions defined in + the CDI TCK API. + </para> + <para> + So in short, JBoss Test Harness takes care of all the interfaces you + need to execute tests in-container except for the implementation of + the <literal>Containers</literal> SPI. That is, unless you are + deploying to one of the containers supported by the JBoss Test Harness + (TODO we need a table showing supported containers). + </para> + </section> +</chapter> Added: beanvalidation/tck/trunk/src/main/docbook/en-US/harness/master.xml =================================================================== --- beanvalidation/tck/trunk/src/main/docbook/en-US/harness/master.xml (rev 0) +++ beanvalidation/tck/trunk/src/main/docbook/en-US/harness/master.xml 2009-09-29 16:34:58 UTC (rev 17578) @@ -0,0 +1,8 @@ +<?xml version='1.0' encoding='UTF-8'?> +<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [ ]> +<book lang="en"> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Book_Info.xml"/> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Book_Preface.xml"/> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="introduction.xml"/> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="configuration.xml"/> +</book> Added: beanvalidation/tck/trunk/src/main/docbook/en-US/images/testng-emailable-report.png =================================================================== (Binary files differ) Property changes on: beanvalidation/tck/trunk/src/main/docbook/en-US/images/testng-emailable-report.png ___________________________________________________________________ Name: svn:mime-type + application/octet-stream Added: beanvalidation/tck/trunk/src/main/docbook/en-US/images/testng-plugin-report.png =================================================================== (Binary files differ) Property changes on: beanvalidation/tck/trunk/src/main/docbook/en-US/images/testng-plugin-report.png ___________________________________________________________________ Name: svn:mime-type + application/octet-stream Added: beanvalidation/tck/trunk/src/main/docbook/en-US/images/testng-suite-detail-report.png =================================================================== (Binary files differ) Property changes on: beanvalidation/tck/trunk/src/main/docbook/en-US/images/testng-suite-detail-report.png ___________________________________________________________________ Name: svn:mime-type + application/octet-stream Added: beanvalidation/tck/trunk/src/main/docbook/en-US/images/testng-summary-report.png =================================================================== (Binary files differ) Property changes on: beanvalidation/tck/trunk/src/main/docbook/en-US/images/testng-summary-report.png ___________________________________________________________________ Name: svn:mime-type + application/octet-stream Added: beanvalidation/tck/trunk/src/main/docbook/en-US/installation.xml =================================================================== --- beanvalidation/tck/trunk/src/main/docbook/en-US/installation.xml (rev 0) +++ beanvalidation/tck/trunk/src/main/docbook/en-US/installation.xml 2009-09-29 16:34:58 UTC (rev 17578) @@ -0,0 +1,228 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" +"http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> +<chapter id="installation"> + <title>Installation</title> + + <para>This chapter explains how to obtain the TCK and supporting software + and provides recommendations for how to install/extract it on your + system.</para> + + <section> + <title>Obtaining the Software</title> + + <para>You can obtain a release of the Bean Validation TCK project from the + from the <ulink url="http://seamframework.org/Download">download + page</ulink> on the Hibernate Validator website. The Bean Validation TCK + is distributed as a ZIP file, which contains the TCK artifacts (the test + suite binary and source, porting package API binary and source, the test + suite descriptor, the audit source and report) in <code>/artifacts</code>, + the TCK library dependencies in <code>/lib</code> and documentation in + <code>/lib</code>.</para> + + <para>You can also download the currnet source code from <ulink + url="http://anonsvn.jboss.org/repos/hibernate/beanvalidation/tck/tru... + SVN repository</ulink>.</para> + + <para>The TCK project is available in the JBoss Maven 2 repository as + org.hibernate.jsr303.tck<code>:</code>org.hibernate.jsr303.tck; the POM + defines all dependencies required to run the TCK.</para> + + <para>Executing the TCK requires a Java EE 5 or better runtime environment + (i.e., application server), to which the test artifacts are deployed and + the individual tests are invoked. The TCK does not depend on any + particular Java EE implementation.</para> + + <para>The JSR-303: Bean Validation reference implementation (RI) project + is named Hibernate Validator. You can obtain the latest Hibernate + Validator release from the <ulink + url="https://www.hibernate.org/6.html">download page</ulink> on Hibernate + website.</para> + + <note> + <para>Hibernate Validator is not required for running the Bean + Validation TCK, but it can be used as a reference for familiarizing + yourself with the TCK before testing your own Bean Validation + implementation.</para> + </note> + + <para>Naturally, to execute Java programs, you must have a Java SE runtime + environment. The TCK requires Java 5 or better, which you can obtain from + the <ulink url="http://java.sun.com">Java Software</ulink> website.</para> + </section> + + <section id="tck-environment"> + <title>The TCK Environment</title> + + <para>The TCK requires the following two Java runtime environments:</para> + + <itemizedlist> + <listitem> + <para>Java 5 or better</para> + </listitem> + + <listitem> + <para>Java EE 5 or better (e.g., JBoss AS 5.x or GlassFish V3)</para> + </listitem> + </itemizedlist> + + <para>You should refer to vendor instructions for how to install the + runtime.</para> + + <para>The rest of the TCK software can simply be extracted. It's + recommended that you create a folder named jsr303 to hold all of the + jsr303-related projects. Then, extract the TCK distribution into a + subfolder named tck. If you have downloaded the Hibernate Validator + distribution, extract it into a sibling folder named ri. The resulting + folder structure is shown here:</para> + + <note> + <para>This layout is assumed through all descriptions in this reference + guide.</para> + </note> + + <programlisting>jsr303/ + ri/ + tck/</programlisting> + + <para>Each test class is treated as an individual artifact (hence the + <literal>@Artifact</literal> annotation on the class). All test methods + (i.e., methods annotated with <literal>@Test</literal>) in the test class + are run in the application, meaning bean discovery occurs exactly once per + artifact and the same BeanManager is used by each test method in the + class.</para> + + <tip id="tck-in-jboss-as"> + <title>Running the TCK against the Bean Validation RI (Hibernate + Validator) and JBoss AS</title> + + <para>Web Beans is built as a modular library, and as such can be + retro-fitted to Java EE 5 products as required. JBoss AS 5.0 and above + releases bundle Web Beans. JBoss AS 5.0 also allows you to upgrade the + Web Beans module to the current release (though some functionality may + be disabled).</para> + + <para>To install JBoss AS 5.1 and update to the latest Web Beans + release:</para> + + <itemizedlist> + <listitem> + <para>First, you should download JBoss AS 5.1 from the JBoss AS + <ulink url="http://jboss.org/jbossas/downloads">project + page</ulink>.</para> + </listitem> + + <listitem> + <para>Set the <literal>JBOSS_HOME</literal> environment variable to + the location of the JBoss AS software.</para> + </listitem> + + <listitem> + <para>Change to the webbeans directory.</para> + </listitem> + + <listitem> + <para>Make sure the <literal>jboss.home</literal> property in the + local.build.properties file in the jboss-as directory references a + JBoss AS 5.1 installation:</para> + + <programlisting>jboss.home=/path/to/jboss-as-5.1</programlisting> + </listitem> + + <listitem> + <para>Then, run Ant from the jboss-as directory to install the + deployer:</para> + + <programlisting>ant update</programlisting> + + <para>The libraries needed by the deployer are fetched from the + Maven 2 repository on demand.</para> + </listitem> + </itemizedlist> + + <para>Web Beans includes a TCK runner that executes the TCK using Web + Beans as the CDI implementation and JBoss AS as the Java EE runtime. To + run the tck:</para> + + <itemizedlist> + <listitem> + <para>You need to install Maven. You can find documention on how to + install Maven 2 in the <ulink + url="http://www.sonatype.com/books/maven-book/reference/installation...: + The Definitive Guide</ulink> book published by Sonatype. Web Beans + bundles a copy of Maven in the <code>lib/maven</code> + directory.</para> + </listitem> + + <listitem> + <para>Next, instruct Maven to run the TCK:</para> + + <programlisting>cd jboss-tck-runner +mvn test -Dincontainer</programlisting> + </listitem> + + <listitem> + <para>TestNG will report, via Maven, the outcome of the run, and + report any failures on the console. Details can be found in + <code>target/surefire-reports/TestSuite.txt</code>.</para> + </listitem> + </itemizedlist> + </tip> + </section> + + <section id="eclipse-plugins"> + <title>Eclipse Plugins</title> + + <para>Eclipse, or any other IDE, is not required to execute or pass the + TCK. However an implementor may wish to execute tests in an IDE to aid + debugging the tests. This section introduces two essential Eclipse + plugins, TestNG and Maven 2, and points you to resources explaining how to + install them.</para> + + <section id="eclipse-testng-plugin"> + <title>TestNG Plugin</title> + + <para>The TCK is built on the JBoss Test Harness, which is in turn built + on TestNG. Therefore, having the TestNG plugin installed in Eclipse is + essential. Instructions for using the TestNG update site to add the + TestNG plugin to Eclipse are provided on the TestNG <ulink + url="http://testng.org/doc/download.html">download page</ulink>. You can + find a tutorial that explains how to use the TestNG plugin on the TestNG + <ulink url="http://testng.org/doc/eclipse.html">Eclipse + page</ulink>.</para> + </section> + + <section id="m2eclipse-plugin"> + <title>Maven 2 Plugin (m2eclipse)</title> + + <para>Another useful plugin is m2eclipse. Both the TCK project and Web + Beans are use Maven 2. Therefore, to work with these projects in + Eclipse, you may wish to have native support for Maven 2 projects, which + the m2eclipse plugin provides. Instructions for using the m2eclipse + update site to add the m2eclipse plugin to Eclipse are provided on the + m2eclipse home page. For more, read the m2eclipse <ulink + url="http://www.sonatype.com/books/m2eclipse-book/reference">... + guide</ulink>.</para> + + <para>m2eclipse is still a rather young project dealing with a complex + domain and you may run into problems using it. If that is the case, you + can alternatively use the Eclipse plugin for Maven 2 to generate native + Eclipse projects from Maven 2 projects.</para> + + <para>If you have Maven 2 installed, you have everything you need. Just + execute the following command from any Maven 2 project to produce the + Eclipse project files.</para> + + <programlisting>mvn eclipse:eclipse</programlisting> + </section> + + <para>Again, the Eclipse plugins are not required to execute the TCK, but + can be very helpful when validating an implementation against the TCK test + suite and especially when using the modules from the Hibernate Validator + project.</para> + </section> + + <!-- +vim: ts=3:sw=3:tw=80:set expandtab +--> +</chapter> Added: beanvalidation/tck/trunk/src/main/docbook/en-US/introduction.xml =================================================================== --- beanvalidation/tck/trunk/src/main/docbook/en-US/introduction.xml (rev 0) +++ beanvalidation/tck/trunk/src/main/docbook/en-US/introduction.xml 2009-09-29 16:34:58 UTC (rev 17578) @@ -0,0 +1,216 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" +"http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> +<chapter id="introduction"> + <title>Introduction (Bean Validation TCK)</title> + + <para>This chapter explains the purpose of a TCK and identifies the + foundation elements of the Bean Validation TCK.</para> + + <section> + <title>TCK Primer</title> + + <para>A TCK, or Technology Compatibility Kit, is one of the three required + pieces for any JSR (the other two being the specification document and the + reference implementation). The TCK is a set of tools and tests to verify + that an implementation of the technology conforms to the specification. + The tests are the primary component, but the tools serve an equally + critical role of providing a framework and/or set of SPIs for executing + the tests.</para> + + <para>The tests in the TCK are derived from assertions in the written + specification document. The assertions are itemized in an XML document, + where they each get assigned a unique identifier, and materialize as a + suite of automated tests that collectively validate whether an + implementation complies with the aforementioned assertions, and in turn + the specification. For a particular implementation to be certified, all of + the required tests must pass (i.e., the provided test suite must be run + unmodified).</para> + + <para>A TCK is entirely implementation agnostic. It should validate + assertions by consulting the specficiation's public API. </para> + </section> + + <section> + <title>Compatibility Testing</title> + + <para>The goal of any specification is to eliminate portability problems + so long as the program which uses the implementation also conforms to the + rules laid out in the specification.</para> + + <para>Executing the TCK is a form of compatibility testing. It's important + to understand that compatibility testing is distinctly different from + product testing. The TCK is not concerned with robustness, performance or + ease of use, and therefore cannot vouch for how well an implementation + meets these criteria. What a TCK can do is to ensure the exactness of an + implementation as it relates to the specification.</para> + + <para>Compatibility testing of any feature relies on both a complete + specification and a complete reference implementation. The reference + implementation demonstrates how each test can be passed and provides + additional context to the implementor during development for the + corresponding assertion.</para> + + <section> + <title>Why Compatibility Is Important</title> + + <para>Java platform compatibility is important to different groups + involved with Java technologies for different reasons:</para> + + <itemizedlist> + <listitem> + <para>Compatibility testing is the means by which the JCP ensures + that the Java platform does not become fragmented as it's ported to + different operating systems and hardware.</para> + </listitem> + + <listitem> + <para>Compatibility testing benefits developers working in the Java + programming language, enabling them to write applications once and + deploy them across heterogeneous computing environments without + porting.</para> + </listitem> + + <listitem> + <para>Compatibility testing enables application users to obtain + applications from disparate sources and deploy them with + confidence.</para> + </listitem> + + <listitem> + <para>Conformance testing benefits Java platform implementors by + ensuring the same extent of reliability for all Java platform + ports.</para> + </listitem> + </itemizedlist> + + <para>The Bean Validation specification goes to great lengths to ensure + that programs written for Java EE are compatible and the TCK is rigorous + about enforcing the rules the specification lays down.</para> + </section> + </section> + + <section> + <title>About the Bean Validation TCK</title> + + <para>The Bean Validation TCK is designed as a portable, configurable and + automated test suite for verifying the compatibility of an implementation + of the JSR-303: Bean Validation specification. The test suite is built + atop TestNG and provides a series of extensions that allow runtime + packaging and deployment of JEE artifacts for in-container testing (JBoss + Test Harness).</para> + + <note> + <para>The Bean Validation TCK harness is based on the JBoss Test + harness, which provides most of the aforementioned functionality.</para> + </note> + + <para>Each test class in the suite acts as a deployable unit. The + deployable units, or artifacts, are defined declaratively using + annotations. The artifact produced can be either a WAR or an EAR.</para> + + <para>The declarative approach allows many of the tests to be executed in + a standalone implementation of Bean Validation, accounting for a boast in + developer productivity. However, an implementation is only valid if all + tests pass using the in-container execution mode. The standalone mode is + merely a developer convenience.</para> + + <section> + <title>Bean Validation TCK Specifications and Requirements</title> + + <para>This section lists the applicable requirements and specifications + for the Bean Validation TCK.</para> + + <itemizedlist> + <listitem> + <para><emphasis role="bold">Bean Validation API</emphasis> - The + Java API defined in the Bean Validation specification and provided + by the reference implementation.</para> + </listitem> + + <listitem> + <para><emphasis role="bold">JBoss Test Harness</emphasis> - The Bean + Validation TCK requires version 1.0.0 of the JBoss Test Harness. The + Harness is based on TestNG 5.x (<ulink + url="http://testng.org">http://testng.org</ulink>). You can read + more about the harness in <xref linkend="test-harness" />.</para> + </listitem> + + <listitem> + <para><emphasis role="bold">TCK Audit Tool</emphasis> - An + itemization of the assertions in the specification documents which + are cross referenced by the individual tests. Describes how well the + TCK covers the specification.</para> + </listitem> + + <listitem> + <para><emphasis role="bold">Reference runtime</emphasis> - The + designated reference runtimes for compatibility testing of the Bean + Validation specification is the Sun Java Platform, Enterprise + Edition (Java EE) 6 reference implementation (RI). See details at + Java EE 6 (<ulink + url="http://java.sun.com/javaee/6/docs/api/">http://java.sun... + </listitem> + </itemizedlist> + </section> + + <section> + <title>Bean Validation TCK Components</title> + + <para>The Bean Validation TCK includes the following components:</para> + + <itemizedlist> + <listitem> + <para><emphasis role="bold">JBoss Test Harness 1.0.0</emphasis> and + related documentation.</para> + </listitem> + + <listitem> + <para><emphasis role="bold">TestNG 5.9</emphasis>, the testing + framework on which the JBoss Test Harness is based and which + provides the extension points for selecting an executing the tests + in the test suite.</para> + </listitem> + + <listitem> + <para><emphasis role="bold">The test suite</emphasis>, which is a + collection of TestNG tests, the TestNG test suite descriptor and + supplemental resources that configure CDI and other software + components.</para> + </listitem> + + <listitem> + <para><emphasis role="bold">The TCK audit</emphasis> is used to list + out the assertions identified in the CDI specification. It matches + the assertions to testcases in the test suite by unique identifier + and produces a coverage report.</para> + + <para>The audit document is provided along with the TCK. Each + assertion is defined with a reference to a chapter, section and + paragraph from the specification document, making it easy for the + implementor to locate the language in the specification document + that supports the feature being tested.</para> + </listitem> + + <listitem> + <para><emphasis role="bold">TCK documentation</emphasis> accompanied + by release notes identifying updates between versions.</para> + </listitem> + </itemizedlist> + + <para>The Bean Validation TCK has been tested run on following + platforms:</para> + + <itemizedlist> + <listitem> + <para>JBoss AS 5.0.0.GA using Sun Java SE 6 on Red Hat Enterprise + Linux 5.2</para> + </listitem> + </itemizedlist> + </section> + </section> + + <!-- +vim: ts=3:sw=3:tw=80:set expandtab +--> +</chapter> Added: beanvalidation/tck/trunk/src/main/docbook/en-US/master.xml =================================================================== --- beanvalidation/tck/trunk/src/main/docbook/en-US/master.xml (rev 0) +++ beanvalidation/tck/trunk/src/main/docbook/en-US/master.xml 2009-09-29 16:34:58 UTC (rev 17578) @@ -0,0 +1,12 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [ ]> +<book lang="en"> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Book_Info.xml"/> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Book_Preface.xml"/> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="part-background.xml"/> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="part-execution.xml"/> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="part-test-harness.xml" /> +<!-- +vim: ts=3:sw=3:tw=80:set expandtab +--> +</book> Added: beanvalidation/tck/trunk/src/main/docbook/en-US/part-background.xml =================================================================== --- beanvalidation/tck/trunk/src/main/docbook/en-US/part-background.xml (rev 0) +++ beanvalidation/tck/trunk/src/main/docbook/en-US/part-background.xml 2009-09-29 16:34:58 UTC (rev 17578) @@ -0,0 +1,38 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE part PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" +"http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> +<part> + <title>Getting Acquainted with the TCK</title> + + <partintro> + <para>The Bean Validation TCK must be used to ensure that your + implementation conforms to the Bean Validation specification. This part + introduces the TCK, gives some background about its purpose, states the + requirements for passing the TCK and outlines the appeals process.</para> + + <para>In this part you will learn where to obtain the Bean Validation TCK + and supporting software. You are then presented with recommendations of + how to organize and configure the software so that you are ready to + execute the TCK.</para> + + <para>Finally, it discusses the reporting provided by the TCK.</para> + </partintro> + + <xi:include href="introduction.xml" + xmlns:xi="http://www.w3.org/2001/XInclude" /> + + <xi:include href="appeals-process.xml" + xmlns:xi="http://www.w3.org/2001/XInclude" /> + + <xi:include href="installation.xml" + xmlns:xi="http://www.w3.org/2001/XInclude" /> + + <xi:include href="configuration.xml" + xmlns:xi="http://www.w3.org/2001/XInclude" /> + + <xi:include href="reporting.xml" xmlns:xi="http://www.w3.org/2001/XInclude" /> + + <!-- +vim: ts=3:sw=3:tw=80:set expandtab +--> +</part> Added: beanvalidation/tck/trunk/src/main/docbook/en-US/part-execution.xml =================================================================== --- beanvalidation/tck/trunk/src/main/docbook/en-US/part-execution.xml (rev 0) +++ beanvalidation/tck/trunk/src/main/docbook/en-US/part-execution.xml 2009-09-29 16:34:58 UTC (rev 17578) @@ -0,0 +1,27 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE part PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" +"http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> +<part> + <title>Executing and Debugging Tests</title> + + <partintro> + <para>In this part you learn how to execute the Bean Validation TCK on the + Bean validation reference implementation (Hibernate Validator). First, you + are walked through the steps necessary to execute the test suite on + Hibernate Validator. Then you discover how to modify the TCK runner to + execute the test suite on your own implementation. Finally, you learn how + to debug tests from the test suite in Eclipse.</para> + </partintro> + + <xi:include href="executing.xml" xmlns:xi="http://www.w3.org/2001/XInclude" /> + + <xi:include href="eclipse-running.xml" + xmlns:xi="http://www.w3.org/2001/XInclude" /> + + <xi:include href="eclipse-debugging.xml" + xmlns:xi="http://www.w3.org/2001/XInclude" /> + + <!-- +vim: ts=3:sw=3:tw=80:set expandtab +--> +</part> Added: beanvalidation/tck/trunk/src/main/docbook/en-US/part-test-harness.xml =================================================================== --- beanvalidation/tck/trunk/src/main/docbook/en-US/part-test-harness.xml (rev 0) +++ beanvalidation/tck/trunk/src/main/docbook/en-US/part-test-harness.xml 2009-09-29 16:34:58 UTC (rev 17578) @@ -0,0 +1,18 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE part PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [ ]> +<part id="test-harness"> + <title>JBoss Test Harness</title> + <partintro> + <para> + In this part you learn about the JBoss Test Harness through selected + chapters from the JBoss Test Harness Reference Guide. You can view the + entire JBoss Test Harness Reference Guide at <ulink url="">TODO</ulink>. + </para> + </partintro> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="harness/introduction.xml" /> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="harness/configuration.xml" /> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="harness/executing.xml" /> +<!-- +vim: ts=3:sw=3:tw=80:set expandtab +--> +</part> Added: beanvalidation/tck/trunk/src/main/docbook/en-US/reporting.xml =================================================================== --- beanvalidation/tck/trunk/src/main/docbook/en-US/reporting.xml (rev 0) +++ beanvalidation/tck/trunk/src/main/docbook/en-US/reporting.xml 2009-09-29 16:34:58 UTC (rev 17578) @@ -0,0 +1,332 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" +"http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> +<chapter id="reporting"> + <title>Reporting</title> + + <para>This chapter covers the two types of reports that can be generated + from the TCK, an assertion coverage report and the test execution results. + The chapter also justifies why the TCK is good indicator of how accurately + an implementation conforms to the JSR-303 specification.</para> + + <section> + <title>Bean Validation TCK Coverage Metrics</title> + + <para>The Bean Validation TCK coverage has been measured as + follows:</para> + + <itemizedlist> + <listitem> + <para><emphasis role="bold">Assertion Breadth + Coverage</emphasis></para> + + <para>The CDI TCK provides at least 75% coverage of identified + assertions with test cases.</para> + </listitem> + + <listitem> + <para><emphasis role="bold">Assertion Breadth Coverage + Variance</emphasis></para> + + <para>The coverage of specification sub-sections shows at least a + normal distribution (centered around 75%).</para> + </listitem> + + <listitem> + <para><emphasis role="bold">Assertion Depth Coverage</emphasis></para> + + <para>The assertion depth coverage has not been measured, as, when an + assertion requires more than one testcase, these have been enumerated + in an assertion group and so are adequately described by the assertion + breadth coverage.</para> + </listitem> + + <listitem> + <para><emphasis role="bold">Method Coverage</emphasis></para> + + <para>TODO</para> + </listitem> + + <listitem> + <para><emphasis role="bold">API Signature Coverage</emphasis></para> + + <para>The CDI TCK covers 100% of all API public methods using the Java + CTT Sig Test tool.</para> + </listitem> + </itemizedlist> + </section> + + <section> + <title>Bean Validation TCK Coverage Report</title> + + <para>A specification can be distilled into a collection of assertions + that define the behavior of the software. This section introduces the Bean + Validation TCK coverage report, which documents the relationship between + the assertions that have been identified in the JSR-303 specification + document and the tests in the TCK test suite.</para> + + <para>The structure of this report is controlled by the assertion + document, so we'll start there.</para> + + <section> + <title>Bean Validation TCK Assertions</title> + + <para>The Bean Validation TCK developers have analyzed the JSR-303 + specification document and identified the assertions that are present in + each chapter. Here's an example of one such assertion found in section + 2.1:</para> + + <blockquote> + Every constraint annotation must define a message element of type String + </blockquote> + + <para>The assertions are listed in the XML file + src/main/resources/tck-audit.xml in the Bean Validation TCK + distribution. Each assertion is identified by the section of the + specification document in which it resides and assigned a unique + paragraph identifier to narrow down the location of the assertion + further. To continue with the example, the assertion shown above is + listed in the tck-audit.xml file using this XML fragment:</para> + + <programlisting role="XML">&lt;section id="2.1.1" title="Constraint definition properties"&gt; + ... + &lt;assertion id="c"&gt; + &lt;text&gt;Every constraint annotation must define a message element of type String&lt;/text&gt; + &lt;/assertion&gt; + ... +&lt;/section&gt;</programlisting> + + <para>The strategy of the Bean Validation TCK is to write a test which + validates this assertion when run against an implementation. A test case + (a method annotated with <literal>@Test</literal> in an + <literal>@Artifact</literal> class) is correlated with an assertion + using the + <literal&gt;(a)org.jboss.test.audit.annotations.SpecAssertion&lt;/literal&gt; + annotation as follows:</para> + + <programlisting role="JAVA">@Test +@SpecAssertion(section = "2.1.1", id = "c") +public void testConstraintDefinitionWithoutMessageParameter() { + try { + Validator validator = TestUtil.getValidatorUnderTest(); + validator.validate( new DummyEntityNoMessage() ); + fail( "The used constraint does not define a message parameter. The validation should have failed." ); + } + catch ( ConstraintDefinitionException e ) { + // success +} +</programlisting> + + <para>To help evaluate the distribution of coverage for these + assertions, the TCK provides a detailed coverage report. This report is + also useful to help implementors match tests with the language in the + specification that supports the behavior being tested.</para> + </section> + + <section> + <title>Producing the Coverage Report</title> + + <para>The coverage report is an HTML report generated as part of the TCK + project build. Specifically, it is generated by an annotation processor + that attaches to the compilation of the classes in the TCK test suite, + another tool from the JBoss Test Utils project. You can enable this + report by setting the commandline property tck-audit to true when + running the Maven 2 build in the tck directory.</para> + + <programlisting>mvn clean install -Dtck-audit=true</programlisting> + + <note> + <para>You must run clean first because the annotation processor + performs it's work when the test class is being compiled. If + compilation is unnecessary, then the assertions referenced in that + class will not be discovered.</para> + </note> + + <para>The report is written to the file target/coverage.html in the same + project. The report has five sections:</para> + + <orderedlist> + <listitem> + <para><emphasis role="bold">Chapter Summary</emphasis> - List the + chapters (that contain assertions) in the specification document + along with total assertions, tests and coverage percentage.</para> + </listitem> + + <listitem> + <para><emphasis role="bold">Section Summary</emphasis> - Lists the + sections (that contain assertions) in the specification document + along with total assertions, tests and coverage percentage.</para> + </listitem> + + <listitem> + <para><emphasis role="bold">Coverage Detail</emphasis> - Each + assertion and the test that covers it, if any.</para> + </listitem> + + <listitem> + <para><emphasis role="bold">Unmatched Tests</emphasis> - A list of + tests for which there is no matching assertion (useful during TCK + development).</para> + </listitem> + + <listitem> + <para><emphasis role="bold">Unversioned Tests</emphasis> - A list of + tests for which there is no <literal>@SpecVersion</literal> + annotation on the test class (useful during TCK development).</para> + </listitem> + </orderedlist> + + <para>The coverage report is color coded to indicate the status of an + assertion, or group of assertions. The status codes are as + follows:</para> + + <itemizedlist> + <listitem> + <para><emphasis role="bold">Covered</emphasis> - a test exists for + this assertion</para> + </listitem> + + <listitem> + <para><emphasis role="bold">Not covered</emphasis> - no test exists + for this assertion</para> + </listitem> + + <listitem> + <para><emphasis role="bold">Problematic</emphasis> - a test exists, + but is currently disabled. For example, this may be because the test + is under development</para> + </listitem> + + <listitem> + <para><emphasis role="bold">Untestable</emphasis> - the assertion + has been deemed untestable, a note, explaining why, is normally + provided</para> + </listitem> + </itemizedlist> + + <para>For reasons provided in the <literal>tck-audit.xml</literal> + document and presented in the coverage report, some assertions are not + testable.</para> + + <para>The coverage report does not give any indication as to whether the + tests are passing. That's where the TestNG reports come in.</para> + </section> + + <section> + <title>TestNG Reports</title> + + <para>As you by now, the CDI TCK test suite is really just a TestNG test + suite. That means an execution of the CDI TCK test suite produces all + the same reports that TestNG produces. This section will go over those + reports and show you were to go to find each of them.</para> + + <section> + <title>Maven 2, Surefire and TestNG</title> + + <para>When the CDI TCK test suite is executed during the Maven 2 test + phase of the TCK runner project, TestNG is invoked indirectly through + the Maven Surefire plugin. Surefire is a test execution abstraction + layer capable of executing a mix of tests written for JUnit, TestNG, + and other supported test frameworks.</para> + + <para>Why is this relevant? It means two things. First, it means that + you are going to get a summary of the test run on the commandline. + Here's the output generated when the tests are run using standalone + mode.</para> + + <programlisting>------------------------------------------------------- + T E S T S +------------------------------------------------------- +Running TestSuite +Tests run: 237, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.062 sec + +Results : + +Tests run: 237, Failures: 0, Errors: 0, Skipped: 0 +</programlisting> + + <note> + <para>The number of tests executed, the execution time, and the + output will differ when you run the tests using in-container mode as + the CDI TCK requires.</para> + </note> + + <para>If the Maven reporting plugin that compliments Surefire is + configured properly, Maven will also generate a generic HTML test + result report. That report is written to the file test-report.html in + the target/surefire-reports directory of the TCK runner project. It + shows how many tests were run, how many failed and the success rate of + the test run.</para> + + <para>The one drawback of the Maven Surefire report plugin is that it + buffers the test failures and puts them in the HTML report rather than + outputting them to the commandline. If you are running the test suite + to determine if there are any failures, it may be more useful to get + this information in the foreground. You can prevent the failures from + being redirected to the report using the following commandline + switch:</para> + + <programlisting>mvn test -Dsurefire.useFile=false</programlisting> + + <para>The information that the Surefire provides is fairly basic and + the detail pales in comparison to what the native TestNG reports + provide.</para> + </section> + + <section> + <title>TestNG HTML Reports</title> + + <para>TestNG produces several HTML reports for a given test run. All + the reports can be found in the target/surefire-reports directory in + the TCK runner project. Below is a list of the three types of + reports:</para> + + <itemizedlist> + <listitem> + <para>Test Summary Report</para> + </listitem> + + <listitem> + <para>Test Suite Detail Report</para> + </listitem> + + <listitem> + <para>Emailable Report</para> + </listitem> + </itemizedlist> + + <para>The first report, the test summary report, show below, is + written to the file index.html. It produces the same information as + the generic Surefire report.</para> + + <graphic align="center" fileref="images/testng-summary-report.png" /> + + <para>The summary report links to the test suite detail report, which + has a wealth of information. It shows a complete list of test groups + along with the classes in each group, which groups were included and + excluded, and any exceptions that were raised, whether from a passed + or failed test. A partial view of the test suite detail report is + shown below.</para> + + <graphic align="center" + fileref="images/testng-suite-detail-report.png" /> + + <para>The test suite detail report is very useful, but it borderlines + on complex. As an alternative, you can have a look at the emailable + report, which is a single HTML document that shows much of the same + information as the test suite detail report in a more compact layout. + A partial view of the emailable report is shown below.</para> + + <graphic align="center" fileref="images/testng-emailable-report.png" /> + + <para>Now that you have seen two ways to get test results from the + Maven test execution, let's switch over to the IDE, specifically + Eclipse, and see how it presents TestNG test results.</para> + </section> + </section> + </section> + + <!-- +vim: ts=3:sw=3:tw=80:set expandtab +--> +</chapter>