[dna-dev] Nightly status of JBoss DNA's compatibility with the JCR API

[Randall Hauch]

One of the primary objectives of this ongoing release is to support  
more of the JCR API, including some of the more basic update  
features.  As part of this effort, we've needed a way to verify that  
the parts of our implementation we've been working on are behaving  
correctly per the specification.  We continue to rely upon our own  
unit tests, but we've also started using a unit test JAR from the  
Jackrabbit project, which includes 1000+ unit tests that verify the  
compatibility of any JCR implementation.

To keep things simple, we wanted to include these tests within our  
Maven builds, whether those are local builds run by developers or  
builds run by our continuous integration server.  That means that we  
only want to include those tests that we're actually passing  
(otherwise the builds will break).  Once we implement a piece of  
behavior, we uncomment those tests; any regression is automatically  
identified.

We've made a lot of progress this way, and it's a great approach for  
those of us down in the trenches.  However, we'd also like to show at  
a higher level our progress towards JCR API compliance, and that  
includes the failures as well as the successes.

Well, to make a long story short, we now have a Hudson build that runs  
these "TCK unit tests".  Right now, this build is running every Monday- 
Friday in the wee hours of the morning, and the latest results are  
always available here:

http://hudson.qa.jboss.com/hudson/view/DNA/job/DNA%20JCR%20API%20compatibility%20on%20JDK1.5/org.jboss.dna$dna-jcr-tck/lastBuild/testReport/org.jboss.dna.jcr/

Hudson's formatting isn't great, but the top of that page shows that  
we're passing 35% of the tests (actually, it shows that we're failing  
683 of the 1045 tests).  However, at the bottom of that page (below  
all the individual failures) are some more useful statistics,  
including a breakdown of the status for each compliance level.

Current status:
- 90.3% pass rate on Level 1
- 12.1% pass rate on Level 2

Watch this space as we quickly start increasing the Level 2 rate!

Best regards,

Randall


P.S.  The TCK unit tests are not included in the Maven builds by  
default.  To run them locally, specify the "jcr-tck" profile.  For  
example:
	$ mvn -P jcr-tck clean install