One of the primary objectives of this ongoing release is to support more of the JCR API, including some of the more basic update features. As part of this effort, we've needed a way to verify that the parts of our implementation we've been working on are behaving correctly per the specification. We continue to rely upon our own unit tests, but we've also started using a unit test JAR from the Jackrabbit project, which includes 1000+ unit tests that verify the compatibility of any JCR implementation.

To keep things simple, we wanted to include these tests within our Maven builds, whether those are local builds run by developers or builds run by our continuous integration server. That means that we only want to include those tests that we're actually passing (otherwise the builds will break). Once we implement a piece of behavior, we uncomment those tests; any regression is automatically identified.

We've made a lot of progress this way, and it's a great approach for those of us down in the trenches. However, we'd also like to show at a higher level our progress towards JCR API compliance, and that includes the failures as well as the successes.

Well, to make a long story short, we now have a Hudson build that runs these "TCK unit tests". Right now, this build is running every Monday-Friday in the wee hours of the morning, and the latest results are always available here:

http://hudson.jboss.org/hudson/job/DNA%20JCR%20API%20compatibility%20on%20JDK1.5/org.jboss.dna$dna-jcr-tck/lastBuild/testReport/org.jboss.dna.jcr/

Hudson's formatting isn't great, but the top of that page shows that we're passing 35% of the tests (actually, it shows that we're failing 683 of the 1045 tests). However, at the bottom of that page (below all the individual failures) are some more useful statistics, including a breakdown of the status for each compliance level.

Current status:
- 90.3% pass rate on Level 1
- 12.1% pass rate on Level 2