[jboss-dev] New Break the Build Policy - testsuite - discussion

[Adrian Brock]

As many of you probably remember during the AS5 development
the testsuite ended up in a horrible state.

This problem was due to a snowball effect. Once one person
broke a few important tests, other developers didn't know
whether their changes also broke things.
I also suspect that people stopped caring about the testsuite
because they knew it was already broken.

We already have a policy that anybody who breaks
the compile or one of the smoke tests can expect to have
their commit rolled back.

Now, we want to extend the policy to the testsuite
as a whole.

The basic policy will be that once we have the Hudson
build running reliably at 100% any commit that breaks
a test can expect to be rolled back.

A branch will be created for the broken build
before rolling back so the commit(s) can be fixed
and tested before re-merging.

To keep things simple, rollbacks will be to the
revision that Hudson says was ok. This means
somebody fixing the build doesn't have to waste
time trying to analyse who broke the build
amongst multiple commits.

Obviously some common sense will be applied.

* If the fix is simple then it should be done
rather than rolling back

* The committer will be given a reasonable
amount time (e.g. until the next Hudson build)
to solve the issue

* We aren't going to rollback if it looks
like a spurious problem with the Hudson build

* If the failing test is one of those
known to be brittle.
e.g. some of the timer or asynchronous tests can fail 
just because their wait()s are too short.
But these tests should be fixed anyway to
make them less prone to false negatives.
-- 
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Adrian Brock
Chief Scientist
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx