On 9/1/11 4:15 AM, Carlo de Wolf wrote:
Another concern has cropped, which has to do with the size (and speed
of
building) of the code base.
That's certainly a reasonable concern. We may have to break things up
further just due to practical side effects of developer tooling. IDEs,
build tools, testsuites etc struggle at large sizes like the AS code
base. I'm hopeful we will maintain a reasonable balance.
At some point (probably after EAP6), I hope we can explore a parallel
testsuite. When I was working on infinispan / boss cache we were able to
reduce the testsuite from hours to around 5-10 minutes by making this
switch. It was not however an easy migration.
As an example (and case) the CalendarBasedTimeoutTestCase is now
disabled.
https://github.com/jbossas/jboss-as/commit/2d04329f4c7ae790e711
At first glance this test looks like a functional test, but it's so
exhaustive that it acts like a stress test. It's ran against every
timezone in the system, and in my system I have 608. It shouldn't be a
big deal to modify this test to execute in a reasonable time frame (at
least by default). During development I sometimes do one-off exhaustive
tests like this, but it's not practical to put them in a test suite. For
example, I tend to run anything that does bytecode analysis against
every class file on my hard drive. There is, of course, nothing wrong
with writing exhaustive tests that can be saved and executed manually.
Maybe we should have a special testsuite module for this?
[NOTE: I would support splitting off generic time math into a separate
library. It's something that could be useful to other projects, and it's
one of those things that once you get it right, you shouldn't need to
update it much]
We could enable this test during an allTests run, but that would move
the burden to somebody checking the Hudson runs.
In general all of the functional testing (includes integration tests
smoke tests etc) should be quick, or at least execute in reasonable time
frames. All slower tests should be moved to a separate area, and they
should require extreme justification. Stress testing is something we
should need to run less often, and we can rely on a less frequent hudson
run for those kind of tests.
Ultimo having too many tests in a single code base leads to issues
like
https://issues.jboss.org/browse/AS7-1689. Which brings back fond
memories of the AS4 code base. ;-)
Is that the right link? I don't see how the test failing in that issue
has anything to do with which codebase it is in.
How are we going to deal with this in general?
I think we need to do the following:
- Keep functional tests efficient and well organized
- Possibly create an area for manual exhaustive tests
- Explorer parallel testing for at least groups of tests that involve
blocking
- Create a separate hudson run for stress testing
--
Jason T. Greene
JBoss AS Lead / EAP Platform Architect
JBoss, a division of Red Hat