A couple of thoughts on this:
1) The testsuite needs to run in a reasonable amount of time, else no
one will run it (already a problem today). We need to think about ways
to get the main testsuite down to under an hour. This likely means
eliminating stress tests, which really don't belong.
2) Tests need to be useful and maintainable. We have several tests that
are old, false alarms, or brittle.
Stan Silvert wrote:
Brian Stansberry wrote:
> Yes and no.
>
> Yes, I definitely agree that the AS testsuite could use beefing up in
> terms of integration testing of some of the stuff the AS brings in.
>
+1
> High level
> integration tests can catch that by luck, but IMHO much better is closer
> unit testing of the deployer class where the problem was. In general, my
> impression is the AS testsuite is overly reliant on integration testing
> and doesn't have enough low level unit testing.
>
-1
The testsuite is there for regression testing. An effective regression
test is one that uncovers a bug. A good regression test is one that
fails. In this case, the integration test did that and it wasn't just
dumb luck. I would submit that as regression tests go, integration
tests find bugs far more often then their low-level unit-testing
counterparts.
Unit tests are written to pass. The person writing the test also wrote
the code. And good programmers tend to write code that works.
If done right, unit tests do help a programmer figure out what his API
needs to look like (TDD). They also serve as a sanity check for that
programmer to make sure his small change didn't mess anything up. But
because they are so fine-grained and isolated, they are unlikely to fail
when later run in the testsuite for regression purposes.
The good test (one more likely to fail), is the one that combines the
work of two or more programmers. That is what integration tests do. We
need more emphasis on integration tests, not less.
Furthermore, an integration test goes through real use cases. It makes
sure that at the end of the day, the system can actually do something
useful. And that's the best test of all.
> On 02/16/2010 06:26 AM, Dimitris Andreadis wrote:
>
>> For future releases, shouldn't we enhance the scope of our standard testsuite
runs to
>> uncover issues earlier? Better testing of components like
seam/weld/jopr/rest-easy/etc.
>>
>> Brian Stansberry wrote:
>>
>>> Just an FYI on the status of the 6.0.0.M2 release. Rajesh Rajasekaran,
>>> Martin Gencur and the QE folks are doing a great job (thanks!) and found
>>> a couple critical issues not exposed by the testsuite. They proved easy
>>> to fix, and are fixed, but it's delayed things by a day.
>>>
>>>
>> _______________________________________________
>> jboss-development mailing list
>> jboss-development(a)lists.jboss.org
>>
https://lists.jboss.org/mailman/listinfo/jboss-development
>>
>
>
>
------------------------------------------------------------------------
_______________________________________________
jboss-development mailing list
jboss-development(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jboss-development
--
Jason T. Greene
JBoss, a division of Red Hat