[jboss-dev] Testing (Was: Re: M2 Status)

[Brian Stansberry]

Copying from below:

> Stan Silvert wrote:
>The testsuite is there for regression testing.

True; the reason we run it over and over is for regression testing.

But regression testing isn't the only reason for testing. You're right; 
most unit and functional tests I write never fail after I commit them. 
But they fail *a lot* before I commit them and the now better-working 
code. Maybe I'm a crappy programmer. :) But I just don't think it's 
possible in practice to adequately test complex logic solely by creating 
deployments and remote request patterns that exercise the logic. It's 
too painful, so things end up being left untested.

I didn't advocate abandoning integration testing. I agree with you on 
its benefits. I did before, but even if I didn't the many trunk 
regressions the testsuite caught leading up to M2 would have convinced me.

One thing that should help bridge the gap is the modularization of the 
AS. Individual components can have lightweight testsuites that don't 
have to be run every time the AS testsuite runs, leaving the overall AS 
testsuite run more focused on integration.

On 02/16/2010 10:15 AM, Jason T. Greene wrote:
> A couple of thoughts on this:
>
> 1) The testsuite needs to run in a reasonable amount of time, else no
> one will run it (already a problem today). We need to think about ways
> to get the main testsuite down to under an hour. This likely means
> eliminating stress tests, which really don't belong.
>
> 2) Tests need to be useful and maintainable. We have several tests that
> are old, false alarms, or brittle.
>
> Stan Silvert wrote:
>> Brian Stansberry wrote:
>>> Yes and no.
>>>
>>> Yes, I definitely agree that the AS testsuite could use beefing up in
>>> terms of integration testing of some of the stuff the AS brings in.
>>>
>> +1
>>> High level
>>> integration tests can catch that by luck, but IMHO much better is closer
>>> unit testing of the deployer class where the problem was. In general, my
>>> impression is the AS testsuite is overly reliant on integration testing
>>> and doesn't have enough low level unit testing.
>>>
>> -1
>>
>> The testsuite is there for regression testing.  An effective regression
>> test is one that uncovers a bug.  A good regression test is one that
>> fails.  In this case, the integration test did that and it wasn't just
>> dumb luck.  I would submit that as regression tests go, integration
>> tests find bugs far more often then their low-level unit-testing
>> counterparts.
>>
>> Unit tests are written to pass.  The person writing the test also wrote
>> the code.  And good programmers tend to write code that works.
>>
>> If done right, unit tests do help a programmer figure out what his API
>> needs to look like (TDD).  They also serve as a sanity check for that
>> programmer to make sure his small change didn't mess anything up.  But
>> because they are so fine-grained and isolated, they are unlikely to fail
>> when later run in the testsuite for regression purposes.
>>
>> The good test (one more likely to fail), is the one that combines the
>> work of two or more programmers.  That is what integration tests do.  We
>> need more emphasis on integration tests, not less.
>>
>> Furthermore, an integration test goes through real use cases.  It makes
>> sure that at the end of the day, the system can actually do something
>> useful.  And that's the best test of all.
>>> On 02/16/2010 06:26 AM, Dimitris Andreadis wrote:
>>>
>>>> For future releases, shouldn't we enhance the scope of our standard testsuite runs to
>>>> uncover issues earlier? Better testing of components like seam/weld/jopr/rest-easy/etc.
>>>>
>>>> Brian Stansberry wrote:
>>>>
>>>>> Just an FYI on the status of the 6.0.0.M2 release. Rajesh Rajasekaran,
>>>>> Martin Gencur and the QE folks are doing a great job (thanks!) and found
>>>>> a couple critical issues not exposed by the testsuite. They proved easy
>>>>> to fix, and are fixed, but it's delayed things by a day.
>>>>>
>>>>>
>>>> _______________________________________________
>>>> jboss-development mailing list
>>>> jboss-development at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/jboss-development
>>>>
>>>
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> jboss-development mailing list
>> jboss-development at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/jboss-development
>
>


-- 
Brian Stansberry
Lead, AS Clustering
JBoss by Red Hat