Smoke tests - Re: [jboss-dev] Re: Big AS testsuite regression

[Bill Burke]

IMO a 5 min smoke test (+ all unit tests in the area of the commit) is 
reasonable for *any* commit.

Any regression should result in an immediate SVN freeze until the 
regressions are fixed.

Adrian Brock wrote:
> I've made a start on this "feature" since we've
> talking about it for some time and nobody has done anything. ;-)
> 
> See:
> http://jira.jboss.com/jira/browse/JBAS-5582
> 
> This could obviously do we some refining and probably some
> of the existing tests splitting up to extra the basic
> standup tests from the other more complicated ones.
> 
> I basically took the tests that I normally use as
> "standup tests" when I do major changes.
> 
> It's hard to tell how long this will take, but with the
> currently broken state it took less than 5 minutes
> (including starting and stopping the server)
> so there can't be any real complaints about not running
> this on all major commits. ;-)
> 
> On Wed, 2008-06-04 at 15:22 +0300, Dimitris Andreadis wrote:
>> True, you could run some basic smoke tests first.
>>
>> Sacha Labourey wrote:
>>> OK, I see. Still, here, we are speaking about +1162 failures, that 
>>> should be easy to detect just by starting any AS+app, no? ;)
>>>
>>> On 06/04/2008 02:15 PM, Dimitris Andreadis wrote:
>>>> I suppose because it's hard for individual users to make a full local 
>>>> testsuite run before every commit. It takes 5-6 hours.
>>>>
>>>> I'd be happy if people would monitor subsequent hudson runs to see the 
>>>> effects of their commits.
>>>>
>>>> The AS5 testsuite is now more or less stable/predictable so it's easy 
>>>> to spot regression.
>>>>
>>>> Sacha Labourey wrote:
>>>>> why aren't we able to catch those regression *before* we commit them 
>>>>> to SVN?
>>>>>
>>>>> On 06/04/2008 12:50 PM, Alexey Loubyansky wrote:
>>>>>> Yes, it comes from metadata. The tests fail to deploy. Emanuel is 
>>>>>> going to look into it.
>>>>>>
>>>>>> Caused by: java.lang.IllegalArgumentException: Can't find interface 
>>>>>> null in 
>>>>>> org.jboss.metadata.ejb.jboss.JBossSessionBeanMetaData at af9ed402{AuditSessionEJB} 
>>>>>>
>>>>>>     at 
>>>>>> org.jboss.metadata.ejb.jboss.JBossEnterpriseBeanMetaData.determineResolvedJndiName(JBossEnterpriseBeanMetaData.java:702) 
>>>>>>
>>>>>>     at 
>>>>>> org.jboss.deployment.MappedReferenceMetaDataResolverDeployer.mapEjbs(MappedReferenceMetaDataResolverDeployer.java:332) 
>>>>>>
>>>>>>     at 
>>>>>> org.jboss.deployment.MappedReferenceMetaDataResolverDeployer.mapEndpoints(MappedReferenceMetaDataResolverDeployer.java:219) 
>>>>>>
>>>>>>     at 
>>>>>> org.jboss.deployment.MappedReferenceMetaDataResolverDeployer.internalDeploy(MappedReferenceMetaDataResolverDeployer.java:140) 
>>>>>>
>>>>>>     at 
>>>>>> org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:50) 
>>>>>>
>>>>>>     at 
>>>>>> org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:174) 
>>>>>>
>>>>>>
>>>>>>
>>>>>> Dimitris Andreadis wrote:
>>>>>>> I suspect this huge regression (+1162 failures) is related to the 
>>>>>>> recent jboss-metadata update:
>>>>>>>
>>>>>>> http://hudson.qa.jboss.com/hudson/view/JBoss%20AS/job/JBoss-AS-5.0.x-TestSuite-sun15/ 
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> jboss-development mailing list
>>>>>> jboss-development at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-development
>> _______________________________________________
>> jboss-development mailing list
>> jboss-development at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/jboss-development

-- 
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com