Those are different modules though.  What about overall?  One explanation could be that an entire module fails to build so its tests aren't counted in the overall count. On 1 Jun 2012, at 00:59, Sanne Grinovero wrote: > Hi all, > > I just noticed the number of tests reported by CI varies significantly > across builds. > > I'm not referring to the number of failures or errors, but the total > number of tests executed; for example, it seems the last build had 249 > less than the build from yesterday. > > It just happens that we have nice graphs to look at, and some are rock-stable: > > https://infinispan.ci.cloudbees.com/job/Infinispan-master-JDK6-tcp/org.in... > > while others are rock-as-mountains ?? > > https://infinispan.ci.cloudbees.com/job/Infinispan-master-JDK6-tcp/org.in... > > Is anyone having a reassuring explanation? > > Cheers, > Sanne > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org twitter.com/maniksurtani Lead, Infinispan http://www.infinispan.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Could be due to a bad check-in where someone annotates a test with @Test(invocationCount = xxxx) - I have seen this a few times.

TestNG/surefire in combo are dreadful at counting the number of tests. If you get errors, it will start counting failed before/after methods as well as real test methods. You basically shouldn't trust the count from any run except one where every test passed. I would strongly advise using JUnit for this, and many other reasons: * much better hooks into surefire, eclipse etc. etc. * much more stable (rarely do I see bugs like this in it) * you can actually debug the JUnit code without going blind * much more flexible / extensible, due to use of runners BUT it doesn't have the built in features TestNG has... On 1 Jun 2012, at 10:59, Sanne Grinovero wrote: > On 1 June 2012 10:54, Manik Surtani <manik(a)jboss.org&gt; wrote: >> Could be due to a bad check-in where someone annotates a test with @Test(invocationCount = xxxx) - I have seen this a few times. > > Maybe, but you didn't convince me. This picture is way too unstable > (overall testsuite) : > > https://infinispan.ci.cloudbees.com/job/Infinispan-master-JDK6-tcp/test/?... > > I'd rather think surefire / jenkins / testng / our listeners can't > count correctly. > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Hmm - how complex/time consuming a changeover do you reckon this would be? On 1 Jun 2012, at 12:07, Sanne Grinovero wrote: > Thanks Pete! > that must be the reason; I'm glad nobody deleted 200 tests yesterday ;-) > > I would prefer JUnit too, definitely more reliable and I'm loving the > latest features of it such as @Rule: makes it almost impossible to > forget proper test clean-up and fights inheritance complexity. > > Cheers, > Sanne > > On 1 June 2012 11:56, Pete Muir <pmuir(a)redhat.com&gt; wrote: >> TestNG/surefire in combo are dreadful at counting the number of tests. If you get errors, it will start counting failed before/after methods as well as real test methods. You basically shouldn't trust the count from any run except one where every test passed. >> >> I would strongly advise using JUnit for this, and many other reasons: >> >> * much better hooks into surefire, eclipse etc. etc. >> * much more stable (rarely do I see bugs like this in it) >> * you can actually debug the JUnit code without going blind >> * much more flexible / extensible, due to use of runners >> >> BUT it doesn't have the built in features TestNG has... >> >> On 1 Jun 2012, at 10:59, Sanne Grinovero wrote: >> >>> On 1 June 2012 10:54, Manik Surtani <manik(a)jboss.org&gt; wrote: >>>> Could be due to a bad check-in where someone annotates a test with @Test(invocationCount = xxxx) - I have seen this a few times. >>> >>> Maybe, but you didn't convince me. This picture is way too unstable >>> (overall testsuite) : >>> >>> https://infinispan.ci.cloudbees.com/job/Infinispan-master-JDK6-tcp/test/?... >>> >>> I'd rather think surefire / jenkins / testng / our listeners can't >>> count correctly. >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org twitter.com/maniksurtani Lead, Infinispan http://www.infinispan.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev