[infinispan-dev] Hidden failures in the testsuite

[Galder Zamarreno]

Hi Sanne,

I've looked at CDI and Compatibility issues, see below.

Cheers,
--
Galder Zamarreño
Infinispan, Red Hat

----- Original Message -----
> Hey Sanne! Yep you are right ignoring output is BAD IDEA. I realized that
> it's difficult to look through all log manually so probably we should write
> some parser in python or bash to grep it and put it into bin/ folder with
> other scripts, So at least we can run this script after all tests were run
> and analyze it somehow. And about "appear to be good" nobody knows why. It
> could be testng/junit issue as we mix it a lot. So this needs further
> discussion and analysis.
> 
> Vitalii
> 
> ----- Вихідне повідомлення -----
> Від: "Sanne Grinovero" <sanne at infinispan.org>
> Кому: "infinispan -Dev List" <infinispan-dev at lists.jboss.org>
> Надіслано: Понеділок, 10 Серпень 2015 р 20:46:06
> Тема: [infinispan-dev] Hidden failures in the testsuite
> 
> Hi all,
> I just updated my local master fork and started the testsuite, as I
> sometimes do.
> 
> It's great to see that the build was successful, and no tests
> *appeared* to have failed.
> 
> But! lazily scrolling up in the console, I see lots of exceptions
> which don't look like intentional (I'm aware that some tests
> intentionally create error conditions). Also some tests are extremely
> verbose, which might be the reason for nobody noticing these.
> 
> Some examples:
>  - org.infinispan.it.compatibility.EmbeddedRestHotRodTest seems to log
> TRACE to the console (and probably the whole module)

^ I've run the compatibility testsuite manually and didn't have such issue with master:
https://gist.github.com/galderz/b59f1ed4599229022f27

Are you still having issues with this?

>  - CDI tests such as org.infinispan.cdi.InfinispanExtensionRemote seem
> to fail in great number because of some ClassNotFoundException(s)
> and/or ResourceLoadingException(s)

^ Hmmmm, not seeing any of that either:
https://gist.github.com/galderz/1143078e6be8869cd602

Are you still having issues with this?

>  - OSGi integration tests seem to be all broken by some invalid
> integration with Aries / Geronimo
>  - OSGi integration tests dump a lot of unnecessary information to the
> build console
>  - the Infinispan Query tests log lots of WARN too, around missing
> configuration properties and in some cases concerning exceptions; I'm
> pretty sure that I had resolved those in the past, seems some
> refactorings were done w/o considering the log outputs.
> 
> Please don't ignore the output; if it's too verbose to watch, that
> needs to be resolved too.
> 
> I also monitor the "expected execution time" of some modules I'm
> interested in, that's been useful in some cases to figure out that
> there was some regression.
> 
> One big question: why is it that so many tests "appear to be good" but
> are actually broken? I would like to understand that.
> 
> Thanks,
> Sanne
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev