[infinispan-dev] Hidden failures in the testsuite

[Sanne Grinovero]

Hi all,
I just updated my local master fork and started the testsuite, as I
sometimes do.

It's great to see that the build was successful, and no tests
*appeared* to have failed.

But! lazily scrolling up in the console, I see lots of exceptions
which don't look like intentional (I'm aware that some tests
intentionally create error conditions). Also some tests are extremely
verbose, which might be the reason for nobody noticing these.

Some examples:
 - org.infinispan.it.compatibility.EmbeddedRestHotRodTest seems to log
TRACE to the console (and probably the whole module)
 - CDI tests such as org.infinispan.cdi.InfinispanExtensionRemote seem
to fail in great number because of some ClassNotFoundException(s)
and/or ResourceLoadingException(s)
 - OSGi integration tests seem to be all broken by some invalid
integration with Aries / Geronimo
 - OSGi integration tests dump a lot of unnecessary information to the
build console
 - the Infinispan Query tests log lots of WARN too, around missing
configuration properties and in some cases concerning exceptions; I'm
pretty sure that I had resolved those in the past, seems some
refactorings were done w/o considering the log outputs.

Please don't ignore the output; if it's too verbose to watch, that
needs to be resolved too.

I also monitor the "expected execution time" of some modules I'm
interested in, that's been useful in some cases to figure out that
there was some regression.

One big question: why is it that so many tests "appear to be good" but
are actually broken? I would like to understand that.

Thanks,
Sanne