Hey Sanne! Yep you are right ignoring output is BAD IDEA. I realized that it's difficult to look through all log manually so probably we should write some parser in python or bash to grep it and put it into bin/ folder with other scripts, So at least we can run this script after all tests were run and analyze it somehow. And about "appear to be good" nobody knows why. It could be testng/junit issue as we mix it a lot. So this needs further discussion and analysis. Vitalii ----- Вихідне повідомлення ----- Від: "Sanne Grinovero" <sanne(a)infinispan.org&gt; Кому: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; Надіслано: Понеділок, 10 Серпень 2015 р 20:46:06 Тема: [infinispan-dev] Hidden failures in the testsuite Hi all, I just updated my local master fork and started the testsuite, as I sometimes do. It's great to see that the build was successful, and no tests *appeared* to have failed. But! lazily scrolling up in the console, I see lots of exceptions which don't look like intentional (I'm aware that some tests intentionally create error conditions). Also some tests are extremely verbose, which might be the reason for nobody noticing these. Some examples: - org.infinispan.it.compatibility.EmbeddedRestHotRodTest seems to log TRACE to the console (and probably the whole module)

- CDI tests such as org.infinispan.cdi.InfinispanExtensionRemote seem to fail in great number because of some ClassNotFoundException(s) and/or ResourceLoadingException(s)

- OSGi integration tests seem to be all broken by some invalid integration with Aries / Geronimo - OSGi integration tests dump a lot of unnecessary information to the build console - the Infinispan Query tests log lots of WARN too, around missing configuration properties and in some cases concerning exceptions; I'm pretty sure that I had resolved those in the past, seems some refactorings were done w/o considering the log outputs. Please don't ignore the output; if it's too verbose to watch, that needs to be resolved too. I also monitor the "expected execution time" of some modules I'm interested in, that's been useful in some cases to figure out that there was some regression. One big question: why is it that so many tests "appear to be good" but are actually broken? I would like to understand that. Thanks, Sanne _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Hi all, sorry for the slow reply, I can't get to test Infinispan very often lately. On top of previously reported issues - which I still have - today I also noticed this one: [UnitTestTestNGListener] Test testPutTimeout(org.infinispan.client.hotrod.ClientSocketReadTimeoutTest) failed. Sep 09, 2015 11:28:16 AM io.netty.util.concurrent.SingleThreadEventExecutor$2 run WARNING: Unexpected exception from an event executor: java.lang.OutOfMemoryError: GC overhead limit exceeded Unless surefire overrides it, all my Maven jobs are assigned 2GB of heap. I know that's not huge but I prefer it to be conservative to serve as "canary". Is that known to be not enough anymore, or worth looking for memory issues? Thanks, Sanne On 3 September 2015 at 14:41, Galder Zamarreno <galder(a)redhat.com&gt; wrote: > Hi Sanne, > > I've looked at CDI and Compatibility issues, see below. > > Cheers, > -- > Galder Zamarreño > Infinispan, Red Hat > > ----- Original Message ----- >> Hey Sanne! Yep you are right ignoring output is BAD IDEA. I realized that >> it's difficult to look through all log manually so probably we should write >> some parser in python or bash to grep it and put it into bin/ folder with >> other scripts, So at least we can run this script after all tests were run >> and analyze it somehow. And about "appear to be good" nobody knows why. It >> could be testng/junit issue as we mix it a lot. So this needs further >> discussion and analysis. >> >> Vitalii >> >> ----- Вихідне повідомлення ----- >> Від: "Sanne Grinovero" <sanne(a)infinispan.org&gt; >> Кому: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; >> Надіслано: Понеділок, 10 Серпень 2015 р 20:46:06 >> Тема: [infinispan-dev] Hidden failures in the testsuite >> >> Hi all, >> I just updated my local master fork and started the testsuite, as I >> sometimes do. >> >> It's great to see that the build was successful, and no tests >> *appeared* to have failed. >> >> But! lazily scrolling up in the console, I see lots of exceptions >> which don't look like intentional (I'm aware that some tests >> intentionally create error conditions). Also some tests are extremely >> verbose, which might be the reason for nobody noticing these. >> >> Some examples: >> - org.infinispan.it.compatibility.EmbeddedRestHotRodTest seems to log >> TRACE to the console (and probably the whole module) > > ^ I've run the compatibility testsuite manually and didn't have such issue with master: > https://gist.github.com/galderz/b59f1ed4599229022f27 > > Are you still having issues with this? > >> - CDI tests such as org.infinispan.cdi.InfinispanExtensionRemote seem >> to fail in great number because of some ClassNotFoundException(s) >> and/or ResourceLoadingException(s) > > ^ Hmmmm, not seeing any of that either: > https://gist.github.com/galderz/1143078e6be8869cd602 > > Are you still having issues with this? > >> - OSGi integration tests seem to be all broken by some invalid >> integration with Aries / Geronimo >> - OSGi integration tests dump a lot of unnecessary information to the >> build console >> - the Infinispan Query tests log lots of WARN too, around missing >> configuration properties and in some cases concerning exceptions; I'm >> pretty sure that I had resolved those in the past, seems some >> refactorings were done w/o considering the log outputs. >> >> Please don't ignore the output; if it's too verbose to watch, that >> needs to be resolved too. >> >> I also monitor the "expected execution time" of some modules I'm >> interested in, that's been useful in some cases to figure out that >> there was some regression. >> >> One big question: why is it that so many tests "appear to be good" but >> are actually broken? I would like to understand that. >> >> Thanks, >> Sanne >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Dan, since Java 8, Compressed Oops are on by default, so that can be removed. Tristan On 09/09/2015 12:58, Dan Berindei wrote: > Sanne, the forked JVM that actually runs the tests uses only 1GB by default: > > <forkJvmArgs>-Xmx1024m -XX:MaxPermSize=256m</forkJvmArgs> > > The CI builds use the same value, although they do enable compressed > oops explicitly and use smaller thread stacks: > > env.MAVEN_FORK_OPTS = %maven_opts.memory.x64% %maven_opts.tuning% > maven_opts.memory.x64 = -XX:+UseCompressedOops -Xmx1024m -Xms256m > -XX:MaxPermSize=256m -Xss512k > > That being said, you're probably seeing ISPN-5727: > https://github.com/infinispan/infinispan/pull/3696 > > Cheers > Dan > > > On Wed, Sep 9, 2015 at 1:33 PM, Sanne Grinovero <sanne(a)infinispan.org&gt; wrote: >> Hi all, >> sorry for the slow reply, I can't get to test Infinispan very often lately. >> >> On top of previously reported issues - which I still have - today I >> also noticed this one: >> >> [UnitTestTestNGListener] Test >> testPutTimeout(org.infinispan.client.hotrod.ClientSocketReadTimeoutTest) >> failed. >> Sep 09, 2015 11:28:16 AM >> io.netty.util.concurrent.SingleThreadEventExecutor$2 run >> WARNING: Unexpected exception from an event executor: >> java.lang.OutOfMemoryError: GC overhead limit exceeded >> >> Unless surefire overrides it, all my Maven jobs are assigned 2GB of >> heap. I know that's not huge but I prefer it to be conservative to >> serve as "canary". >> >> Is that known to be not enough anymore, or worth looking for memory issues? >> >> Thanks, >> Sanne >> >> >> On 3 September 2015 at 14:41, Galder Zamarreno <galder(a)redhat.com&gt; wrote: >>> Hi Sanne, >>> >>> I've looked at CDI and Compatibility issues, see below. >>> >>> Cheers, >>> -- >>> Galder Zamarreño >>> Infinispan, Red Hat >>> >>> ----- Original Message ----- >>>> Hey Sanne! Yep you are right ignoring output is BAD IDEA. I realized that >>>> it's difficult to look through all log manually so probably we should write >>>> some parser in python or bash to grep it and put it into bin/ folder with >>>> other scripts, So at least we can run this script after all tests were run >>>> and analyze it somehow. And about "appear to be good" nobody knows why. It >>>> could be testng/junit issue as we mix it a lot. So this needs further >>>> discussion and analysis. >>>> >>>> Vitalii >>>> >>>> ----- Вихідне повідомлення ----- >>>> Від: "Sanne Grinovero" <sanne(a)infinispan.org&gt; >>>> Кому: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; >>>> Надіслано: Понеділок, 10 Серпень 2015 р 20:46:06 >>>> Тема: [infinispan-dev] Hidden failures in the testsuite >>>> >>>> Hi all, >>>> I just updated my local master fork and started the testsuite, as I >>>> sometimes do. >>>> >>>> It's great to see that the build was successful, and no tests >>>> *appeared* to have failed. >>>> >>>> But! lazily scrolling up in the console, I see lots of exceptions >>>> which don't look like intentional (I'm aware that some tests >>>> intentionally create error conditions). Also some tests are extremely >>>> verbose, which might be the reason for nobody noticing these. >>>> >>>> Some examples: >>>> - org.infinispan.it.compatibility.EmbeddedRestHotRodTest seems to log >>>> TRACE to the console (and probably the whole module) >>> >>> ^ I've run the compatibility testsuite manually and didn't have such issue with master: >>> https://gist.github.com/galderz/b59f1ed4599229022f27 >>> >>> Are you still having issues with this? >>> >>>> - CDI tests such as org.infinispan.cdi.InfinispanExtensionRemote seem >>>> to fail in great number because of some ClassNotFoundException(s) >>>> and/or ResourceLoadingException(s) >>> >>> ^ Hmmmm, not seeing any of that either: >>> https://gist.github.com/galderz/1143078e6be8869cd602 >>> >>> Are you still having issues with this? >>> >>>> - OSGi integration tests seem to be all broken by some invalid >>>> integration with Aries / Geronimo >>>> - OSGi integration tests dump a lot of unnecessary information to the >>>> build console >>>> - the Infinispan Query tests log lots of WARN too, around missing >>>> configuration properties and in some cases concerning exceptions; I'm >>>> pretty sure that I had resolved those in the past, seems some >>>> refactorings were done w/o considering the log outputs. >>>> >>>> Please don't ignore the output; if it's too verbose to watch, that >>>> needs to be resolved too. >>>> >>>> I also monitor the "expected execution time" of some modules I'm >>>> interested in, that's been useful in some cases to figure out that >>>> there was some regression. >>>> >>>> One big question: why is it that so many tests "appear to be good" but >>>> are actually broken? I would like to understand that. >>>> >>>> Thanks, >>>> Sanne >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > -- Tristan Tarrant Infinispan Lead JBoss, a division of Red Hat _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev