On 04 Feb 2014, at 10:38, Stuart Douglas <stuart.w.douglas(a)gmail.com&gt; wrote: > It is almost certainly something to do with this: > > <module-def name="org.infinispan.server.rest" src="${infinispan.server.modules.dir}"> > > <maven-resource-with-classifier group="org.infinispan" artifact="infinispan-server-rest" classifier="classes" /> > > </module-def> > > I guess sometimes the classes artefact is being attached as a reference to the classes directory, rather than a reference to a jar, which causes the issue. Here’s a gist with a subset of the build log [1]. When it works fine, it’s copying a jar, when it’s not, it’s copying an empty folder. However, this is not only happening for the org.infinispan.server.rest module, others show the same issue [2]. What seems to be a pattern is that it only happens with modules that are built by us, it’s not happening for modules coming with the base AS/WF instance. I’ve traced back and this might be due to build failures that are not producing the right jars [3]. @Stuart, this is really our problem. Sorry for the inconvenience! [1] https://gist.github.com/galderz/b9286f385aad1316df51 [2] https://gist.github.com/galderz/9e6a9bd9b18b805db323 [3] https://gist.github.com/galderz/6ab662a1027cd96cbd8c > > Stuart > > > On Tue, Feb 4, 2014 at 11:14 AM, Galder Zamarreño <galder(a)redhat.com&gt; wrote: > > On 04 Feb 2014, at 10:01, Stuart Douglas <stuart.w.douglas(a)gmail.com&gt; wrote: > >> >> >> >> On Tue, Feb 4, 2014 at 11:00 AM, Stuart Douglas <stuart.w.douglas(a)gmail.com&gt; wrote: >> Yes, there is nothing in the server code that modified the modules directory. >> >> Well, except for the new patching stuff, but that is not really relevant here. > > The testsuite AS/WF builds are built out of the distribution build, which shows the same problem. The distribution we build uses the scripts we got from AS [1]. > > Do you see anything in there that could be causing this? We are using maven-antrun-plugin version 1.3, and take into account the lib.xml in [2]. > > Finally, do you have any suggestions on changes we could make to these files to further debug the issue? > > Thanks a lot for your help! > > [1] https://github.com/infinispan/infinispan/blob/master/server/integration/b... > [2] https://github.com/infinispan/infinispan/blob/master/server/integration/b... > >> >> Stuart >> >> >> Stuart >> >> >> On Tue, Feb 4, 2014 at 10:56 AM, Galder Zamarreño <galder(a)redhat.com&gt; wrote: >> >> On 04 Feb 2014, at 09:37, Stuart Douglas <stuart.w.douglas(a)gmail.com&gt; wrote: >> >>> This looks like an issue with your environment. The modules directory is static. Wildfly does not contain any code that messes with it. I would say the culprit is probably something in either your build process or your test suite. >> >> Correction, this is happening with AS 7.2.0.Final (Wildfly 8 used somewhere else). I guess your answer still applies? >> >> Cheers, >> >>> >>> Stuart >>> >>> >>> On Tue, Feb 4, 2014 at 10:21 AM, Galder Zamarreño <galder(a)redhat.com&gt; wrote: >>> Hi all, >>> >>> We’re having issues with our Infinispan Server integration tests, which run within Wildfly 8.0.0.Beta1 (as I’m typing I’m wondering if we should just upgrade it to see if this goes away...?). >>> >>> Quite often some of the runs fail with error message [1]. >>> >>> Having looked at the build environment when a run fails, you see this: >>> >>> -- >>> $ ls modules/system/layers/base/org/infinispan/server/rest/main >>> drwxrwxr-x 2 g staff 68B Feb 3 18:41 classes (<— a directory??) >>> -rw-r--r-- 1 g staff 1B Feb 3 18:41 classes.index >>> -rw-r--r-- 1 g staff 2.1K Feb 3 18:41 module.xml >>> >>> $ ls modules/system/layers/base/org/infinispan/server/rest/main/classes >>> drwxrwxr-x 2 g staff 68B Feb 3 18:41 . >>> drwxrwxr-x 5 g staff 170B Feb 3 18:41 .. >>> >>> $ more modules/system/layers/base/org/infinispan/server/rest/main/module.xml >>> <module xmlns="urn:jboss:module:1.1" name="org.infinispan.server.rest”> >>> ... >>> <resource-root path="classes"/> >>> ... >>> >>> This is completely different to what happens with a successful run: >>> >>> -- >>> $ ls modules/system/layers/base/org/infinispan/server/rest/main >>> -rw-r--r-- 1 g staff 103K Feb 3 19:40 infinispan-classes.jar (<— a jar file!) >>> -rw-r--r-- 1 g staff 278B Feb 3 19:40 infinispan-classes.jar.index >>> -rw-r--r-- 1 g staff 2.1K Feb 3 19:40 module.xml >>> >>> $ jar tf modules/system/layers/base/org/infinispan/server/rest/main/infinispan-classes.jar | grep ExtendedHeaders >>> org/infinispan/rest/configuration/ExtendedHeaders.class >>> >>> $ more modules/system/layers/base/org/infinispan/server/rest/main/module.xml >>> <module xmlns="urn:jboss:module:1.1" name="org.infinispan.server.rest”> >>> ... >>> <resource-root path="infinispan-classes.jar”/> >>> — >>> >>> Anyone can explain what is going on here? Does it ring a bell to anyone? Is this a known Wildfly issue by any chance? >>> >>> [1] https://gist.github.com/galderz/bd74cebfc840ef3ae284 >>> >>> Cheers, >>> -- >>> Galder Zamarreño >>> galder(a)redhat.com >>> twitter.com/galderz >>> >>> Project Lead, Escalante >>> http://escalante.io >>> >>> Engineer, Infinispan >>> http://infinispan.org >>> >>> >>> _______________________________________________ >>> jboss-as7-dev mailing list >>> jboss-as7-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>> >> >> >> -- >> Galder Zamarreño >> galder(a)redhat.com >> twitter.com/galderz >> >> Project Lead, Escalante >> http://escalante.io >> >> Engineer, Infinispan >> http://infinispan.org >> >> >> > > > -- > Galder Zamarreño > galder(a)redhat.com > twitter.com/galderz > > Project Lead, Escalante > http://escalante.io > > Engineer, Infinispan > http://infinispan.org > > -- Galder Zamarreño galder(a)redhat.com twitter.com/galderz Project Lead, Escalante http://escalante.io Engineer, Infinispan http://infinispan.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On Tue, Feb 4, 2014 at 2:36 PM, Galder Zamarreño <galder(a)redhat.com&gt; wrote: Narrowing down the list now, since this is a problem of how our CI is doing builds. These logs are retrieved from [1]. Dunno how our CI is configured but this is odd. Seems like the build is halt due to test failures, but it continues somehow? I mean, the jars are not being produced properly, but the build is not halting. We run the build with -fn (fail-never), so the build should never be halted because of a test failure. The configuration is here: http://ci.infinispan.org/admin/editRunType.html?id=buildType:bt8&runn...

It’s about time we did the following: 1) Any test failures should halt the build there and then. IOW, do not continue the build at all. Will having 100 tests in one run and 2000 tests in another really help?

2) Any tests that fail randomly should be disabled. Let's go ahead and disable all the server tests then? ;)

Cheers, [1] https://dl.dropboxusercontent.com/u/6148072/does-not-work.log On 04 Feb 2014, at 13:30, Galder Zamarreño <galder(a)redhat.com&gt; wrote: > > On 04 Feb 2014, at 10:38, Stuart Douglas <stuart.w.douglas(a)gmail.com&gt; wrote: > >> It is almost certainly something to do with this: >> >> <module-def name="org.infinispan.server.rest" src="${infinispan.server.modules.dir}"> >> >> <maven-resource-with-classifier group="org.infinispan" artifact="infinispan-server-rest" classifier="classes" /> >> >> </module-def> >> >> I guess sometimes the classes artefact is being attached as a reference to the classes directory, rather than a reference to a jar, which causes the issue. > > Here’s a gist with a subset of the build log [1]. When it works fine, it’s copying a jar, when it’s not, it’s copying an empty folder. > > However, this is not only happening for the org.infinispan.server.rest module, others show the same issue [2]. What seems to be a pattern is that it only happens with modules that are built by us, it’s not happening for modules coming with the base AS/WF instance. > > I’ve traced back and this might be due to build failures that are not producing the right jars [3]. > > @Stuart, this is really our problem. Sorry for the inconvenience! > > [1] https://gist.github.com/galderz/b9286f385aad1316df51 > [2] https://gist.github.com/galderz/9e6a9bd9b18b805db323 > [3] https://gist.github.com/galderz/6ab662a1027cd96cbd8c > >> >> Stuart >> >> >> On Tue, Feb 4, 2014 at 11:14 AM, Galder Zamarreño <galder(a)redhat.com&gt; wrote: >> >> On 04 Feb 2014, at 10:01, Stuart Douglas <stuart.w.douglas(a)gmail.com&gt; wrote: >> >>> >>> >>> >>> On Tue, Feb 4, 2014 at 11:00 AM, Stuart Douglas <stuart.w.douglas(a)gmail.com&gt; wrote: >>> Yes, there is nothing in the server code that modified the modules directory. >>> >>> Well, except for the new patching stuff, but that is not really relevant here. >> >> The testsuite AS/WF builds are built out of the distribution build, which shows the same problem. The distribution we build uses the scripts we got from AS [1]. >> >> Do you see anything in there that could be causing this? We are using maven-antrun-plugin version 1.3, and take into account the lib.xml in [2]. >> >> Finally, do you have any suggestions on changes we could make to these files to further debug the issue? >> >> Thanks a lot for your help! >> >> [1] https://github.com/infinispan/infinispan/blob/master/server/integration/b... >> [2] https://github.com/infinispan/infinispan/blob/master/server/integration/b... >> >>> >>> Stuart >>> >>> >>> Stuart >>> >>> >>> On Tue, Feb 4, 2014 at 10:56 AM, Galder Zamarreño <galder(a)redhat.com&gt; wrote: >>> >>> On 04 Feb 2014, at 09:37, Stuart Douglas <stuart.w.douglas(a)gmail.com&gt; wrote: >>> >>>> This looks like an issue with your environment. The modules directory is static. Wildfly does not contain any code that messes with it. I would say the culprit is probably something in either your build process or your test suite. >>> >>> Correction, this is happening with AS 7.2.0.Final (Wildfly 8 used somewhere else). I guess your answer still applies? >>> >>> Cheers, >>> >>>> >>>> Stuart >>>> >>>> >>>> On Tue, Feb 4, 2014 at 10:21 AM, Galder Zamarreño <galder(a)redhat.com&gt; wrote: >>>> Hi all, >>>> >>>> We’re having issues with our Infinispan Server integration tests, which run within Wildfly 8.0.0.Beta1 (as I’m typing I’m wondering if we should just upgrade it to see if this goes away...?). >>>> >>>> Quite often some of the runs fail with error message [1]. >>>> >>>> Having looked at the build environment when a run fails, you see this: >>>> >>>> -- >>>> $ ls modules/system/layers/base/org/infinispan/server/rest/main >>>> drwxrwxr-x 2 g staff 68B Feb 3 18:41 classes (<— a directory??) >>>> -rw-r--r-- 1 g staff 1B Feb 3 18:41 classes.index >>>> -rw-r--r-- 1 g staff 2.1K Feb 3 18:41 module.xml >>>> >>>> $ ls modules/system/layers/base/org/infinispan/server/rest/main/classes >>>> drwxrwxr-x 2 g staff 68B Feb 3 18:41 . >>>> drwxrwxr-x 5 g staff 170B Feb 3 18:41 .. >>>> >>>> $ more modules/system/layers/base/org/infinispan/server/rest/main/module.xml >>>> <module xmlns="urn:jboss:module:1.1" name="org.infinispan.server.rest”> >>>> ... >>>> <resource-root path="classes"/> >>>> ... >>>> >>>> This is completely different to what happens with a successful run: >>>> >>>> -- >>>> $ ls modules/system/layers/base/org/infinispan/server/rest/main >>>> -rw-r--r-- 1 g staff 103K Feb 3 19:40 infinispan-classes.jar (<— a jar file!) >>>> -rw-r--r-- 1 g staff 278B Feb 3 19:40 infinispan-classes.jar.index >>>> -rw-r--r-- 1 g staff 2.1K Feb 3 19:40 module.xml >>>> >>>> $ jar tf modules/system/layers/base/org/infinispan/server/rest/main/infinispan-classes.jar | grep ExtendedHeaders >>>> org/infinispan/rest/configuration/ExtendedHeaders.class >>>> >>>> $ more modules/system/layers/base/org/infinispan/server/rest/main/module.xml >>>> <module xmlns="urn:jboss:module:1.1" name="org.infinispan.server.rest”> >>>> ... >>>> <resource-root path="infinispan-classes.jar”/> >>>> — >>>> >>>> Anyone can explain what is going on here? Does it ring a bell to anyone? Is this a known Wildfly issue by any chance? >>>> >>>> [1] https://gist.github.com/galderz/bd74cebfc840ef3ae284 >>>> >>>> Cheers, >>>> -- >>>> Galder Zamarreño >>>> galder(a)redhat.com >>>> twitter.com/galderz >>>> >>>> Project Lead, Escalante >>>> http://escalante.io >>>> >>>> Engineer, Infinispan >>>> http://infinispan.org >>>> >>>> >>>> _______________________________________________ >>>> jboss-as7-dev mailing list >>>> jboss-as7-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>> >>> >>> >>> -- >>> Galder Zamarreño >>> galder(a)redhat.com >>> twitter.com/galderz >>> >>> Project Lead, Escalante >>> http://escalante.io >>> >>> Engineer, Infinispan >>> http://infinispan.org >>> >>> >>> >> >> >> -- >> Galder Zamarreño >> galder(a)redhat.com >> twitter.com/galderz >> >> Project Lead, Escalante >> http://escalante.io >> >> Engineer, Infinispan >> http://infinispan.org >> >> > > > -- > Galder Zamarreño > galder(a)redhat.com > twitter.com/galderz > > Project Lead, Escalante > http://escalante.io > > Engineer, Infinispan > http://infinispan.org > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño galder(a)redhat.com twitter.com/galderz Project Lead, Escalante http://escalante.io Engineer, Infinispan http://infinispan.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On Tue, Feb 4, 2014 at 3:03 PM, Galder Zamarreño <galder(a)redhat.com&gt; wrote: On 04 Feb 2014, at 13:50, Dan Berindei <dan.berindei(a)gmail.com&gt; wrote: > > On Tue, Feb 4, 2014 at 2:36 PM, Galder Zamarreño <galder(a)redhat.com&gt; wrote: > Narrowing down the list now, since this is a problem of how our CI is doing builds. > > These logs are retrieved from [1]. > > Dunno how our CI is configured but this is odd. Seems like the build is halt due to test failures, but it continues somehow? I mean, the jars are not being produced properly, but the build is not halting. > > We run the build with -fn (fail-never), so the build should never be halted because of a test failure. The configuration is here: http://ci.infinispan.org/admin/editRunType.html?id=buildType:bt8&runn... ^ That’s not working as expected, see the build log, my snippets…etc. Sorry, I didn't understand what's happening in those snippets.

All I saw was an Ant script that doesn't do what it's supposed to do :)

I did see some differences in the configuration between the JDK6 and the JDK7 builds: * JDK7 uses -fn and JDK6 uses -Dmaven.test.failure.ignore * JDK7 uses -nsu (no snapshot updates), JDK6 doesn't I've changed both builds to use -Dmaven.test.failure.ignore and -nsu, let's see how it goes.

> > > > It’s about time we did the following: > 1) Any test failures should halt the build there and then. IOW, do not continue the build at all. > > Will having 100 tests in one run and 2000 tests in another really help? As you disable randomly failing tests, and do not integrate commits making the testsuite fail, these number should even out. Not integrating commits that fail every time is easy, not integrating commits that fail randomly (maybe only in some environments) is trickier.

> > > 2) Any tests that fail randomly should be disabled. > > Let's go ahead and disable all the server tests then? ;) Those server tests that are randomly failing should be disabled and looked at. Those tests that are failing as a result of container not starting are side effects of things not working properly, and these should not be disabled. Why treat the tests that are failing because of a build problem differently? What about the tests that fail only on IBM JDK6?

> > > > Cheers, > > [1] https://dl.dropboxusercontent.com/u/6148072/does-not-work.log > > On 04 Feb 2014, at 13:30, Galder Zamarreño <galder(a)redhat.com&gt; wrote: > > > > > On 04 Feb 2014, at 10:38, Stuart Douglas <stuart.w.douglas(a)gmail.com&gt; wrote: > > > >> It is almost certainly something to do with this: > >> > >> <module-def name="org.infinispan.server.rest" src="${infinispan.server.modules.dir}"> > >> > >> <maven-resource-with-classifier group="org.infinispan" artifact="infinispan-server-rest" classifier="classes" /> > >> > >> </module-def> > >> > >> I guess sometimes the classes artefact is being attached as a reference to the classes directory, rather than a reference to a jar, which causes the issue. > > > > Here’s a gist with a subset of the build log [1]. When it works fine, it’s copying a jar, when it’s not, it’s copying an empty folder. > > > > However, this is not only happening for the org.infinispan.server.rest module, others show the same issue [2]. What seems to be a pattern is that it only happens with modules that are built by us, it’s not happening for modules coming with the base AS/WF instance. > > > > I’ve traced back and this might be due to build failures that are not producing the right jars [3]. > > > > @Stuart, this is really our problem. Sorry for the inconvenience! > > > > [1] https://gist.github.com/galderz/b9286f385aad1316df51 > > [2] https://gist.github.com/galderz/9e6a9bd9b18b805db323 > > [3] https://gist.github.com/galderz/6ab662a1027cd96cbd8c > > > >> > >> Stuart > >> > >> > >> On Tue, Feb 4, 2014 at 11:14 AM, Galder Zamarreño <galder(a)redhat.com&gt; wrote: > >> > >> On 04 Feb 2014, at 10:01, Stuart Douglas <stuart.w.douglas(a)gmail.com&gt; wrote: > >> > >>> > >>> > >>> > >>> On Tue, Feb 4, 2014 at 11:00 AM, Stuart Douglas <stuart.w.douglas(a)gmail.com&gt; wrote: > >>> Yes, there is nothing in the server code that modified the modules directory. > >>> > >>> Well, except for the new patching stuff, but that is not really relevant here. > >> > >> The testsuite AS/WF builds are built out of the distribution build, which shows the same problem. The distribution we build uses the scripts we got from AS [1]. > >> > >> Do you see anything in there that could be causing this? We are using maven-antrun-plugin version 1.3, and take into account the lib.xml in [2]. > >> > >> Finally, do you have any suggestions on changes we could make to these files to further debug the issue? > >> > >> Thanks a lot for your help! > >> > >> [1] https://github.com/infinispan/infinispan/blob/master/server/integration/b... > >> [2] https://github.com/infinispan/infinispan/blob/master/server/integration/b... > >> > >>> > >>> Stuart > >>> > >>> > >>> Stuart > >>> > >>> > >>> On Tue, Feb 4, 2014 at 10:56 AM, Galder Zamarreño <galder(a)redhat.com&gt; wrote: > >>> > >>> On 04 Feb 2014, at 09:37, Stuart Douglas <stuart.w.douglas(a)gmail.com&gt; wrote: > >>> > >>>> This looks like an issue with your environment. The modules directory is static. Wildfly does not contain any code that messes with it. I would say the culprit is probably something in either your build process or your test suite. > >>> > >>> Correction, this is happening with AS 7.2.0.Final (Wildfly 8 used somewhere else). I guess your answer still applies? > >>> > >>> Cheers, > >>> > >>>> > >>>> Stuart > >>>> > >>>> > >>>> On Tue, Feb 4, 2014 at 10:21 AM, Galder Zamarreño <galder(a)redhat.com&gt; wrote: > >>>> Hi all, > >>>> > >>>> We’re having issues with our Infinispan Server integration tests, which run within Wildfly 8.0.0.Beta1 (as I’m typing I’m wondering if we should just upgrade it to see if this goes away...?). > >>>> > >>>> Quite often some of the runs fail with error message [1]. > >>>> > >>>> Having looked at the build environment when a run fails, you see this: > >>>> > >>>> -- > >>>> $ ls modules/system/layers/base/org/infinispan/server/rest/main > >>>> drwxrwxr-x 2 g staff 68B Feb 3 18:41 classes (<— a directory??) > >>>> -rw-r--r-- 1 g staff 1B Feb 3 18:41 classes.index > >>>> -rw-r--r-- 1 g staff 2.1K Feb 3 18:41 module.xml > >>>> > >>>> $ ls modules/system/layers/base/org/infinispan/server/rest/main/classes > >>>> drwxrwxr-x 2 g staff 68B Feb 3 18:41 . > >>>> drwxrwxr-x 5 g staff 170B Feb 3 18:41 .. > >>>> > >>>> $ more modules/system/layers/base/org/infinispan/server/rest/main/module.xml > >>>> <module xmlns="urn:jboss:module:1.1" name="org.infinispan.server.rest”> > >>>> ... > >>>> <resource-root path="classes"/> > >>>> ... > >>>> > >>>> This is completely different to what happens with a successful run: > >>>> > >>>> -- > >>>> $ ls modules/system/layers/base/org/infinispan/server/rest/main > >>>> -rw-r--r-- 1 g staff 103K Feb 3 19:40 infinispan-classes.jar (<— a jar file!) > >>>> -rw-r--r-- 1 g staff 278B Feb 3 19:40 infinispan-classes.jar.index > >>>> -rw-r--r-- 1 g staff 2.1K Feb 3 19:40 module.xml > >>>> > >>>> $ jar tf modules/system/layers/base/org/infinispan/server/rest/main/infinispan-classes.jar | grep ExtendedHeaders > >>>> org/infinispan/rest/configuration/ExtendedHeaders.class > >>>> > >>>> $ more modules/system/layers/base/org/infinispan/server/rest/main/module.xml > >>>> <module xmlns="urn:jboss:module:1.1" name="org.infinispan.server.rest”> > >>>> ... > >>>> <resource-root path="infinispan-classes.jar”/> > >>>> — > >>>> > >>>> Anyone can explain what is going on here? Does it ring a bell to anyone? Is this a known Wildfly issue by any chance? > >>>> > >>>> [1] https://gist.github.com/galderz/bd74cebfc840ef3ae284 > >>>> > >>>> Cheers, > >>>> -- > >>>> Galder Zamarreño > >>>> galder(a)redhat.com > >>>> twitter.com/galderz > >>>> > >>>> Project Lead, Escalante > >>>> http://escalante.io > >>>> > >>>> Engineer, Infinispan > >>>> http://infinispan.org > >>>> > >>>> > >>>> _______________________________________________ > >>>> jboss-as7-dev mailing list > >>>> jboss-as7-dev(a)lists.jboss.org > >>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > >>>> > >>> > >>> > >>> -- > >>> Galder Zamarreño > >>> galder(a)redhat.com > >>> twitter.com/galderz > >>> > >>> Project Lead, Escalante > >>> http://escalante.io > >>> > >>> Engineer, Infinispan > >>> http://infinispan.org > >>> > >>> > >>> > >> > >> > >> -- > >> Galder Zamarreño > >> galder(a)redhat.com > >> twitter.com/galderz > >> > >> Project Lead, Escalante > >> http://escalante.io > >> > >> Engineer, Infinispan > >> http://infinispan.org > >> > >> > > > > > > -- > > Galder Zamarreño > > galder(a)redhat.com > > twitter.com/galderz > > > > Project Lead, Escalante > > http://escalante.io > > > > Engineer, Infinispan > > http://infinispan.org > > > > > > _______________________________________________ > > infinispan-dev mailing list > > infinispan-dev(a)lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Galder Zamarreño > galder(a)redhat.com > twitter.com/galderz > > Project Lead, Escalante > http://escalante.io > > Engineer, Infinispan > http://infinispan.org > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño galder(a)redhat.com twitter.com/galderz Project Lead, Escalante http://escalante.io Engineer, Infinispan http://infinispan.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

All, Sanne, Pedro, Dan and I had a very productive discussion on IRC on this topic [1]. We've decided that instead of disabling tests, we need them to run in order to get recent stacktraces, logs, etc. So, we've decided to create a new test group called "unstable". This test group would only be run in CI once a day and it'd be run in a different build. This build would also enable TRACE logging for standalone and server tests. For server, I need to create a task to do this selectively. The rest of builds, masters and PRS would not run the "unstable" group, and would not have TRACE enabled. The responsibility of unstable tests are the component owners. They need to handle them and decide what to do with them. Cheers, [1] https://gist.github.com/galderz/3563d1b23b5d50f80d82 On 04 Feb 2014, at 15:10, Galder Zamarreño <galder(a)redhat.com&gt; wrote: > > On 04 Feb 2014, at 14:47, Dan Berindei <dan.berindei(a)gmail.com&gt; wrote: > >> >> >> >> On Tue, Feb 4, 2014 at 3:03 PM, Galder Zamarreño <galder(a)redhat.com&gt; wrote: >> >> On 04 Feb 2014, at 13:50, Dan Berindei <dan.berindei(a)gmail.com&gt; wrote: >> >>> >>> On Tue, Feb 4, 2014 at 2:36 PM, Galder Zamarreño <galder(a)redhat.com&gt; wrote: >>> Narrowing down the list now, since this is a problem of how our CI is doing builds. >>> >>> These logs are retrieved from [1]. >>> >>> Dunno how our CI is configured but this is odd. Seems like the build is halt due to test failures, but it continues somehow? I mean, the jars are not being produced properly, but the build is not halting. >>> >>> We run the build with -fn (fail-never), so the build should never be halted because of a test failure. The configuration is here: http://ci.infinispan.org/admin/editRunType.html?id=buildType:bt8&runn... >> >> ^ That's not working as expected, see the build log, my snippets...etc. >> >> Sorry, I didn't understand what's happening in those snippets. > > The log shows it quite clearly that after those tests fail, nothing else runs in that module, including producing the jar. It halts. That's > >> All I saw was an Ant script that doesn't do what it's supposed to do :) > > The ant script not doing it's job is because there modules are not completing the build. There's a direct correlation between the three modules that fail with tests and the 3 modules that are copying an empty folder instead of the jar. > >> >> I did see some differences in the configuration between the JDK6 and the JDK7 builds: >> * JDK7 uses -fn and JDK6 uses -Dmaven.test.failure.ignore >> * JDK7 uses -nsu (no snapshot updates), JDK6 doesn't >> >> I've changed both builds to use -Dmaven.test.failure.ignore and -nsu, let's see how it goes. > > You are solving the wrong problem. > >> >> >> >>> >>> >>> >>> It's about time we did the following: >>> 1) Any test failures should halt the build there and then. IOW, do not continue the build at all. >>> >>> Will having 100 tests in one run and 2000 tests in another really help? >> >> As you disable randomly failing tests, and do not integrate commits making the testsuite fail, these number should even out. >> >> Not integrating commits that fail every time is easy, not integrating commits that fail randomly (maybe only in some environments) is trickier. > > I know it's tricky, but the only thing we can do is disable those really. I don't see how keeping them enabled is helping at all. > >> >> >>> >>> >>> 2) Any tests that fail randomly should be disabled. >>> >>> Let's go ahead and disable all the server tests then? ;) >> >> Those server tests that are randomly failing should be disabled and looked at. Those tests that are failing as a result of container not starting are side effects of things not working properly, and these should not be disabled. >> >> Why treat the tests that are failing because of a build problem differently? What about the tests that fail only on IBM JDK6? > > Disable and indicate that the test fails on IBM JDK6. Once the issue is fixed, reenable it. > >> >> >>> >>> >>> >>> Cheers, >>> >>> [1] https://dl.dropboxusercontent.com/u/6148072/does-not-work.log >>> >>> On 04 Feb 2014, at 13:30, Galder Zamarreño <galder(a)redhat.com&gt; wrote: >>> >>>> >>>> On 04 Feb 2014, at 10:38, Stuart Douglas <stuart.w.douglas(a)gmail.com&gt; wrote: >>>> >>>>> It is almost certainly something to do with this: >>>>> >>>>> <module-def name="org.infinispan.server.rest" src="${infinispan.server.modules.dir}"> >>>>> >>>>> <maven-resource-with-classifier group="org.infinispan" artifact="infinispan-server-rest" classifier="classes" /> >>>>> >>>>> </module-def> >>>>> >>>>> I guess sometimes the classes artefact is being attached as a reference to the classes directory, rather than a reference to a jar, which causes the issue. >>>> >>>> Here's a gist with a subset of the build log [1]. When it works fine, it's copying a jar, when it's not, it's copying an empty folder. >>>> >>>> However, this is not only happening for the org.infinispan.server.rest module, others show the same issue [2]. What seems to be a pattern is that it only happens with modules that are built by us, it's not happening for modules coming with the base AS/WF instance. >>>> >>>> I've traced back and this might be due to build failures that are not producing the right jars [3]. >>>> >>>> @Stuart, this is really our problem. Sorry for the inconvenience! >>>> >>>> [1] https://gist.github.com/galderz/b9286f385aad1316df51 >>>> [2] https://gist.github.com/galderz/9e6a9bd9b18b805db323 >>>> [3] https://gist.github.com/galderz/6ab662a1027cd96cbd8c >>>> >>>>> >>>>> Stuart >>>>> >>>>> >>>>> On Tue, Feb 4, 2014 at 11:14 AM, Galder Zamarreño <galder(a)redhat.com&gt; wrote: >>>>> >>>>> On 04 Feb 2014, at 10:01, Stuart Douglas <stuart.w.douglas(a)gmail.com&gt; wrote: >>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Feb 4, 2014 at 11:00 AM, Stuart Douglas < stuart.w.douglas(a)gmail.com&gt; wrote: >>>>>> Yes, there is nothing in the server code that modified the modules directory. >>>>>> >>>>>> Well, except for the new patching stuff, but that is not really relevant here. >>>>> >>>>> The testsuite AS/WF builds are built out of the distribution build, which shows the same problem. The distribution we build uses the scripts we got from AS [1]. >>>>> >>>>> Do you see anything in there that could be causing this? We are using maven-antrun-plugin version 1.3, and take into account the lib.xml in [2]. >>>>> >>>>> Finally, do you have any suggestions on changes we could make to these files to further debug the issue? >>>>> >>>>> Thanks a lot for your help! >>>>> >>>>> [1] https://github.com/infinispan/infinispan/blob/master/server/integration/b... >>>>> [2] https://github.com/infinispan/infinispan/blob/master/server/integration/b... >>>>> >>>>>> >>>>>> Stuart >>>>>> >>>>>> >>>>>> Stuart >>>>>> >>>>>> >>>>>> On Tue, Feb 4, 2014 at 10:56 AM, Galder Zamarreño < galder(a)redhat.com&gt; wrote: >>>>>> >>>>>> On 04 Feb 2014, at 09:37, Stuart Douglas < stuart.w.douglas(a)gmail.com&gt; wrote: >>>>>> >>>>>>> This looks like an issue with your environment. The modules directory is static. Wildfly does not contain any code that messes with it. I would say the culprit is probably something in either your build process or your test suite. >>>>>> >>>>>> Correction, this is happening with AS 7.2.0.Final (Wildfly 8 used somewhere else). I guess your answer still applies? >>>>>> >>>>>> Cheers, >>>>>> >>>>>>> >>>>>>> Stuart >>>>>>> >>>>>>> >>>>>>> On Tue, Feb 4, 2014 at 10:21 AM, Galder Zamarreño < galder(a)redhat.com&gt; wrote: >>>>>>> Hi all, >>>>>>> >>>>>>> We're having issues with our Infinispan Server integration tests, which run within Wildfly 8.0.0.Beta1 (as I'm typing I'm wondering if we should just upgrade it to see if this goes away...?). >>>>>>> >>>>>>> Quite often some of the runs fail with error message [1]. >>>>>>> >>>>>>> Having looked at the build environment when a run fails, you see this: >>>>>>> >>>>>>> -- >>>>>>> $ ls modules/system/layers/base/org/infinispan/server/rest/main >>>>>>> drwxrwxr-x 2 g staff 68B Feb 3 18:41 classes (<-- a directory??) >>>>>>> -rw-r--r-- 1 g staff 1B Feb 3 18:41 classes.index >>>>>>> -rw-r--r-- 1 g staff 2.1K Feb 3 18:41 module.xml >>>>>>> >>>>>>> $ ls modules/system/layers/base/org/infinispan/server/rest/main/classes >>>>>>> drwxrwxr-x 2 g staff 68B Feb 3 18:41 . >>>>>>> drwxrwxr-x 5 g staff 170B Feb 3 18:41 .. >>>>>>> >>>>>>> $ more modules/system/layers/base/org/infinispan/server/rest/main/module.xml >>>>>>> <module xmlns="urn:jboss:module:1.1" name="org.infinispan.server.rest"> >>>>>>> ... >>>>>>> <resource-root path="classes"/> >>>>>>> ... >>>>>>> >>>>>>> This is completely different to what happens with a successful run: >>>>>>> >>>>>>> -- >>>>>>> $ ls modules/system/layers/base/org/infinispan/server/rest/main >>>>>>> -rw-r--r-- 1 g staff 103K Feb 3 19:40 infinispan-classes.jar (<-- a jar file!) >>>>>>> -rw-r--r-- 1 g staff 278B Feb 3 19:40 infinispan-classes.jar.index >>>>>>> -rw-r--r-- 1 g staff 2.1K Feb 3 19:40 module.xml >>>>>>> >>>>>>> $ jar tf modules/system/layers/base/org/infinispan/server/rest/main/infinispan-classes.jar | grep ExtendedHeaders >>>>>>> org/infinispan/rest/configuration/ExtendedHeaders.class >>>>>>> >>>>>>> $ more modules/system/layers/base/org/infinispan/server/rest/main/module.xml >>>>>>> <module xmlns="urn:jboss:module:1.1" name="org.infinispan.server.rest"> >>>>>>> ... >>>>>>> <resource-root path="infinispan-classes.jar"/> >>>>>>> -- >>>>>>> >>>>>>> Anyone can explain what is going on here? Does it ring a bell to anyone? Is this a known Wildfly issue by any chance? >>>>>>> >>>>>>> [1] https://gist.github.com/galderz/bd74cebfc840ef3ae284 >>>>>>> >>>>>>> Cheers, >>>>>>> -- >>>>>>> Galder Zamarreño >>>>>>> galder(a)redhat.com >>>>>>> twitter.com/galderz >>>>>>> >>>>>>> Project Lead, Escalante >>>>>>> http://escalante.io >>>>>>> >>>>>>> Engineer, Infinispan >>>>>>> http://infinispan.org >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> jboss-as7-dev mailing list >>>>>>> jboss-as7-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Galder Zamarreño >>>>>> galder(a)redhat.com >>>>>> twitter.com/galderz >>>>>> >>>>>> Project Lead, Escalante >>>>>> http://escalante.io >>>>>> >>>>>> Engineer, Infinispan >>>>>> http://infinispan.org >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Galder Zamarreño >>>>> galder(a)redhat.com >>>>> twitter.com/galderz >>>>> >>>>> Project Lead, Escalante >>>>> http://escalante.io >>>>> >>>>> Engineer, Infinispan >>>>> http://infinispan.org >>>>> >>>>> >>>> >>>> >>>> -- >>>> Galder Zamarreño >>>> galder(a)redhat.com >>>> twitter.com/galderz >>>> >>>> Project Lead, Escalante >>>> http://escalante.io >>>> >>>> Engineer, Infinispan >>>> http://infinispan.org >>>> >>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> >>> -- >>> Galder Zamarreño >>> galder(a)redhat.com >>> twitter.com/galderz >>> >>> Project Lead, Escalante >>> http://escalante.io >>> >>> Engineer, Infinispan >>> http://infinispan.org >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> -- >> Galder Zamarreño >> galder(a)redhat.com >> twitter.com/galderz >> >> Project Lead, Escalante >> http://escalante.io >> >> Engineer, Infinispan >> http://infinispan.org >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Galder Zamarreño > galder(a)redhat.com > twitter.com/galderz > > Project Lead, Escalante > http://escalante.io > > Engineer, Infinispan > http://infinispan.org > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño galder(a)redhat.com twitter.com/galderz Project Lead, Escalante http://escalante.io Engineer, Infinispan http://infinispan.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Narrowing down the list now, since this is a problem of how our CI is doing builds. These logs are retrieved from [1]. Dunno how our CI is configured but this is odd. Seems like the build is halt due to test failures, but it continues somehow? I mean, the jars are not being produced properly, but the build is not halting. It’s about time we did the following: 1) Any test failures should halt the build there and then. IOW, do not continue the build at all. 2) Any tests that fail randomly should be disabled.

Cheers, [1] https://dl.dropboxusercontent.com/u/6148072/does-not-work.log On 04 Feb 2014, at 13:30, Galder Zamarreño <galder(a)redhat.com&gt; wrote: > > On 04 Feb 2014, at 10:38, Stuart Douglas <stuart.w.douglas(a)gmail.com&gt; wrote: > >> It is almost certainly something to do with this: >> >> <module-def name="org.infinispan.server.rest" src="${infinispan.server.modules.dir}"> >> >> <maven-resource-with-classifier group="org.infinispan" artifact="infinispan-server-rest" classifier="classes" /> >> >> </module-def> >> >> I guess sometimes the classes artefact is being attached as a reference to the classes directory, rather than a reference to a jar, which causes the issue. > > Here’s a gist with a subset of the build log [1]. When it works fine, it’s copying a jar, when it’s not, it’s copying an empty folder. > > However, this is not only happening for the org.infinispan.server.rest module, others show the same issue [2]. What seems to be a pattern is that it only happens with modules that are built by us, it’s not happening for modules coming with the base AS/WF instance. > > I’ve traced back and this might be due to build failures that are not producing the right jars [3]. > > @Stuart, this is really our problem. Sorry for the inconvenience! > > [1] https://gist.github.com/galderz/b9286f385aad1316df51 > [2] https://gist.github.com/galderz/9e6a9bd9b18b805db323 > [3] https://gist.github.com/galderz/6ab662a1027cd96cbd8c > >> >> Stuart >> >> >> On Tue, Feb 4, 2014 at 11:14 AM, Galder Zamarreño <galder(a)redhat.com&gt; wrote: >> >> On 04 Feb 2014, at 10:01, Stuart Douglas <stuart.w.douglas(a)gmail.com&gt; wrote: >> >>> >>> >>> >>> On Tue, Feb 4, 2014 at 11:00 AM, Stuart Douglas <stuart.w.douglas(a)gmail.com&gt; wrote: >>> Yes, there is nothing in the server code that modified the modules directory. >>> >>> Well, except for the new patching stuff, but that is not really relevant here. >> >> The testsuite AS/WF builds are built out of the distribution build, which shows the same problem. The distribution we build uses the scripts we got from AS [1]. >> >> Do you see anything in there that could be causing this? We are using maven-antrun-plugin version 1.3, and take into account the lib.xml in [2]. >> >> Finally, do you have any suggestions on changes we could make to these files to further debug the issue? >> >> Thanks a lot for your help! >> >> [1] https://github.com/infinispan/infinispan/blob/master/server/integration/b... >> [2] https://github.com/infinispan/infinispan/blob/master/server/integration/b... >> >>> >>> Stuart >>> >>> >>> Stuart >>> >>> >>> On Tue, Feb 4, 2014 at 10:56 AM, Galder Zamarreño <galder(a)redhat.com&gt; wrote: >>> >>> On 04 Feb 2014, at 09:37, Stuart Douglas <stuart.w.douglas(a)gmail.com&gt; wrote: >>> >>>> This looks like an issue with your environment. The modules directory is static. Wildfly does not contain any code that messes with it. I would say the culprit is probably something in either your build process or your test suite. >>> >>> Correction, this is happening with AS 7.2.0.Final (Wildfly 8 used somewhere else). I guess your answer still applies? >>> >>> Cheers, >>> >>>> >>>> Stuart >>>> >>>> >>>> On Tue, Feb 4, 2014 at 10:21 AM, Galder Zamarreño <galder(a)redhat.com&gt; wrote: >>>> Hi all, >>>> >>>> We’re having issues with our Infinispan Server integration tests, which run within Wildfly 8.0.0.Beta1 (as I’m typing I’m wondering if we should just upgrade it to see if this goes away...?). >>>> >>>> Quite often some of the runs fail with error message [1]. >>>> >>>> Having looked at the build environment when a run fails, you see this: >>>> >>>> -- >>>> $ ls modules/system/layers/base/org/infinispan/server/rest/main >>>> drwxrwxr-x 2 g staff 68B Feb 3 18:41 classes (<— a directory??) >>>> -rw-r--r-- 1 g staff 1B Feb 3 18:41 classes.index >>>> -rw-r--r-- 1 g staff 2.1K Feb 3 18:41 module.xml >>>> >>>> $ ls modules/system/layers/base/org/infinispan/server/rest/main/classes >>>> drwxrwxr-x 2 g staff 68B Feb 3 18:41 . >>>> drwxrwxr-x 5 g staff 170B Feb 3 18:41 .. >>>> >>>> $ more modules/system/layers/base/org/infinispan/server/rest/main/module.xml >>>> <module xmlns="urn:jboss:module:1.1" name="org.infinispan.server.rest”> >>>> ... >>>> <resource-root path="classes"/> >>>> ... >>>> >>>> This is completely different to what happens with a successful run: >>>> >>>> -- >>>> $ ls modules/system/layers/base/org/infinispan/server/rest/main >>>> -rw-r--r-- 1 g staff 103K Feb 3 19:40 infinispan-classes.jar (<— a jar file!) >>>> -rw-r--r-- 1 g staff 278B Feb 3 19:40 infinispan-classes.jar.index >>>> -rw-r--r-- 1 g staff 2.1K Feb 3 19:40 module.xml >>>> >>>> $ jar tf modules/system/layers/base/org/infinispan/server/rest/main/infinispan-classes.jar | grep ExtendedHeaders >>>> org/infinispan/rest/configuration/ExtendedHeaders.class >>>> >>>> $ more modules/system/layers/base/org/infinispan/server/rest/main/module.xml >>>> <module xmlns="urn:jboss:module:1.1" name="org.infinispan.server.rest”> >>>> ... >>>> <resource-root path="infinispan-classes.jar”/> >>>> — >>>> >>>> Anyone can explain what is going on here? Does it ring a bell to anyone? Is this a known Wildfly issue by any chance? >>>> >>>> [1] https://gist.github.com/galderz/bd74cebfc840ef3ae284 >>>> >>>> Cheers, >>>> -- >>>> Galder Zamarreño >>>> galder(a)redhat.com >>>> twitter.com/galderz >>>> >>>> Project Lead, Escalante >>>> http://escalante.io >>>> >>>> Engineer, Infinispan >>>> http://infinispan.org >>>> >>>> >>>> _______________________________________________ >>>> jboss-as7-dev mailing list >>>> jboss-as7-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>> >>> >>> >>> -- >>> Galder Zamarreño >>> galder(a)redhat.com >>> twitter.com/galderz >>> >>> Project Lead, Escalante >>> http://escalante.io >>> >>> Engineer, Infinispan >>> http://infinispan.org >>> >>> >>> >> >> >> -- >> Galder Zamarreño >> galder(a)redhat.com >> twitter.com/galderz >> >> Project Lead, Escalante >> http://escalante.io >> >> Engineer, Infinispan >> http://infinispan.org >> >> > > > -- > Galder Zamarreño > galder(a)redhat.com > twitter.com/galderz > > Project Lead, Escalante > http://escalante.io > > Engineer, Infinispan > http://infinispan.org > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño galder(a)redhat.com twitter.com/galderz Project Lead, Escalante http://escalante.io Engineer, Infinispan http://infinispan.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev