[infinispan-dev] [jboss-as7-dev] Module jars dissapearing leaving empty classes/ folders and errors

Tue Feb 4 13:13:38 EST 2014

For the record, -Dmaven.test.failure.ignore seems to do the right thing,
and the JDK7 build now only has 7 test failures (+ 4 ignored):

http://ci.infinispan.org/viewLog.html?buildId=5912&tab=buildResultsDiv&buildTypeId=bt8

On Tue, Feb 4, 2014 at 4:36 PM, Galder Zamarreño <galder at redhat.com> wrote:

> All,
>
> Sanne, Pedro, Dan and I had a very productive discussion on IRC on this
> topic [1].
>
> We've decided that instead of disabling tests, we need them to run in
> order to get recent stacktraces, logs, etc. So, we've decided to create a
> new test group called "unstable". This test group would only be run in CI
> once a day and it'd be run in a different build. This build would also
> enable TRACE logging for standalone and server tests. For server, I need to
> create a task to do this selectively.
>
> The rest of builds, masters and PRS would not run the "unstable" group,
> and would not have TRACE enabled.
>
> The responsibility of unstable tests are the component owners. They need
> to handle them and decide what to do with them.
>
> Cheers,
>
> [1] https://gist.github.com/galderz/3563d1b23b5d50f80d82
>
> On 04 Feb 2014, at 15:10, Galder Zamarreño <galder at redhat.com> wrote:
>
> >
> > On 04 Feb 2014, at 14:47, Dan Berindei <dan.berindei at gmail.com> wrote:
> >
> >>
> >>
> >>
> >> On Tue, Feb 4, 2014 at 3:03 PM, Galder Zamarreño <galder at redhat.com>
> wrote:
> >>
> >> On 04 Feb 2014, at 13:50, Dan Berindei <dan.berindei at gmail.com> wrote:
> >>
> >>>
> >>> On Tue, Feb 4, 2014 at 2:36 PM, Galder Zamarreño <galder at redhat.com>
> wrote:
> >>> Narrowing down the list now, since this is a problem of how our CI is
> doing builds.
> >>>
> >>> These logs are retrieved from [1].
> >>>
> >>> Dunno how our CI is configured but this is odd. Seems like the build
> is halt due to test failures, but it continues somehow? I mean, the jars
> are not being produced properly, but the build is not halting.
> >>>
> >>> We run the build with -fn (fail-never), so the build should never be
> halted because of a test failure. The configuration is here:
> http://ci.infinispan.org/admin/editRunType.html?id=buildType:bt8&runnerId=RUNNER_1
> >>
> >> ^ That's not working as expected, see the build log, my snippets...etc.
> >>
> >> Sorry, I didn't understand what's happening in those snippets.
> >
> > The log shows it quite clearly that after those tests fail, nothing else
> runs in that module, including producing the jar. It halts. That's
> >
> >> All I saw was an Ant script that doesn't do what it's supposed to do :)
> >
> > The ant script not doing it's job is because there modules are not
> completing the build. There's a direct correlation between the three
> modules that fail with tests and the 3 modules that are copying an empty
> folder instead of the jar.
> >
> >>
> >> I did see some differences in the configuration between the JDK6 and
> the JDK7 builds:
> >> * JDK7 uses -fn and JDK6 uses -Dmaven.test.failure.ignore
> >> * JDK7 uses -nsu (no snapshot updates), JDK6 doesn't
> >>
> >> I've changed both builds to use -Dmaven.test.failure.ignore and -nsu,
> let's see how it goes.
> >
> > You are solving the wrong problem.
> >
> >>
> >>
> >>
> >>>
> >>>
> >>>
> >>> It's about time we did the following:
> >>> 1) Any test failures should halt the build there and then. IOW, do not
> continue the build at all.
> >>>
> >>> Will having 100 tests in one run and 2000 tests in another really help?
> >>
> >> As you disable randomly failing tests, and do not integrate commits
> making the testsuite fail, these number should even out.
> >>
> >> Not integrating commits that fail every time is easy, not integrating
> commits that fail randomly (maybe only in some environments) is trickier.
> >
> > I know it's tricky, but the only thing we can do is disable those
> really. I don't see how keeping them enabled is helping at all.
> >
> >>
> >>
> >>>
> >>>
> >>> 2) Any tests that fail randomly should be disabled.
> >>>
> >>> Let's go ahead and disable all the server tests then? ;)
> >>
> >> Those server tests that are randomly failing should be disabled and
> looked at. Those tests that are failing as a result of container not
> starting are side effects of things not working properly, and these should
> not be disabled.
> >>
> >> Why treat the tests that are failing because of a build problem
> differently? What about the tests that fail only on IBM JDK6?
> >
> > Disable and indicate that the test fails on IBM JDK6. Once the issue is
> fixed, reenable it.
> >
> >>
> >>
> >>>
> >>>
> >>>
> >>> Cheers,
> >>>
> >>> [1] https://dl.dropboxusercontent.com/u/6148072/does-not-work.log
> >>>
> >>> On 04 Feb 2014, at 13:30, Galder Zamarreño <galder at redhat.com> wrote:
> >>>
> >>>>
> >>>> On 04 Feb 2014, at 10:38, Stuart Douglas <stuart.w.douglas at gmail.com>
> wrote:
> >>>>
> >>>>> It is almost certainly something to do with this:
> >>>>>
> >>>>> <module-def name="org.infinispan.server.rest"
> src="${infinispan.server.modules.dir}">
> >>>>>
> >>>>>        <maven-resource-with-classifier group="org.infinispan"
> artifact="infinispan-server-rest" classifier="classes" />
> >>>>>
> >>>>> </module-def>
> >>>>>
> >>>>> I guess sometimes the classes artefact is being attached as a
> reference to the classes directory, rather than a reference to a jar, which
> causes the issue.
> >>>>
> >>>> Here's a gist with a subset of the build log [1]. When it works fine,
> it's copying a jar, when it's not, it's copying an empty folder.
> >>>>
> >>>> However, this is not only happening for the
> org.infinispan.server.rest module, others show the same issue [2]. What
> seems to be a pattern is that it only happens with modules that are built
> by us, it's not happening for modules coming with the base AS/WF instance.
> >>>>
> >>>> I've traced back and this might be due to build failures that are not
> producing the right jars [3].
> >>>>
> >>>> @Stuart, this is really our problem. Sorry for the inconvenience!
> >>>>
> >>>> [1] https://gist.github.com/galderz/b9286f385aad1316df51
> >>>> [2] https://gist.github.com/galderz/9e6a9bd9b18b805db323
> >>>> [3] https://gist.github.com/galderz/6ab662a1027cd96cbd8c
> >>>>
> >>>>>
> >>>>> Stuart
> >>>>>
> >>>>>
> >>>>> On Tue, Feb 4, 2014 at 11:14 AM, Galder Zamarreño <galder at redhat.com>
> wrote:
> >>>>>
> >>>>> On 04 Feb 2014, at 10:01, Stuart Douglas <stuart.w.douglas at gmail.com>
> wrote:
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Feb 4, 2014 at 11:00 AM, Stuart Douglas <
> stuart.w.douglas at gmail.com> wrote:
> >>>>>> Yes, there is nothing in the server code that modified the modules
> directory.
> >>>>>>
> >>>>>> Well, except for the new patching stuff, but that is not really
> relevant here.
> >>>>>
> >>>>> The testsuite AS/WF builds are built out of the distribution build,
> which shows the same problem. The distribution we build uses the scripts we
> got from AS [1].
> >>>>>
> >>>>> Do you see anything in there that could be causing this? We are
> using maven-antrun-plugin version 1.3, and take into account the lib.xml in
> [2].
> >>>>>
> >>>>> Finally, do you have any suggestions on changes we could make to
> these files to further debug the issue?
> >>>>>
> >>>>> Thanks a lot for your help!
> >>>>>
> >>>>> [1]
> https://github.com/infinispan/infinispan/blob/master/server/integration/build/build.xml
> >>>>> [2]
> https://github.com/infinispan/infinispan/blob/master/server/integration/build/lib.xml
> >>>>>
> >>>>>>
> >>>>>> Stuart
> >>>>>>
> >>>>>>
> >>>>>> Stuart
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Feb 4, 2014 at 10:56 AM, Galder Zamarreño <
> galder at redhat.com> wrote:
> >>>>>>
> >>>>>> On 04 Feb 2014, at 09:37, Stuart Douglas <
> stuart.w.douglas at gmail.com> wrote:
> >>>>>>
> >>>>>>> This looks like an issue with your environment. The modules
> directory is static. Wildfly does not contain any code that messes with it.
> I would say the culprit is probably something in either your build process
> or your test suite.
> >>>>>>
> >>>>>> Correction, this is happening with AS 7.2.0.Final (Wildfly 8 used
> somewhere else). I guess your answer still applies?
> >>>>>>
> >>>>>> Cheers,
> >>>>>>
> >>>>>>>
> >>>>>>> Stuart
> >>>>>>>
> >>>>>>>
> >>>>>>> On Tue, Feb 4, 2014 at 10:21 AM, Galder Zamarreño <
> galder at redhat.com> wrote:
> >>>>>>> Hi all,
> >>>>>>>
> >>>>>>> We're having issues with our Infinispan Server integration tests,
> which run within Wildfly 8.0.0.Beta1 (as I'm typing I'm wondering if we
> should just upgrade it to see if this goes away...?).
> >>>>>>>
> >>>>>>> Quite often some of the runs fail with error message [1].
> >>>>>>>
> >>>>>>> Having looked at the build environment when a run fails, you see
> this:
> >>>>>>>
> >>>>>>> --
> >>>>>>> $ ls modules/system/layers/base/org/infinispan/server/rest/main
> >>>>>>> drwxrwxr-x  2 g  staff    68B Feb  3 18:41 classes (<-- a
> directory??)
> >>>>>>> -rw-r--r--  1 g  staff     1B Feb  3 18:41 classes.index
> >>>>>>> -rw-r--r--  1 g  staff   2.1K Feb  3 18:41 module.xml
> >>>>>>>
> >>>>>>> $ ls
> modules/system/layers/base/org/infinispan/server/rest/main/classes
> >>>>>>> drwxrwxr-x  2 g  staff    68B Feb  3 18:41 .
> >>>>>>> drwxrwxr-x  5 g  staff   170B Feb  3 18:41 ..
> >>>>>>>
> >>>>>>> $ more
> modules/system/layers/base/org/infinispan/server/rest/main/module.xml
> >>>>>>> <module xmlns="urn:jboss:module:1.1"
> name="org.infinispan.server.rest">
> >>>>>>> ...
> >>>>>>> <resource-root path="classes"/>
> >>>>>>> ...
> >>>>>>>
> >>>>>>> This is completely different to what happens with a successful run:
> >>>>>>>
> >>>>>>> --
> >>>>>>> $ ls modules/system/layers/base/org/infinispan/server/rest/main
> >>>>>>> -rw-r--r--  1 g  staff   103K Feb  3 19:40 infinispan-classes.jar
> (<-- a jar file!)
> >>>>>>> -rw-r--r--  1 g  staff   278B Feb  3 19:40
> infinispan-classes.jar.index
> >>>>>>> -rw-r--r--  1 g  staff   2.1K Feb  3 19:40 module.xml
> >>>>>>>
> >>>>>>> $ jar tf
> modules/system/layers/base/org/infinispan/server/rest/main/infinispan-classes.jar
> | grep ExtendedHeaders
> >>>>>>> org/infinispan/rest/configuration/ExtendedHeaders.class
> >>>>>>>
> >>>>>>> $ more
> modules/system/layers/base/org/infinispan/server/rest/main/module.xml
> >>>>>>> <module xmlns="urn:jboss:module:1.1"
> name="org.infinispan.server.rest">
> >>>>>>> ...
> >>>>>>> <resource-root path="infinispan-classes.jar"/>
> >>>>>>> --
> >>>>>>>
> >>>>>>> Anyone can explain what is going on here? Does it ring a bell to
> anyone? Is this a known Wildfly issue by any chance?
> >>>>>>>
> >>>>>>> [1] https://gist.github.com/galderz/bd74cebfc840ef3ae284
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>> --
> >>>>>>> Galder Zamarreño
> >>>>>>> galder at redhat.com
> >>>>>>> twitter.com/galderz
> >>>>>>>
> >>>>>>> Project Lead, Escalante
> >>>>>>> http://escalante.io
> >>>>>>>
> >>>>>>> Engineer, Infinispan
> >>>>>>> http://infinispan.org
> >>>>>>>
> >>>>>>>
> >>>>>>> _______________________________________________
> >>>>>>> jboss-as7-dev mailing list
> >>>>>>> jboss-as7-dev at lists.jboss.org
> >>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Galder Zamarreño
> >>>>>> galder at redhat.com
> >>>>>> twitter.com/galderz
> >>>>>>
> >>>>>> Project Lead, Escalante
> >>>>>> http://escalante.io
> >>>>>>
> >>>>>> Engineer, Infinispan
> >>>>>> http://infinispan.org
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Galder Zamarreño
> >>>>> galder at redhat.com
> >>>>> twitter.com/galderz
> >>>>>
> >>>>> Project Lead, Escalante
> >>>>> http://escalante.io
> >>>>>
> >>>>> Engineer, Infinispan
> >>>>> http://infinispan.org
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Galder Zamarreño
> >>>> galder at redhat.com
> >>>> twitter.com/galderz
> >>>>
> >>>> Project Lead, Escalante
> >>>> http://escalante.io
> >>>>
> >>>> Engineer, Infinispan
> >>>> http://infinispan.org
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> infinispan-dev mailing list
> >>>> infinispan-dev at lists.jboss.org
> >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>>
> >>>
> >>> --
> >>> Galder Zamarreño
> >>> galder at redhat.com
> >>> twitter.com/galderz
> >>>
> >>> Project Lead, Escalante
> >>> http://escalante.io
> >>>
> >>> Engineer, Infinispan
> >>> http://infinispan.org
> >>>
> >>>
> >>> _______________________________________________
> >>> infinispan-dev mailing list
> >>> infinispan-dev at lists.jboss.org
> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>>
> >>> _______________________________________________
> >>> infinispan-dev mailing list
> >>> infinispan-dev at lists.jboss.org
> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> >>
> >> --
> >> Galder Zamarreño
> >> galder at redhat.com
> >> twitter.com/galderz
> >>
> >> Project Lead, Escalante
> >> http://escalante.io
> >>
> >> Engineer, Infinispan
> >> http://infinispan.org
> >>
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> >
> > --
> > Galder Zamarreño
> > galder at redhat.com
> > twitter.com/galderz
> >
> > Project Lead, Escalante
> > http://escalante.io
> >
> > Engineer, Infinispan
> > http://infinispan.org
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Galder Zamarreño
> galder at redhat.com
> twitter.com/galderz
>
> Project Lead, Escalante
> http://escalante.io
>
> Engineer, Infinispan
> http://infinispan.org
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20140204/2166b5b2/attachment-0001.html 