[infinispan-dev] [jboss-as7-dev] Module jars dissapearing leaving empty classes/ folders and errors

Galder Zamarreño galder at redhat.com
Tue Feb 4 09:10:35 EST 2014 

On 04 Feb 2014, at 14:47, Dan Berindei <dan.berindei at gmail.com> wrote:

> 
> 
> 
> On Tue, Feb 4, 2014 at 3:03 PM, Galder Zamarreño <galder at redhat.com> wrote:
> 
> On 04 Feb 2014, at 13:50, Dan Berindei <dan.berindei at gmail.com> wrote:
> 
> >
> > On Tue, Feb 4, 2014 at 2:36 PM, Galder Zamarreño <galder at redhat.com> wrote:
> > Narrowing down the list now, since this is a problem of how our CI is doing builds.
> >
> > These logs are retrieved from [1].
> >
> > Dunno how our CI is configured but this is odd. Seems like the build is halt due to test failures, but it continues somehow? I mean, the jars are not being produced properly, but the build is not halting.
> >
> > We run the build with -fn (fail-never), so the build should never be halted because of a test failure. The configuration is here: http://ci.infinispan.org/admin/editRunType.html?id=buildType:bt8&runnerId=RUNNER_1
> 
> ^ That’s not working as expected, see the build log, my snippets…etc.
> 
> Sorry, I didn't understand what's happening in those snippets.

The log shows it quite clearly that after those tests fail, nothing else runs in that module, including producing the jar. It halts. That’s 

> All I saw was an Ant script that doesn't do what it's supposed to do :)

The ant script not doing it’s job is because there modules are not completing the build. There’s a direct correlation between the three modules that fail with tests and the 3 modules that are copying an empty folder instead of the jar.

> 
> I did see some differences in the configuration between the JDK6 and the JDK7 builds:
> * JDK7 uses -fn and JDK6 uses -Dmaven.test.failure.ignore
> * JDK7 uses -nsu (no snapshot updates), JDK6 doesn't
> 
> I've changed both builds to use -Dmaven.test.failure.ignore and -nsu, let's see how it goes.

You are solving the wrong problem.

> 
>  
> 
> >
> >
> >
> > It’s about time we did the following:
> > 1) Any test failures should halt the build there and then. IOW, do not continue the build at all.
> >
> > Will having 100 tests in one run and 2000 tests in another really help?
> 
> As you disable randomly failing tests, and do not integrate commits making the testsuite fail, these number should even out.
> 
> Not integrating commits that fail every time is easy, not integrating commits that fail randomly (maybe only in some environments) is trickier.

I know it’s tricky, but the only thing we can do is disable those really. I don’t see how keeping them enabled is helping at all.

>  
> 
> >
> >
> > 2) Any tests that fail randomly should be disabled.
> >
> > Let's go ahead and disable all the server tests then? ;)
> 
> Those server tests that are randomly failing should be disabled and looked at. Those tests that are failing as a result of container not starting are side effects of things not working properly, and these should not be disabled.
> 
> Why treat the tests that are failing because of a build problem differently? What about the tests that fail only on IBM JDK6?

Disable and indicate that the test fails on IBM JDK6. Once the issue is fixed, reenable it.

> 
> 
> >
> >
> >
> > Cheers,
> >
> > [1] https://dl.dropboxusercontent.com/u/6148072/does-not-work.log
> >
> > On 04 Feb 2014, at 13:30, Galder Zamarreño <galder at redhat.com> wrote:
> >
> > >
> > > On 04 Feb 2014, at 10:38, Stuart Douglas <stuart.w.douglas at gmail.com> wrote:
> > >
> > >> It is almost certainly something to do with this:
> > >>
> > >> <module-def name="org.infinispan.server.rest" src="${infinispan.server.modules.dir}">
> > >>
> > >>         <maven-resource-with-classifier group="org.infinispan" artifact="infinispan-server-rest" classifier="classes" />
> > >>
> > >> </module-def>
> > >>
> > >> I guess sometimes the classes artefact is being attached as a reference to the classes directory, rather than a reference to a jar, which causes the issue.
> > >
> > > Here’s a gist with a subset of the build log [1]. When it works fine, it’s copying a jar, when it’s not, it’s copying an empty folder.
> > >
> > > However, this is not only happening for the org.infinispan.server.rest module, others show the same issue [2]. What seems to be a pattern is that it only happens with modules that are built by us, it’s not happening for modules coming with the base AS/WF instance.
> > >
> > > I’ve traced back and this might be due to build failures that are not producing the right jars [3].
> > >
> > > @Stuart, this is really our problem. Sorry for the inconvenience!
> > >
> > > [1] https://gist.github.com/galderz/b9286f385aad1316df51
> > > [2] https://gist.github.com/galderz/9e6a9bd9b18b805db323
> > > [3] https://gist.github.com/galderz/6ab662a1027cd96cbd8c
> > >
> > >>
> > >> Stuart
> > >>
> > >>
> > >> On Tue, Feb 4, 2014 at 11:14 AM, Galder Zamarreño <galder at redhat.com> wrote:
> > >>
> > >> On 04 Feb 2014, at 10:01, Stuart Douglas <stuart.w.douglas at gmail.com> wrote:
> > >>
> > >>>
> > >>>
> > >>>
> > >>> On Tue, Feb 4, 2014 at 11:00 AM, Stuart Douglas <stuart.w.douglas at gmail.com> wrote:
> > >>> Yes, there is nothing in the server code that modified the modules directory.
> > >>>
> > >>> Well, except for the new patching stuff, but that is not really relevant here.
> > >>
> > >> The testsuite AS/WF builds are built out of the distribution build, which shows the same problem. The distribution we build uses the scripts we got from AS [1].
> > >>
> > >> Do you see anything in there that could be causing this? We are using maven-antrun-plugin version 1.3, and take into account the lib.xml in [2].
> > >>
> > >> Finally, do you have any suggestions on changes we could make to these files to further debug the issue?
> > >>
> > >> Thanks a lot for your help!
> > >>
> > >> [1] https://github.com/infinispan/infinispan/blob/master/server/integration/build/build.xml
> > >> [2] https://github.com/infinispan/infinispan/blob/master/server/integration/build/lib.xml
> > >>
> > >>>
> > >>> Stuart
> > >>>
> > >>>
> > >>> Stuart
> > >>>
> > >>>
> > >>> On Tue, Feb 4, 2014 at 10:56 AM, Galder Zamarreño <galder at redhat.com> wrote:
> > >>>
> > >>> On 04 Feb 2014, at 09:37, Stuart Douglas <stuart.w.douglas at gmail.com> wrote:
> > >>>
> > >>>> This looks like an issue with your environment. The modules directory is static. Wildfly does not contain any code that messes with it. I would say the culprit is probably something in either your build process or your test suite.
> > >>>
> > >>> Correction, this is happening with AS 7.2.0.Final (Wildfly 8 used somewhere else). I guess your answer still applies?
> > >>>
> > >>> Cheers,
> > >>>
> > >>>>
> > >>>> Stuart
> > >>>>
> > >>>>
> > >>>> On Tue, Feb 4, 2014 at 10:21 AM, Galder Zamarreño <galder at redhat.com> wrote:
> > >>>> Hi all,
> > >>>>
> > >>>> We’re having issues with our Infinispan Server integration tests, which run within Wildfly 8.0.0.Beta1 (as I’m typing I’m wondering if we should just upgrade it to see if this goes away...?).
> > >>>>
> > >>>> Quite often some of the runs fail with error message [1].
> > >>>>
> > >>>> Having looked at the build environment when a run fails, you see this:
> > >>>>
> > >>>> --
> > >>>> $ ls modules/system/layers/base/org/infinispan/server/rest/main
> > >>>> drwxrwxr-x  2 g  staff    68B Feb  3 18:41 classes (<— a directory??)
> > >>>> -rw-r--r--  1 g  staff     1B Feb  3 18:41 classes.index
> > >>>> -rw-r--r--  1 g  staff   2.1K Feb  3 18:41 module.xml
> > >>>>
> > >>>> $ ls modules/system/layers/base/org/infinispan/server/rest/main/classes
> > >>>> drwxrwxr-x  2 g  staff    68B Feb  3 18:41 .
> > >>>> drwxrwxr-x  5 g  staff   170B Feb  3 18:41 ..
> > >>>>
> > >>>> $ more modules/system/layers/base/org/infinispan/server/rest/main/module.xml
> > >>>> <module xmlns="urn:jboss:module:1.1" name="org.infinispan.server.rest”>
> > >>>> ...
> > >>>> <resource-root path="classes"/>
> > >>>> ...
> > >>>>
> > >>>> This is completely different to what happens with a successful run:
> > >>>>
> > >>>> --
> > >>>> $ ls modules/system/layers/base/org/infinispan/server/rest/main
> > >>>> -rw-r--r--  1 g  staff   103K Feb  3 19:40 infinispan-classes.jar (<— a jar file!)
> > >>>> -rw-r--r--  1 g  staff   278B Feb  3 19:40 infinispan-classes.jar.index
> > >>>> -rw-r--r--  1 g  staff   2.1K Feb  3 19:40 module.xml
> > >>>>
> > >>>> $ jar tf modules/system/layers/base/org/infinispan/server/rest/main/infinispan-classes.jar | grep ExtendedHeaders
> > >>>> org/infinispan/rest/configuration/ExtendedHeaders.class
> > >>>>
> > >>>> $ more modules/system/layers/base/org/infinispan/server/rest/main/module.xml
> > >>>> <module xmlns="urn:jboss:module:1.1" name="org.infinispan.server.rest”>
> > >>>> ...
> > >>>> <resource-root path="infinispan-classes.jar”/>
> > >>>> —
> > >>>>
> > >>>> Anyone can explain what is going on here? Does it ring a bell to anyone? Is this a known Wildfly issue by any chance?
> > >>>>
> > >>>> [1] https://gist.github.com/galderz/bd74cebfc840ef3ae284
> > >>>>
> > >>>> Cheers,
> > >>>> --
> > >>>> Galder Zamarreño
> > >>>> galder at redhat.com
> > >>>> twitter.com/galderz
> > >>>>
> > >>>> Project Lead, Escalante
> > >>>> http://escalante.io
> > >>>>
> > >>>> Engineer, Infinispan
> > >>>> http://infinispan.org
> > >>>>
> > >>>>
> > >>>> _______________________________________________
> > >>>> jboss-as7-dev mailing list
> > >>>> jboss-as7-dev at lists.jboss.org
> > >>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
> > >>>>
> > >>>
> > >>>
> > >>> --
> > >>> Galder Zamarreño
> > >>> galder at redhat.com
> > >>> twitter.com/galderz
> > >>>
> > >>> Project Lead, Escalante
> > >>> http://escalante.io
> > >>>
> > >>> Engineer, Infinispan
> > >>> http://infinispan.org
> > >>>
> > >>>
> > >>>
> > >>
> > >>
> > >> --
> > >> Galder Zamarreño
> > >> galder at redhat.com
> > >> twitter.com/galderz
> > >>
> > >> Project Lead, Escalante
> > >> http://escalante.io
> > >>
> > >> Engineer, Infinispan
> > >> http://infinispan.org
> > >>
> > >>
> > >
> > >
> > > --
> > > Galder Zamarreño
> > > galder at redhat.com
> > > twitter.com/galderz
> > >
> > > Project Lead, Escalante
> > > http://escalante.io
> > >
> > > Engineer, Infinispan
> > > http://infinispan.org
> > >
> > >
> > > _______________________________________________
> > > infinispan-dev mailing list
> > > infinispan-dev at lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> >
> > --
> > Galder Zamarreño
> > galder at redhat.com
> > twitter.com/galderz
> >
> > Project Lead, Escalante
> > http://escalante.io
> >
> > Engineer, Infinispan
> > http://infinispan.org
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> 
> --
> Galder Zamarreño
> galder at redhat.com
> twitter.com/galderz
> 
> Project Lead, Escalante
> http://escalante.io
> 
> Engineer, Infinispan
> http://infinispan.org
> 
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


--
Galder Zamarreño
galder at redhat.com
twitter.com/galderz

Project Lead, Escalante
http://escalante.io

Engineer, Infinispan
http://infinispan.org

More information about the infinispan-dev mailing list