<div dir="ltr">For the record, -Dmaven.test.failure.ignore seems to do the right thing, and the JDK7 build now only has 7 test failures (+ 4 ignored): <br><br><a href="http://ci.infinispan.org/viewLog.html?buildId=5912&tab=buildResultsDiv&buildTypeId=bt8">http://ci.infinispan.org/viewLog.html?buildId=5912&tab=buildResultsDiv&buildTypeId=bt8</a><br>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Feb 4, 2014 at 4:36 PM, Galder Zamarreņo <span dir="ltr"><<a href="mailto:galder@redhat.com" target="_blank">galder@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">All,<br>
<br>
Sanne, Pedro, Dan and I had a very productive discussion on IRC on this topic [1].<br>
<br>
We’ve decided that instead of disabling tests, we need them to run in order to get recent stacktraces, logs, etc. So, we’ve decided to create a new test group called “unstable”. This test group would only be run in CI once a day and it’d be run in a different build. This build would also enable TRACE logging for standalone and server tests. For server, I need to create a task to do this selectively.<br>
<br>
The rest of builds, masters and PRS would not run the “unstable” group, and would not have TRACE enabled.<br>
<br>
The responsibility of unstable tests are the component owners. They need to handle them and decide what to do with them.<br>
<br>
Cheers,<br>
<br>
[1] <a href="https://gist.github.com/galderz/3563d1b23b5d50f80d82" target="_blank">https://gist.github.com/galderz/3563d1b23b5d50f80d82</a><br>
<div class="HOEnZb"><div class="h5"><br>
On 04 Feb 2014, at 15:10, Galder Zamarreņo <<a href="mailto:galder@redhat.com">galder@redhat.com</a>> wrote:<br>
<br>
><br>
> On 04 Feb 2014, at 14:47, Dan Berindei <<a href="mailto:dan.berindei@gmail.com">dan.berindei@gmail.com</a>> wrote:<br>
><br>
>><br>
>><br>
>><br>
>> On Tue, Feb 4, 2014 at 3:03 PM, Galder Zamarreņo <<a href="mailto:galder@redhat.com">galder@redhat.com</a>> wrote:<br>
>><br>
>> On 04 Feb 2014, at 13:50, Dan Berindei <<a href="mailto:dan.berindei@gmail.com">dan.berindei@gmail.com</a>> wrote:<br>
>><br>
>>><br>
>>> On Tue, Feb 4, 2014 at 2:36 PM, Galder Zamarreņo <<a href="mailto:galder@redhat.com">galder@redhat.com</a>> wrote:<br>
>>> Narrowing down the list now, since this is a problem of how our CI is doing builds.<br>
>>><br>
>>> These logs are retrieved from [1].<br>
>>><br>
>>> Dunno how our CI is configured but this is odd. Seems like the build is halt due to test failures, but it continues somehow? I mean, the jars are not being produced properly, but the build is not halting.<br>
>>><br>
>>> We run the build with -fn (fail-never), so the build should never be halted because of a test failure. The configuration is here: <a href="http://ci.infinispan.org/admin/editRunType.html?id=buildType:bt8&runnerId=RUNNER_1" target="_blank">http://ci.infinispan.org/admin/editRunType.html?id=buildType:bt8&runnerId=RUNNER_1</a><br>
>><br>
>> ^ That’s not working as expected, see the build log, my snippets…etc.<br>
>><br>
>> Sorry, I didn't understand what's happening in those snippets.<br>
><br>
> The log shows it quite clearly that after those tests fail, nothing else runs in that module, including producing the jar. It halts. That’s<br>
><br>
>> All I saw was an Ant script that doesn't do what it's supposed to do :)<br>
><br>
> The ant script not doing it’s job is because there modules are not completing the build. There’s a direct correlation between the three modules that fail with tests and the 3 modules that are copying an empty folder instead of the jar.<br>
><br>
>><br>
>> I did see some differences in the configuration between the JDK6 and the JDK7 builds:<br>
>> * JDK7 uses -fn and JDK6 uses -Dmaven.test.failure.ignore<br>
>> * JDK7 uses -nsu (no snapshot updates), JDK6 doesn't<br>
>><br>
>> I've changed both builds to use -Dmaven.test.failure.ignore and -nsu, let's see how it goes.<br>
><br>
> You are solving the wrong problem.<br>
><br>
>><br>
>><br>
>><br>
>>><br>
>>><br>
>>><br>
>>> It’s about time we did the following:<br>
>>> 1) Any test failures should halt the build there and then. IOW, do not continue the build at all.<br>
>>><br>
>>> Will having 100 tests in one run and 2000 tests in another really help?<br>
>><br>
>> As you disable randomly failing tests, and do not integrate commits making the testsuite fail, these number should even out.<br>
>><br>
>> Not integrating commits that fail every time is easy, not integrating commits that fail randomly (maybe only in some environments) is trickier.<br>
><br>
> I know it’s tricky, but the only thing we can do is disable those really. I don’t see how keeping them enabled is helping at all.<br>
><br>
>><br>
>><br>
>>><br>
>>><br>
>>> 2) Any tests that fail randomly should be disabled.<br>
>>><br>
>>> Let's go ahead and disable all the server tests then? ;)<br>
>><br>
>> Those server tests that are randomly failing should be disabled and looked at. Those tests that are failing as a result of container not starting are side effects of things not working properly, and these should not be disabled.<br>
>><br>
>> Why treat the tests that are failing because of a build problem differently? What about the tests that fail only on IBM JDK6?<br>
><br>
> Disable and indicate that the test fails on IBM JDK6. Once the issue is fixed, reenable it.<br>
><br>
>><br>
>><br>
>>><br>
>>><br>
>>><br>
>>> Cheers,<br>
>>><br>
>>> [1] <a href="https://dl.dropboxusercontent.com/u/6148072/does-not-work.log" target="_blank">https://dl.dropboxusercontent.com/u/6148072/does-not-work.log</a><br>
>>><br>
>>> On 04 Feb 2014, at 13:30, Galder Zamarreņo <<a href="mailto:galder@redhat.com">galder@redhat.com</a>> wrote:<br>
>>><br>
>>>><br>
>>>> On 04 Feb 2014, at 10:38, Stuart Douglas <<a href="mailto:stuart.w.douglas@gmail.com">stuart.w.douglas@gmail.com</a>> wrote:<br>
>>>><br>
>>>>> It is almost certainly something to do with this:<br>
>>>>><br>
>>>>> <module-def name="org.infinispan.server.rest" src="${infinispan.server.modules.dir}"><br>
>>>>><br>
>>>>> <maven-resource-with-classifier group="org.infinispan" artifact="infinispan-server-rest" classifier="classes" /><br>
>>>>><br>
>>>>> </module-def><br>
>>>>><br>
>>>>> I guess sometimes the classes artefact is being attached as a reference to the classes directory, rather than a reference to a jar, which causes the issue.<br>
>>>><br>
>>>> Here’s a gist with a subset of the build log [1]. When it works fine, it’s copying a jar, when it’s not, it’s copying an empty folder.<br>
>>>><br>
>>>> However, this is not only happening for the org.infinispan.server.rest module, others show the same issue [2]. What seems to be a pattern is that it only happens with modules that are built by us, it’s not happening for modules coming with the base AS/WF instance.<br>
>>>><br>
>>>> I’ve traced back and this might be due to build failures that are not producing the right jars [3].<br>
>>>><br>
>>>> @Stuart, this is really our problem. Sorry for the inconvenience!<br>
>>>><br>
>>>> [1] <a href="https://gist.github.com/galderz/b9286f385aad1316df51" target="_blank">https://gist.github.com/galderz/b9286f385aad1316df51</a><br>
>>>> [2] <a href="https://gist.github.com/galderz/9e6a9bd9b18b805db323" target="_blank">https://gist.github.com/galderz/9e6a9bd9b18b805db323</a><br>
>>>> [3] <a href="https://gist.github.com/galderz/6ab662a1027cd96cbd8c" target="_blank">https://gist.github.com/galderz/6ab662a1027cd96cbd8c</a><br>
>>>><br>
>>>>><br>
>>>>> Stuart<br>
>>>>><br>
>>>>><br>
>>>>> On Tue, Feb 4, 2014 at 11:14 AM, Galder Zamarreņo <<a href="mailto:galder@redhat.com">galder@redhat.com</a>> wrote:<br>
>>>>><br>
>>>>> On 04 Feb 2014, at 10:01, Stuart Douglas <<a href="mailto:stuart.w.douglas@gmail.com">stuart.w.douglas@gmail.com</a>> wrote:<br>
>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>> On Tue, Feb 4, 2014 at 11:00 AM, Stuart Douglas <<a href="mailto:stuart.w.douglas@gmail.com">stuart.w.douglas@gmail.com</a>> wrote:<br>
>>>>>> Yes, there is nothing in the server code that modified the modules directory.<br>
>>>>>><br>
>>>>>> Well, except for the new patching stuff, but that is not really relevant here.<br>
>>>>><br>
>>>>> The testsuite AS/WF builds are built out of the distribution build, which shows the same problem. The distribution we build uses the scripts we got from AS [1].<br>
>>>>><br>
>>>>> Do you see anything in there that could be causing this? We are using maven-antrun-plugin version 1.3, and take into account the lib.xml in [2].<br>
>>>>><br>
>>>>> Finally, do you have any suggestions on changes we could make to these files to further debug the issue?<br>
>>>>><br>
>>>>> Thanks a lot for your help!<br>
>>>>><br>
>>>>> [1] <a href="https://github.com/infinispan/infinispan/blob/master/server/integration/build/build.xml" target="_blank">https://github.com/infinispan/infinispan/blob/master/server/integration/build/build.xml</a><br>
>>>>> [2] <a href="https://github.com/infinispan/infinispan/blob/master/server/integration/build/lib.xml" target="_blank">https://github.com/infinispan/infinispan/blob/master/server/integration/build/lib.xml</a><br>
>>>>><br>
>>>>>><br>
>>>>>> Stuart<br>
>>>>>><br>
>>>>>><br>
>>>>>> Stuart<br>
>>>>>><br>
>>>>>><br>
>>>>>> On Tue, Feb 4, 2014 at 10:56 AM, Galder Zamarreņo <<a href="mailto:galder@redhat.com">galder@redhat.com</a>> wrote:<br>
>>>>>><br>
>>>>>> On 04 Feb 2014, at 09:37, Stuart Douglas <<a href="mailto:stuart.w.douglas@gmail.com">stuart.w.douglas@gmail.com</a>> wrote:<br>
>>>>>><br>
>>>>>>> This looks like an issue with your environment. The modules directory is static. Wildfly does not contain any code that messes with it. I would say the culprit is probably something in either your build process or your test suite.<br>
>>>>>><br>
>>>>>> Correction, this is happening with AS 7.2.0.Final (Wildfly 8 used somewhere else). I guess your answer still applies?<br>
>>>>>><br>
>>>>>> Cheers,<br>
>>>>>><br>
>>>>>>><br>
>>>>>>> Stuart<br>
>>>>>>><br>
>>>>>>><br>
>>>>>>> On Tue, Feb 4, 2014 at 10:21 AM, Galder Zamarreņo <<a href="mailto:galder@redhat.com">galder@redhat.com</a>> wrote:<br>
>>>>>>> Hi all,<br>
>>>>>>><br>
>>>>>>> We’re having issues with our Infinispan Server integration tests, which run within Wildfly 8.0.0.Beta1 (as I’m typing I’m wondering if we should just upgrade it to see if this goes away...?).<br>
>>>>>>><br>
>>>>>>> Quite often some of the runs fail with error message [1].<br>
>>>>>>><br>
>>>>>>> Having looked at the build environment when a run fails, you see this:<br>
>>>>>>><br>
>>>>>>> --<br>
>>>>>>> $ ls modules/system/layers/base/org/infinispan/server/rest/main<br>
>>>>>>> drwxrwxr-x 2 g staff 68B Feb 3 18:41 classes (<— a directory??)<br>
>>>>>>> -rw-r--r-- 1 g staff 1B Feb 3 18:41 classes.index<br>
>>>>>>> -rw-r--r-- 1 g staff 2.1K Feb 3 18:41 module.xml<br>
>>>>>>><br>
>>>>>>> $ ls modules/system/layers/base/org/infinispan/server/rest/main/classes<br>
>>>>>>> drwxrwxr-x 2 g staff 68B Feb 3 18:41 .<br>
>>>>>>> drwxrwxr-x 5 g staff 170B Feb 3 18:41 ..<br>
>>>>>>><br>
>>>>>>> $ more modules/system/layers/base/org/infinispan/server/rest/main/module.xml<br>
>>>>>>> <module xmlns="urn:jboss:module:1.1" name="org.infinispan.server.rest”><br>
>>>>>>> ...<br>
>>>>>>> <resource-root path="classes"/><br>
>>>>>>> ...<br>
>>>>>>><br>
>>>>>>> This is completely different to what happens with a successful run:<br>
>>>>>>><br>
>>>>>>> --<br>
>>>>>>> $ ls modules/system/layers/base/org/infinispan/server/rest/main<br>
>>>>>>> -rw-r--r-- 1 g staff 103K Feb 3 19:40 infinispan-classes.jar (<— a jar file!)<br>
>>>>>>> -rw-r--r-- 1 g staff 278B Feb 3 19:40 infinispan-classes.jar.index<br>
>>>>>>> -rw-r--r-- 1 g staff 2.1K Feb 3 19:40 module.xml<br>
>>>>>>><br>
>>>>>>> $ jar tf modules/system/layers/base/org/infinispan/server/rest/main/infinispan-classes.jar | grep ExtendedHeaders<br>
>>>>>>> org/infinispan/rest/configuration/ExtendedHeaders.class<br>
>>>>>>><br>
>>>>>>> $ more modules/system/layers/base/org/infinispan/server/rest/main/module.xml<br>
>>>>>>> <module xmlns="urn:jboss:module:1.1" name="org.infinispan.server.rest”><br>
>>>>>>> ...<br>
>>>>>>> <resource-root path="infinispan-classes.jar”/><br>
>>>>>>> —<br>
>>>>>>><br>
>>>>>>> Anyone can explain what is going on here? Does it ring a bell to anyone? Is this a known Wildfly issue by any chance?<br>
>>>>>>><br>
>>>>>>> [1] <a href="https://gist.github.com/galderz/bd74cebfc840ef3ae284" target="_blank">https://gist.github.com/galderz/bd74cebfc840ef3ae284</a><br>
>>>>>>><br>
>>>>>>> Cheers,<br>
>>>>>>> --<br>
>>>>>>> Galder Zamarreņo<br>
>>>>>>> <a href="mailto:galder@redhat.com">galder@redhat.com</a><br>
>>>>>>> <a href="http://twitter.com/galderz" target="_blank">twitter.com/galderz</a><br>
>>>>>>><br>
>>>>>>> Project Lead, Escalante<br>
>>>>>>> <a href="http://escalante.io" target="_blank">http://escalante.io</a><br>
>>>>>>><br>
>>>>>>> Engineer, Infinispan<br>
>>>>>>> <a href="http://infinispan.org" target="_blank">http://infinispan.org</a><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>> _______________________________________________<br>
>>>>>>> jboss-as7-dev mailing list<br>
>>>>>>> <a href="mailto:jboss-as7-dev@lists.jboss.org">jboss-as7-dev@lists.jboss.org</a><br>
>>>>>>> <a href="https://lists.jboss.org/mailman/listinfo/jboss-as7-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/jboss-as7-dev</a><br>
>>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>> --<br>
>>>>>> Galder Zamarreņo<br>
>>>>>> <a href="mailto:galder@redhat.com">galder@redhat.com</a><br>
>>>>>> <a href="http://twitter.com/galderz" target="_blank">twitter.com/galderz</a><br>
>>>>>><br>
>>>>>> Project Lead, Escalante<br>
>>>>>> <a href="http://escalante.io" target="_blank">http://escalante.io</a><br>
>>>>>><br>
>>>>>> Engineer, Infinispan<br>
>>>>>> <a href="http://infinispan.org" target="_blank">http://infinispan.org</a><br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>><br>
>>>>><br>
>>>>> --<br>
>>>>> Galder Zamarreņo<br>
>>>>> <a href="mailto:galder@redhat.com">galder@redhat.com</a><br>
>>>>> <a href="http://twitter.com/galderz" target="_blank">twitter.com/galderz</a><br>
>>>>><br>
>>>>> Project Lead, Escalante<br>
>>>>> <a href="http://escalante.io" target="_blank">http://escalante.io</a><br>
>>>>><br>
>>>>> Engineer, Infinispan<br>
>>>>> <a href="http://infinispan.org" target="_blank">http://infinispan.org</a><br>
>>>>><br>
>>>>><br>
>>>><br>
>>>><br>
>>>> --<br>
>>>> Galder Zamarreņo<br>
>>>> <a href="mailto:galder@redhat.com">galder@redhat.com</a><br>
>>>> <a href="http://twitter.com/galderz" target="_blank">twitter.com/galderz</a><br>
>>>><br>
>>>> Project Lead, Escalante<br>
>>>> <a href="http://escalante.io" target="_blank">http://escalante.io</a><br>
>>>><br>
>>>> Engineer, Infinispan<br>
>>>> <a href="http://infinispan.org" target="_blank">http://infinispan.org</a><br>
>>>><br>
>>>><br>
>>>> _______________________________________________<br>
>>>> infinispan-dev mailing list<br>
>>>> <a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
>>>> <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
>>><br>
>>><br>
>>> --<br>
>>> Galder Zamarreņo<br>
>>> <a href="mailto:galder@redhat.com">galder@redhat.com</a><br>
>>> <a href="http://twitter.com/galderz" target="_blank">twitter.com/galderz</a><br>
>>><br>
>>> Project Lead, Escalante<br>
>>> <a href="http://escalante.io" target="_blank">http://escalante.io</a><br>
>>><br>
>>> Engineer, Infinispan<br>
>>> <a href="http://infinispan.org" target="_blank">http://infinispan.org</a><br>
>>><br>
>>><br>
>>> _______________________________________________<br>
>>> infinispan-dev mailing list<br>
>>> <a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
>>> <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
>>><br>
>>> _______________________________________________<br>
>>> infinispan-dev mailing list<br>
>>> <a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
>>> <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
>><br>
>><br>
>> --<br>
>> Galder Zamarreņo<br>
>> <a href="mailto:galder@redhat.com">galder@redhat.com</a><br>
>> <a href="http://twitter.com/galderz" target="_blank">twitter.com/galderz</a><br>
>><br>
>> Project Lead, Escalante<br>
>> <a href="http://escalante.io" target="_blank">http://escalante.io</a><br>
>><br>
>> Engineer, Infinispan<br>
>> <a href="http://infinispan.org" target="_blank">http://infinispan.org</a><br>
>><br>
>><br>
>> _______________________________________________<br>
>> infinispan-dev mailing list<br>
>> <a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
>> <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
>><br>
>> _______________________________________________<br>
>> infinispan-dev mailing list<br>
>> <a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
>> <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
><br>
><br>
> --<br>
> Galder Zamarreņo<br>
> <a href="mailto:galder@redhat.com">galder@redhat.com</a><br>
> <a href="http://twitter.com/galderz" target="_blank">twitter.com/galderz</a><br>
><br>
> Project Lead, Escalante<br>
> <a href="http://escalante.io" target="_blank">http://escalante.io</a><br>
><br>
> Engineer, Infinispan<br>
> <a href="http://infinispan.org" target="_blank">http://infinispan.org</a><br>
><br>
><br>
> _______________________________________________<br>
> infinispan-dev mailing list<br>
> <a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
<br>
<br>
--<br>
Galder Zamarreņo<br>
<a href="mailto:galder@redhat.com">galder@redhat.com</a><br>
<a href="http://twitter.com/galderz" target="_blank">twitter.com/galderz</a><br>
<br>
Project Lead, Escalante<br>
<a href="http://escalante.io" target="_blank">http://escalante.io</a><br>
<br>
Engineer, Infinispan<br>
<a href="http://infinispan.org" target="_blank">http://infinispan.org</a><br>
<br>
<br>
_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
</div></div></blockquote></div><br></div>