4 p.m.
Hi,
The new invocation library, which is the basis of for the ejb client etc. is currently
developed in a branch. There are still about 130-140 test failures, but the team feels it
is time to merge to wildfly master at some stage later this week. This will get more
visibility of the failures and also lower the barrier of entry for whoever can jump in and
help fix the failures.
Are there any objections to merging this?
From my point of view we would need to @Ignore the failing tests since
there are enough transient failures in our testsuite to make it hard to find usual
suspects if the number of failed tests is large. We could set up another CI job against a
branch which is master with the @Ignores reverted, and I could keep that up to date as I
merge to master, so that we get a good picture of the current testsuite failures.
Thanks,
Kabir
4:05 p.m.
I agree with this approach but we need a way to group these failures in such a way that we
know whether or not a change has regressions.
On Jan 31, 2017, at 9:03 AM, Kabir Khan <kabir.khan(a)jboss.com>
wrote:
Hi,
The new invocation library, which is the basis of for the ejb client etc. is currently
developed in a branch. There are still about 130-140 test failures, but the team feels it
is time to merge to wildfly master at some stage later this week. This will get more
visibility of the failures and also lower the barrier of entry for whoever can jump in and
help fix the failures.
Are there any objections to merging this?
> From my point of view we would need to @Ignore the failing tests since there are
enough transient failures in our testsuite to make it hard to find usual suspects if the
number of failed tests is large. We could set up another CI job against a branch which is
master with the @Ignores reverted, and I could keep that up to date as I merge to master,
so that we get a good picture of the current testsuite failures.
Thanks,
Kabir
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
4:09 p.m.
How about the tests are changed to require a sys prop. That way you don't need a
branch.
On Jan 31, 2017, at 9:07 AM, Jason T. Greene
<jason.greene(a)redhat.com> wrote:
I agree with this approach but we need a way to group these failures in such a way that
we know whether or not a change has regressions.
> On Jan 31, 2017, at 9:03 AM, Kabir Khan <kabir.khan(a)jboss.com> wrote:
>
> Hi,
>
> The new invocation library, which is the basis of for the ejb client etc. is
currently developed in a branch. There are still about 130-140 test failures, but the team
feels it is time to merge to wildfly master at some stage later this week. This will get
more visibility of the failures and also lower the barrier of entry for whoever can jump
in and help fix the failures.
>
> Are there any objections to merging this?
>
>> From my point of view we would need to @Ignore the failing tests since there are
enough transient failures in our testsuite to make it hard to find usual suspects if the
number of failed tests is large. We could set up another CI job against a branch which is
master with the @Ignores reverted, and I could keep that up to date as I merge to master,
so that we get a good picture of the current testsuite failures.
>
> Thanks,
>
> Kabir
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
4:14 p.m.
On 31 Jan 2017, at 15:09, Jason T. Greene <jason.greene(a)redhat.com> wrote:
How about the tests are changed to require a sys prop. That way you don't need a
branch.
So rather than @Ignore we add something like
@BeforeClass
public static checkRunInvocationTests() {
//WFLY-XXXX
Assume.assumeTrue(System.getProperty("wildfly.temp.disabled-invocation"))
}
> On Jan 31, 2017, at 9:07 AM, Jason T. Greene <jason.greene(a)redhat.com> wrote:
>
> I agree with this approach but we need a way to group these failures in such a way
that we know whether or not a change has regressions.
Can you elaborate a little
bit?
>
>> On Jan 31, 2017, at 9:03 AM, Kabir Khan <kabir.khan(a)jboss.com> wrote:
>>
>> Hi,
>>
>> The new invocation library, which is the basis of for the ejb client etc. is
currently developed in a branch. There are still about 130-140 test failures, but the team
feels it is time to merge to wildfly master at some stage later this week. This will get
more visibility of the failures and also lower the barrier of entry for whoever can jump
in and help fix the failures.
>>
>> Are there any objections to merging this?
>>
>>> From my point of view we would need to @Ignore the failing tests since there
are enough transient failures in our testsuite to make it hard to find usual suspects if
the number of failed tests is large. We could set up another CI job against a branch which
is master with the @Ignores reverted, and I could keep that up to date as I merge to
master, so that we get a good picture of the current testsuite failures.
>>
>> Thanks,
>>
>> Kabir
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
4:16 p.m.
Right exactly.
What does everyone think about a daily or weekly email of the test output to this list?
On Jan 31, 2017, at 9:14 AM, Kabir Khan <kabir.khan(a)jboss.com> wrote:
>
> On 31 Jan 2017, at 15:09, Jason T. Greene <jason.greene(a)redhat.com> wrote:
>
> How about the tests are changed to require a sys prop. That way you don't need a
branch.
So rather than @Ignore we add something like
@BeforeClass
public static checkRunInvocationTests() {
//WFLY-XXXX
Assume.assumeTrue(System.getProperty("wildfly.temp.disabled-invocation"))
}
>
>> On Jan 31, 2017, at 9:07 AM, Jason T. Greene <jason.greene(a)redhat.com>
wrote:
>>
>> I agree with this approach but we need a way to group these failures in such a
way that we know whether or not a change has regressions.
Can you elaborate a little bit?
>>
>>> On Jan 31, 2017, at 9:03 AM, Kabir Khan <kabir.khan(a)jboss.com> wrote:
>>>
>>> Hi,
>>>
>>> The new invocation library, which is the basis of for the ejb client etc. is
currently developed in a branch. There are still about 130-140 test failures, but the team
feels it is time to merge to wildfly master at some stage later this week. This will get
more visibility of the failures and also lower the barrier of entry for whoever can jump
in and help fix the failures.
>>>
>>> Are there any objections to merging this?
>>>
>>>> From my point of view we would need to @Ignore the failing tests since
there are enough transient failures in our testsuite to make it hard to find usual
suspects if the number of failed tests is large. We could set up another CI job against a
branch which is master with the @Ignores reverted, and I could keep that up to date as I
merge to master, so that we get a good picture of the current testsuite failures.
>>>
>>> Thanks,
>>>
>>> Kabir
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> wildfly-dev(a)lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
4:20 p.m.
On 31 Jan 2017, at 15:16, Jason T. Greene
<jason.greene(a)redhat.com> wrote:
Right exactly.
What does everyone think about a daily or weekly email of the test output to this list?
Daily would narrow down the range of suspicious commits. Although that information
should still be available on TeamCity itself, I think this makes it more visible.
(I take it this is the elaboration about the grouping of the failures? :))
On Jan 31, 2017, at 9:14 AM, Kabir Khan <kabir.khan(a)jboss.com> wrote:
>>
>> On 31 Jan 2017, at 15:09, Jason T. Greene <jason.greene(a)redhat.com> wrote:
>>
>> How about the tests are changed to require a sys prop. That way you don't
need a branch.
>
> So rather than @Ignore we add something like
>
> @BeforeClass
> public static checkRunInvocationTests() {
> //WFLY-XXXX
>
Assume.assumeTrue(System.getProperty("wildfly.temp.disabled-invocation"))
> }
>
>>
>>> On Jan 31, 2017, at 9:07 AM, Jason T. Greene <jason.greene(a)redhat.com>
wrote:
>>>
>>> I agree with this approach but we need a way to group these failures in such
a way that we know whether or not a change has regressions.
> Can you elaborate a little bit?
>>>
>>>> On Jan 31, 2017, at 9:03 AM, Kabir Khan <kabir.khan(a)jboss.com>
wrote:
>>>>
>>>> Hi,
>>>>
>>>> The new invocation library, which is the basis of for the ejb client etc.
is currently developed in a branch. There are still about 130-140 test failures, but the
team feels it is time to merge to wildfly master at some stage later this week. This will
get more visibility of the failures and also lower the barrier of entry for whoever can
jump in and help fix the failures.
>>>>
>>>> Are there any objections to merging this?
>>>>
>>>>> From my point of view we would need to @Ignore the failing tests
since there are enough transient failures in our testsuite to make it hard to find usual
suspects if the number of failed tests is large. We could set up another CI job against a
branch which is master with the @Ignores reverted, and I could keep that up to date as I
merge to master, so that we get a good picture of the current testsuite failures.
>>>>
>>>> Thanks,
>>>>
>>>> Kabir
>>>> _______________________________________________
>>>> wildfly-dev mailing list
>>>> wildfly-dev(a)lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> wildfly-dev(a)lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
5:28 p.m.
On Jan 31, 2017, at 9:20 AM, Kabir Khan <kabir.khan(a)jboss.com>
wrote:
> On 31 Jan 2017, at 15:16, Jason T. Greene <jason.greene(a)redhat.com> wrote:
>
> Right exactly.
>
> What does everyone think about a daily or weekly email of the test output to this
list?
Daily would narrow down the range of suspicious commits. Although that information should
still be available on TeamCity itself, I think this makes it more visible.
(I take it this is the elaboration about the grouping of the failures? :))
Right I just mean if the goal is to recruit help in resolving test failures, then
highlighting it here might be a good way to do that. It might be better than expecting
folks to periodically check a special run on team city.
On the other hand it can’t be too noisy, else everyone will just filter it.
This is just an idea, feedback from any and all is welcome.
--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat
7:42 a.m.
Same could be achieved by annotating classes with JUnit
@Category(InvocationIssue.class).
Then you can:
- run only InvocationIssue by specifying -Dgroups=InvocationIssue
- run all tests except of InvocationIssue by specifying
-DexcludedGroups=InvocationIssue
- run all tests by omitting any additional system property
Advandtage of @Category is you can annotate test method as well.
On Tue, Jan 31, 2017 at 4:14 PM, Kabir Khan <kabir.khan(a)jboss.com> wrote:
>
> On 31 Jan 2017, at 15:09, Jason T. Greene <jason.greene(a)redhat.com>
wrote:
>
> How about the tests are changed to require a sys prop. That way you
don't need a branch.
So rather than @Ignore we add something like
@BeforeClass
public static checkRunInvocationTests() {
//WFLY-XXXX
Assume.assumeTrue(System.getProperty("wildfly.temp.
disabled-invocation"))
}
>
>> On Jan 31, 2017, at 9:07 AM, Jason T. Greene <jason.greene(a)redhat.com>
wrote:
>>
>> I agree with this approach but we need a way to group these failures in
such a way that we know whether or not a change has regressions.
Can you elaborate a little bit?
>>
>>> On Jan 31, 2017, at 9:03 AM, Kabir Khan <kabir.khan(a)jboss.com> wrote:
>>>
>>> Hi,
>>>
>>> The new invocation library, which is the basis of for the ejb client
etc. is currently developed in a branch. There are still about 130-140 test
failures, but the team feels it is time to merge to wildfly master at some
stage later this week. This will get more visibility of the failures and
also lower the barrier of entry for whoever can jump in and help fix the
failures.
>>>
>>> Are there any objections to merging this?
>>>
>>>> From my point of view we would need to @Ignore the failing tests
since there are enough transient failures in our testsuite to make it hard
to find usual suspects if the number of failed tests is large. We could set
up another CI job against a branch which is master with the @Ignores
reverted, and I could keep that up to date as I merge to master, so that we
get a good picture of the current testsuite failures.
>>>
>>> Thanks,
>>>
>>> Kabir
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> wildfly-dev(a)lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
11:55 a.m.
On 1 Feb 2017, at 06:42, Martin Choma <mchoma(a)redhat.com>
wrote:
Same could be achieved by annotating classes with JUnit
@Category(InvocationIssue.class).
Then you can:
- run only InvocationIssue by specifying -Dgroups=InvocationIssue
- run all tests except of InvocationIssue by specifying
-DexcludedGroups=InvocationIssue
- run all tests by omitting any additional system property
Advandtage of @Category is you can annotate test method as well.
That does sound nice, but it sounds like omitting the tests becomes opt-in, rather than
opt-out. We would have to change all our CI jobs to use -DexcludedGroups=InvocationIssue,
apart from the new one we want to create to catch regressions/give up to date progress.
Or is there something which can be configured at pom level to reverse the behaviour?
On Tue, Jan 31, 2017 at 4:14 PM, Kabir Khan <kabir.khan(a)jboss.com> wrote:
>
> On 31 Jan 2017, at 15:09, Jason T. Greene <jason.greene(a)redhat.com> wrote:
>
> How about the tests are changed to require a sys prop. That way you don't need a
branch.
So rather than @Ignore we add something like
@BeforeClass
public static checkRunInvocationTests() {
//WFLY-XXXX
Assume.assumeTrue(System.getProperty("wildfly.temp.disabled-invocation"))
}
>
>> On Jan 31, 2017, at 9:07 AM, Jason T. Greene <jason.greene(a)redhat.com>
wrote:
>>
>> I agree with this approach but we need a way to group these failures in such a
way that we know whether or not a change has regressions.
Can you elaborate a little bit?
>>
>>> On Jan 31, 2017, at 9:03 AM, Kabir Khan <kabir.khan(a)jboss.com> wrote:
>>>
>>> Hi,
>>>
>>> The new invocation library, which is the basis of for the ejb client etc. is
currently developed in a branch. There are still about 130-140 test failures, but the team
feels it is time to merge to wildfly master at some stage later this week. This will get
more visibility of the failures and also lower the barrier of entry for whoever can jump
in and help fix the failures.
>>>
>>> Are there any objections to merging this?
>>>
>>>> From my point of view we would need to @Ignore the failing tests since
there are enough transient failures in our testsuite to make it hard to find usual
suspects if the number of failed tests is large. We could set up another CI job against a
branch which is master with the @Ignores reverted, and I could keep that up to date as I
merge to master, so that we get a good picture of the current testsuite failures.
>>>
>>> Thanks,
>>>
>>> Kabir
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> wildfly-dev(a)lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
2:41 p.m.
Right, I see.
You can add property in testsuite/pom.xml to switch InvocationIssue test by
default (works for me)
<properties>
<excludedGroups>InvocationIssue</excludedGroups>
New CI jobs have to be executed with empty excludedGroups, e.g.
-DexcludedGroups=
But I admit SP approach is more obvious here.
4:50 p.m.
Is there any way we can get this to report when a test that was
potentially excluded actually passed?
I think David has pointed out elsewhere TeamCity is good at showing us
new failures but not new passes.
Regards,
Darran Lofthouse.
On 01/02/17 13:41, Martin Choma wrote:
Right, I see.
You can add property in testsuite/pom.xml to switch InvocationIssue test
by default (works for me)
<properties>
<excludedGroups>InvocationIssue</excludedGroups>
New CI jobs have to be executed with empty excludedGroups, e.g.
-DexcludedGroups=
But I admit SP approach is more obvious here.
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
7:25 p.m.
I am going to give @Category a go - with Assume.assumeTrue I think we
have the problem that some tests may be failing in their initialisation
code and that risks leaving behind orphaned processes that break further
tests.
On 01/02/17 13:41, Martin Choma wrote:
Right, I see.
You can add property in testsuite/pom.xml to switch InvocationIssue test
by default (works for me)
<properties>
<excludedGroups>InvocationIssue</excludedGroups>
New CI jobs have to be executed with empty excludedGroups, e.g.
-DexcludedGroups=
But I admit SP approach is more obvious here.
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
7:36 p.m.
That was a short lived decision.
We can only use @Category with the Categories runner but we are already
using the Arquillian runner.
I think back to the system property.
On 02/02/17 18:25, Darran Lofthouse wrote:
I am going to give @Category a go - with Assume.assumeTrue I think
we
have the problem that some tests may be failing in their initialisation
code and that risks leaving behind orphaned processes that break further
tests.
On 01/02/17 13:41, Martin Choma wrote:
> Right, I see.
> You can add property in testsuite/pom.xml to switch InvocationIssue test
> by default (works for me)
>
> <properties>
> <excludedGroups>InvocationIssue</excludedGroups>
>
> New CI jobs have to be executed with empty excludedGroups, e.g.
> -DexcludedGroups=
>
> But I admit SP approach is more obvious here.
>
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
9:49 p.m.
What do you mean?
@Category is already used in testsuite in conjuction with Arquillian, e.g.
WebSecurityBASICTestCase.
On Thu, Feb 2, 2017 at 7:36 PM, Darran Lofthouse <darran.lofthouse(a)jboss.com
wrote:
That was a short lived decision.
We can only use @Category with the Categories runner but we are already
using the Arquillian runner.
I think back to the system property.
On 02/02/17 18:25, Darran Lofthouse wrote:
> I am going to give @Category a go - with Assume.assumeTrue I think we
> have the problem that some tests may be failing in their initialisation
> code and that risks leaving behind orphaned processes that break further
> tests.
>
> On 01/02/17 13:41, Martin Choma wrote:
>> Right, I see.
>> You can add property in testsuite/pom.xml to switch InvocationIssue test
>> by default (works for me)
>>
>> <properties>
>> <excludedGroups>InvocationIssue</excludedGroups>
>>
>> New CI jobs have to be executed with empty excludedGroups, e.g.
>> -DexcludedGroups=
>>
>> But I admit SP approach is more obvious here.
>>
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
9:54 p.m.
Maybe I misread the doc - it was suggesting to me you needed
@RunWith(Categories)
On 02/02/17 20:49, Martin Choma wrote:
What do you mean?
@Category is already used in testsuite in conjuction with Arquillian,
e.g. WebSecurityBASICTestCase.
On Thu, Feb 2, 2017 at 7:36 PM, Darran Lofthouse
<darran.lofthouse(a)jboss.com <mailto:darran.lofthouse@jboss.com>> wrote:
That was a short lived decision.
We can only use @Category with the Categories runner but we are already
using the Arquillian runner.
I think back to the system property.
On 02/02/17 18:25, Darran Lofthouse wrote:
> I am going to give @Category a go - with Assume.assumeTrue I think we
> have the problem that some tests may be failing in their
initialisation
> code and that risks leaving behind orphaned processes that break
further
> tests.
>
> On 01/02/17 13:41, Martin Choma wrote:
>> Right, I see.
>> You can add property in testsuite/pom.xml to switch
InvocationIssue test
>> by default (works for me)
>>
>> <properties>
>> <excludedGroups>InvocationIssue</excludedGroups>
>>
>> New CI jobs have to be executed with empty excludedGroups, e.g.
>> -DexcludedGroups=
>>
>> But I admit SP approach is more obvious here.
>>
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
<https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>>
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
<https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/wildfly-dev
<https://lists.jboss.org/mailman/listinfo/wildfly-dev>
4:14 p.m.
e.g assumeTrue(toBool(System.getProperty(Xxx)))
On Jan 31, 2017, at 9:09 AM, Jason T. Greene
<jason.greene(a)redhat.com> wrote:
How about the tests are changed to require a sys prop. That way you don't need a
branch.
> On Jan 31, 2017, at 9:07 AM, Jason T. Greene <jason.greene(a)redhat.com> wrote:
>
> I agree with this approach but we need a way to group these failures in such a way
that we know whether or not a change has regressions.
>
>> On Jan 31, 2017, at 9:03 AM, Kabir Khan <kabir.khan(a)jboss.com> wrote:
>>
>> Hi,
>>
>> The new invocation library, which is the basis of for the ejb client etc. is
currently developed in a branch. There are still about 130-140 test failures, but the team
feels it is time to merge to wildfly master at some stage later this week. This will get
more visibility of the failures and also lower the barrier of entry for whoever can jump
in and help fix the failures.
>>
>> Are there any objections to merging this?
>>
>>> From my point of view we would need to @Ignore the failing tests since there
are enough transient failures in our testsuite to make it hard to find usual suspects if
the number of failed tests is large. We could set up another CI job against a branch which
is master with the @Ignores reverted, and I could keep that up to date as I merge to
master, so that we get a good picture of the current testsuite failures.
>>
>> Thanks,
>>
>> Kabir
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
2639
days inactive
2641
days old
15 comments
5 participants
participants (5)
-
Darran Lofthouse -
Jason Greene -
Jason T. Greene
-
Kabir Khan -
Martin Choma