3:11 a.m.
Hi,
EAP QE and Elytron Dev teams would like to raise a discussion about how the Elytron
integration tests will be added to the WildFly/EAP testsuite.
We would like to agree on a unified approach across the testsuite, so we don't end up
in situation where each component comes with its own "best-fitting" solution.
We see following requirements:
1. new features which come with Elytron need to be covered by integration tests
2. existing scenarios have to stay working when Elytron is configured as the default
security subsystem
3. legacy security subsystem must stay working in parallel
Our teams came to agreement that the safest way would be to copy whole existing testsuite
modules into new ones. E.g.
cp -r testsuite/integration/basic testsuite/integration/basic-legacy-security
# + Reconfigure testsuite/integration/basic to use Elytron
# + Add @Ignore to all tests which fail with Elytron or use directly the old security
configuration
The modules would just live side by side - basic would use Elytron configuration,
basic-legacy-security would use configuration similar to (or same as) the current server
configuration.
In the next step we will go through the Ignored tests (the TODO list) for Elytron and
rewrite them or report issues if necessary.
By this way we would keep all the test coverage for legacy security. We would also have
all existing scenarios covered (or at least present) for Elytron configuration.
Are there any objections to this suggestion?
Thanks,
-- Josef
6:03 a.m.
On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek <jcacek(a)redhat.com> wrote:
The modules would just live side by side - basic would use Elytron
configuration, basic-legacy-security would use configuration similar to (or
same as) the current server configuration.
What would this actually mean?
we will have two copies basic tests suites one running with elytron another
with legacy security subsystem?
Do I read that right? Please say I am not.
6:08 a.m.
On 02/12/16 11:03, Tomaž Cerar wrote:
On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek <jcacek(a)redhat.com
<mailto:jcacek@redhat.com>> wrote:
The modules would just live side by side - basic would use Elytron
configuration, basic-legacy-security would use configuration similar
to (or same as) the current server configuration.
What would this actually mean?
we will have two copies basic tests suites one running with elytron
another with legacy security subsystem?
Do I read that right? Please say I am not.
That is correct - we have two security implementations they both need
testing.
One needs testing for backwards compatibility and regressions, the other
for equivalent behaviour and then new features and bugs.
Needing to test both was discussed previously so this is more about how
to separate both and also give the Elytron testing a good foundation to
start from.
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
6:09 a.m.
Probably should add - any duplication should only be for security tests
- not everything else in there!
On 02/12/16 11:08, Darran Lofthouse wrote:
On 02/12/16 11:03, Tomaž Cerar wrote:
>
> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek <jcacek(a)redhat.com
> <mailto:jcacek@redhat.com>> wrote:
>
> The modules would just live side by side - basic would use Elytron
> configuration, basic-legacy-security would use configuration similar
> to (or same as) the current server configuration.
>
>
>
> What would this actually mean?
> we will have two copies basic tests suites one running with elytron
> another with legacy security subsystem?
>
> Do I read that right? Please say I am not.
That is correct - we have two security implementations they both need
testing.
One needs testing for backwards compatibility and regressions, the other
for equivalent behaviour and then new features and bugs.
Needing to test both was discussed previously so this is more about how
to separate both and also give the Elytron testing a good foundation to
start from.
>
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
6:37 a.m.
That is probably fine, but! it should be done differently.
instead of duplicating whole testsuite (and adding extra hour to execution
and extra headaches with intermittent problems and duplication of
maintenance)
I would suggest that all security related tests get extracted to new
"security" testsuite module and than only that part is duplicated.
This way we will have all security related stuff in one place.
On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse <
darran.lofthouse(a)jboss.com> wrote:
Probably should add - any duplication should only be for security
tests
- not everything else in there!
On 02/12/16 11:08, Darran Lofthouse wrote:
> On 02/12/16 11:03, Tomaž Cerar wrote:
>>
>> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek <jcacek(a)redhat.com
>> <mailto:jcacek@redhat.com>> wrote:
>>
>> The modules would just live side by side - basic would use Elytron
>> configuration, basic-legacy-security would use configuration similar
>> to (or same as) the current server configuration.
>>
>>
>>
>> What would this actually mean?
>> we will have two copies basic tests suites one running with elytron
>> another with legacy security subsystem?
>>
>> Do I read that right? Please say I am not.
>
> That is correct - we have two security implementations they both need
> testing.
>
> One needs testing for backwards compatibility and regressions, the other
> for equivalent behaviour and then new features and bugs.
>
> Needing to test both was discussed previously so this is more about how
> to separate both and also give the Elytron testing a good foundation to
> start from.
>
>>
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
>
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
6:45 a.m.
Although after my e-mail I hear from Josef they do want to duplicate all ;-)
From my perspective I think we want to ensure the security tests are
clearly split in some way so we can identify legacy tests and Elytron
tests independently.
When it comes to code review the legacy tests are much more about
backwards compatibility so any changes to those should be given greater
scrutiny.
Secondly we should hit a point where we can remove the old tests so
separation will make that easier.
A package split would be fine but maybe a project split would make it
easier to define which tests are run and when.
Instead maybe we could split it into three: -
basic, basic-legacy-security, basic-security
The two 'security' projects would be the different form of security
testing, maybe the 'basic' project could have an optional profile to
switch it to a 100% legacy config mode.
On 02/12/16 11:37, Tomaž Cerar wrote:
That is probably fine, but! it should be done differently.
instead of duplicating whole testsuite (and adding extra hour to
execution and extra headaches with intermittent problems and duplication
of maintenance)
I would suggest that all security related tests get extracted to new
"security" testsuite module and than only that part is duplicated.
This way we will have all security related stuff in one place.
On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse
<darran.lofthouse(a)jboss.com <mailto:darran.lofthouse@jboss.com>> wrote:
Probably should add - any duplication should only be for security tests
- not everything else in there!
On 02/12/16 11:08, Darran Lofthouse wrote:
> On 02/12/16 11:03, Tomaž Cerar wrote:
>>
>> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek <jcacek(a)redhat.com
<mailto:jcacek@redhat.com>
>> <mailto:jcacek@redhat.com <mailto:jcacek@redhat.com>>>
wrote:
>>
>> The modules would just live side by side - basic would use
Elytron
>> configuration, basic-legacy-security would use configuration
similar
>> to (or same as) the current server configuration.
>>
>>
>>
>> What would this actually mean?
>> we will have two copies basic tests suites one running with elytron
>> another with legacy security subsystem?
>>
>> Do I read that right? Please say I am not.
>
> That is correct - we have two security implementations they both need
> testing.
>
> One needs testing for backwards compatibility and regressions, the
other
> for equivalent behaviour and then new features and bugs.
>
> Needing to test both was discussed previously so this is more
about how
> to separate both and also give the Elytron testing a good
foundation to
> start from.
>
>>
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
<https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>>
>
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/wildfly-dev
<https://lists.jboss.org/mailman/listinfo/wildfly-dev>
6:58 a.m.
We prefer duplication of all tests in the testsuite to keep test coverage.
To be sure we don't have regressions with introducing Elytron and at the same time we
are able to cover current features with Elytron, we must run all tests with both security
subsystems.
Even if a test is not security related on the first glance, the tested component may use a
security feature internally. By switching to the new security subsystem the feature may
stop work.
Let's keep the current (legacy security based) tests alive until the legacy security
subsystem is fully removed.
Then we'll simply remove the *-legacy-security testsuite modules.
-- Josef
----- Original Message -----
From: "Tomaž Cerar" <tomaz.cerar(a)gmail.com>
To: "Darran Lofthouse" <darran.lofthouse(a)jboss.com>
Cc: wildfly-dev(a)lists.jboss.org
Sent: Friday, December 2, 2016 12:37:22 PM
Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite
That is probably fine, but! it should be done differently.
instead of duplicating whole testsuite (and adding extra hour to execution
and extra headaches with intermittent problems and duplication of
maintenance)
I would suggest that all security related tests get extracted to new
"security" testsuite module and than only that part is duplicated.
This way we will have all security related stuff in one place.
On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse <
darran.lofthouse(a)jboss.com > wrote:
Probably should add - any duplication should only be for security tests
- not everything else in there!
On 02/12/16 11:08, Darran Lofthouse wrote:
> On 02/12/16 11:03, Tomaž Cerar wrote:
>>
>> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek < jcacek(a)redhat.com
>> <mailto: jcacek(a)redhat.com >> wrote:
>>
>> The modules would just live side by side - basic would use Elytron
>> configuration, basic-legacy-security would use configuration similar
>> to (or same as) the current server configuration.
>>
>>
>>
>> What would this actually mean?
>> we will have two copies basic tests suites one running with elytron
>> another with legacy security subsystem?
>>
>> Do I read that right? Please say I am not.
>
> That is correct - we have two security implementations they both need
> testing.
>
> One needs testing for backwards compatibility and regressions, the other
> for equivalent behaviour and then new features and bugs.
>
> Needing to test both was discussed previously so this is more about how
> to separate both and also give the Elytron testing a good foundation to
> start from.
>
>>
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
>
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
7:13 a.m.
I agree with Josef - it looks like more or less most of integration
tests can be affected by security features.
On Fri, Dec 2, 2016 at 12:58 PM, Josef Cacek <jcacek(a)redhat.com> wrote:
We prefer duplication of all tests in the testsuite to keep test
coverage.
To be sure we don't have regressions with introducing Elytron and at the same time we
are able to cover current features with Elytron, we must run all tests with both security
subsystems.
Even if a test is not security related on the first glance, the tested component may use
a security feature internally. By switching to the new security subsystem the feature may
stop work.
Let's keep the current (legacy security based) tests alive until the legacy security
subsystem is fully removed.
Then we'll simply remove the *-legacy-security testsuite modules.
-- Josef
----- Original Message -----
> From: "Tomaž Cerar" <tomaz.cerar(a)gmail.com>
> To: "Darran Lofthouse" <darran.lofthouse(a)jboss.com>
> Cc: wildfly-dev(a)lists.jboss.org
> Sent: Friday, December 2, 2016 12:37:22 PM
> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite
>
> That is probably fine, but! it should be done differently.
>
> instead of duplicating whole testsuite (and adding extra hour to execution
> and extra headaches with intermittent problems and duplication of
> maintenance)
> I would suggest that all security related tests get extracted to new
> "security" testsuite module and than only that part is duplicated.
>
> This way we will have all security related stuff in one place.
>
>
>
> On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse <
> darran.lofthouse(a)jboss.com > wrote:
>
>
> Probably should add - any duplication should only be for security tests
> - not everything else in there!
>
> On 02/12/16 11:08, Darran Lofthouse wrote:
> > On 02/12/16 11:03, Tomaž Cerar wrote:
> >>
> >> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek < jcacek(a)redhat.com
> >> <mailto: jcacek(a)redhat.com >> wrote:
> >>
> >> The modules would just live side by side - basic would use Elytron
> >> configuration, basic-legacy-security would use configuration similar
> >> to (or same as) the current server configuration.
> >>
> >>
> >>
> >> What would this actually mean?
> >> we will have two copies basic tests suites one running with elytron
> >> another with legacy security subsystem?
> >>
> >> Do I read that right? Please say I am not.
> >
> > That is correct - we have two security implementations they both need
> > testing.
> >
> > One needs testing for backwards compatibility and regressions, the other
> > for equivalent behaviour and then new features and bugs.
> >
> > Needing to test both was discussed previously so this is more about how
> > to separate both and also give the Elytron testing a good foundation to
> > start from.
> >
> >>
> >>
> >> _______________________________________________
> >> wildfly-dev mailing list
> >> wildfly-dev(a)lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/wildfly-dev
> >>
> >
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
9:47 a.m.
So if I understood all this correctly,
this mail here was more of a heads up what is happening
than a discussion on how could or should be best done.
I just hope you (the team behind decisions) considered all scenarios how
will that impact everyone involved.
--
tomaz
On Fri, Dec 2, 2016 at 12:58 PM, Josef Cacek <jcacek(a)redhat.com> wrote:
We prefer duplication of all tests in the testsuite to keep test
coverage.
To be sure we don't have regressions with introducing Elytron and at the
same time we are able to cover current features with Elytron, we must run
all tests with both security subsystems.
Even if a test is not security related on the first glance, the tested
component may use a security feature internally. By switching to the new
security subsystem the feature may stop work.
Let's keep the current (legacy security based) tests alive until the
legacy security subsystem is fully removed.
Then we'll simply remove the *-legacy-security testsuite modules.
-- Josef
----- Original Message -----
> From: "Tomaž Cerar" <tomaz.cerar(a)gmail.com>
> To: "Darran Lofthouse" <darran.lofthouse(a)jboss.com>
> Cc: wildfly-dev(a)lists.jboss.org
> Sent: Friday, December 2, 2016 12:37:22 PM
> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite
>
> That is probably fine, but! it should be done differently.
>
> instead of duplicating whole testsuite (and adding extra hour to
execution
> and extra headaches with intermittent problems and duplication of
> maintenance)
> I would suggest that all security related tests get extracted to new
> "security" testsuite module and than only that part is duplicated.
>
> This way we will have all security related stuff in one place.
>
>
>
> On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse <
> darran.lofthouse(a)jboss.com > wrote:
>
>
> Probably should add - any duplication should only be for security tests
> - not everything else in there!
>
> On 02/12/16 11:08, Darran Lofthouse wrote:
> > On 02/12/16 11:03, Tomaž Cerar wrote:
> >>
> >> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek < jcacek(a)redhat.com
> >> <mailto: jcacek(a)redhat.com >> wrote:
> >>
> >> The modules would just live side by side - basic would use Elytron
> >> configuration, basic-legacy-security would use configuration similar
> >> to (or same as) the current server configuration.
> >>
> >>
> >>
> >> What would this actually mean?
> >> we will have two copies basic tests suites one running with elytron
> >> another with legacy security subsystem?
> >>
> >> Do I read that right? Please say I am not.
> >
> > That is correct - we have two security implementations they both need
> > testing.
> >
> > One needs testing for backwards compatibility and regressions, the
other
> > for equivalent behaviour and then new features and bugs.
> >
> > Needing to test both was discussed previously so this is more about how
> > to separate both and also give the Elytron testing a good foundation to
> > start from.
> >
> >>
> >>
> >> _______________________________________________
> >> wildfly-dev mailing list
> >> wildfly-dev(a)lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/wildfly-dev
> >>
> >
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
9:59 a.m.
No the part we can't avoid is that we have two security implementations
in the server and we have to have tests for both. The e-mail from Josef
is a suggestion as to how this will be achieved.
Other suggestions are welcome ;-)
But we have two security implementations to test.
One of these is deprecated and will hopefully be removed at some point
so ideally we have something that allows testing to legacy to be removed
without too much pain.
Writing sufficient test coverage is not a small task so anything that
gives us more than starting from scratch is a benefit.
But maybe it would be worth summarising these additional scenarios that
will be affected by changes.
Regards,
Darran Lofthouse.
On 02/12/16 14:47, Tomaž Cerar wrote:
So if I understood all this correctly,
this mail here was more of a heads up what is happening
than a discussion on how could or should be best done.
I just hope you (the team behind decisions) considered all scenarios how
will that impact everyone involved.
--
tomaz
On Fri, Dec 2, 2016 at 12:58 PM, Josef Cacek <jcacek(a)redhat.com
<mailto:jcacek@redhat.com>> wrote:
We prefer duplication of all tests in the testsuite to keep test
coverage.
To be sure we don't have regressions with introducing Elytron and at
the same time we are able to cover current features with Elytron, we
must run all tests with both security subsystems.
Even if a test is not security related on the first glance, the
tested component may use a security feature internally. By switching
to the new security subsystem the feature may stop work.
Let's keep the current (legacy security based) tests alive until the
legacy security subsystem is fully removed.
Then we'll simply remove the *-legacy-security testsuite modules.
-- Josef
----- Original Message -----
> From: "Tomaž Cerar" <tomaz.cerar(a)gmail.com
<mailto:tomaz.cerar@gmail.com>>
> To: "Darran Lofthouse" <darran.lofthouse(a)jboss.com
<mailto:darran.lofthouse@jboss.com>>
> Cc: wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
> Sent: Friday, December 2, 2016 12:37:22 PM
> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly
testsuite
>
> That is probably fine, but! it should be done differently.
>
> instead of duplicating whole testsuite (and adding extra hour to
execution
> and extra headaches with intermittent problems and duplication of
> maintenance)
> I would suggest that all security related tests get extracted to new
> "security" testsuite module and than only that part is duplicated.
>
> This way we will have all security related stuff in one place.
>
>
>
> On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse <
> darran.lofthouse(a)jboss.com <mailto:darran.lofthouse@jboss.com> >
wrote:
>
>
> Probably should add - any duplication should only be for security
tests
> - not everything else in there!
>
> On 02/12/16 11:08, Darran Lofthouse wrote:
> > On 02/12/16 11:03, Tomaž Cerar wrote:
> >>
> >> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek < jcacek(a)redhat.com
<mailto:jcacek@redhat.com>
> >> <mailto: jcacek(a)redhat.com <mailto:jcacek@redhat.com> >>
wrote:
> >>
> >> The modules would just live side by side - basic would use Elytron
> >> configuration, basic-legacy-security would use configuration
similar
> >> to (or same as) the current server configuration.
> >>
> >>
> >>
> >> What would this actually mean?
> >> we will have two copies basic tests suites one running with elytron
> >> another with legacy security subsystem?
> >>
> >> Do I read that right? Please say I am not.
> >
> > That is correct - we have two security implementations they both
need
> > testing.
> >
> > One needs testing for backwards compatibility and regressions,
the other
> > for equivalent behaviour and then new features and bugs.
> >
> > Needing to test both was discussed previously so this is more
about how
> > to separate both and also give the Elytron testing a good
foundation to
> > start from.
> >
> >>
> >>
> >> _______________________________________________
> >> wildfly-dev mailing list
> >> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
> >> https://lists.jboss.org/mailman/listinfo/wildfly-dev
<https://lists.jboss.org/mailman/listinfo/wildfly-dev>
> >>
> >
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
<https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
<https://lists.jboss.org/mailman/listinfo/wildfly-dev>
10:20 a.m.
Duplicating the whole testsuite is basically a minimal effort way of getting coverage by
avoiding the task of test coverage analysis.
Only justification for that is expediency. That may be sufficient justification; I don’t
know.
But assuming for the sake of argument it’s sufficient, doubling the amount of time needed
to run the testsuite is a massive productivity loss and that means cutting that time down
to the necessary minimum and nothing more would need to be a critical extreme high
priority task.
So, with this proposal, who would be the people assigned to this critical extreme high
priority task? I’d like to see names, lots and lots of names. Darran Lofthouse not among
them. ;)
On Dec 2, 2016, at 8:59 AM, Darran Lofthouse
<darran.lofthouse(a)jboss.com> wrote:
No the part we can't avoid is that we have two security implementations
in the server and we have to have tests for both. The e-mail from Josef
is a suggestion as to how this will be achieved.
Other suggestions are welcome ;-)
But we have two security implementations to test.
One of these is deprecated and will hopefully be removed at some point
so ideally we have something that allows testing to legacy to be removed
without too much pain.
Writing sufficient test coverage is not a small task so anything that
gives us more than starting from scratch is a benefit.
But maybe it would be worth summarising these additional scenarios that
will be affected by changes.
Regards,
Darran Lofthouse.
On 02/12/16 14:47, Tomaž Cerar wrote:
> So if I understood all this correctly,
> this mail here was more of a heads up what is happening
> than a discussion on how could or should be best done.
>
> I just hope you (the team behind decisions) considered all scenarios how
> will that impact everyone involved.
>
> --
> tomaz
>
> On Fri, Dec 2, 2016 at 12:58 PM, Josef Cacek <jcacek(a)redhat.com
> <mailto:jcacek@redhat.com>> wrote:
>
> We prefer duplication of all tests in the testsuite to keep test
> coverage.
>
> To be sure we don't have regressions with introducing Elytron and at
> the same time we are able to cover current features with Elytron, we
> must run all tests with both security subsystems.
> Even if a test is not security related on the first glance, the
> tested component may use a security feature internally. By switching
> to the new security subsystem the feature may stop work.
>
> Let's keep the current (legacy security based) tests alive until the
> legacy security subsystem is fully removed.
> Then we'll simply remove the *-legacy-security testsuite modules.
>
> -- Josef
>
> ----- Original Message -----
>> From: "Tomaž Cerar" <tomaz.cerar(a)gmail.com
> <mailto:tomaz.cerar@gmail.com>>
>> To: "Darran Lofthouse" <darran.lofthouse(a)jboss.com
> <mailto:darran.lofthouse@jboss.com>>
>> Cc: wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
>> Sent: Friday, December 2, 2016 12:37:22 PM
>> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly
> testsuite
>>
>> That is probably fine, but! it should be done differently.
>>
>> instead of duplicating whole testsuite (and adding extra hour to
> execution
>> and extra headaches with intermittent problems and duplication of
>> maintenance)
>> I would suggest that all security related tests get extracted to new
>> "security" testsuite module and than only that part is duplicated.
>>
>> This way we will have all security related stuff in one place.
>>
>>
>>
>> On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse <
>> darran.lofthouse(a)jboss.com <mailto:darran.lofthouse@jboss.com> >
> wrote:
>>
>>
>> Probably should add - any duplication should only be for security
> tests
>> - not everything else in there!
>>
>> On 02/12/16 11:08, Darran Lofthouse wrote:
>>> On 02/12/16 11:03, Tomaž Cerar wrote:
>>>>
>>>> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek < jcacek(a)redhat.com
> <mailto:jcacek@redhat.com>
>>>> <mailto: jcacek(a)redhat.com <mailto:jcacek@redhat.com> >>
wrote:
>>>>
>>>> The modules would just live side by side - basic would use Elytron
>>>> configuration, basic-legacy-security would use configuration
> similar
>>>> to (or same as) the current server configuration.
>>>>
>>>>
>>>>
>>>> What would this actually mean?
>>>> we will have two copies basic tests suites one running with elytron
>>>> another with legacy security subsystem?
>>>>
>>>> Do I read that right? Please say I am not.
>>>
>>> That is correct - we have two security implementations they both
> need
>>> testing.
>>>
>>> One needs testing for backwards compatibility and regressions,
> the other
>>> for equivalent behaviour and then new features and bugs.
>>>
>>> Needing to test both was discussed previously so this is more
> about how
>>> to separate both and also give the Elytron testing a good
> foundation to
>>> start from.
>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> wildfly-dev mailing list
>>>> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
> <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>>>>
>>>
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
> <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>>
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
> <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>
>
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
--
Brian Stansberry
Manager, Senior Principal Software Engineer
JBoss by Red Hat
10:28 a.m.
+1 doubling the testsuite size/time is not practical. Also, there is the issue that if all
tests are duplicated, when people come to enhance an existing test they will need to do so
in two places.
Could we not do what was suggested earlier and pull out tests which obviously target
security into two legacy/elytron modules, and leave the rest where they are? Then if
something later turns out to have 'hidden' implications, that test gets pulled out
into the two modules.
Regarding the deprecated legacy stuff, I think we will have to keep that around for many
years still due to it being used by the product (and in any case in that branch, although
removing it from WF might be possible sooner).
On 2 Dec 2016, at 15:20, Brian Stansberry
<brian.stansberry(a)redhat.com> wrote:
Duplicating the whole testsuite is basically a minimal effort way of getting coverage by
avoiding the task of test coverage analysis.
Only justification for that is expediency. That may be sufficient justification; I don’t
know.
But assuming for the sake of argument it’s sufficient, doubling the amount of time needed
to run the testsuite is a massive productivity loss and that means cutting that time down
to the necessary minimum and nothing more would need to be a critical extreme high
priority task.
So, with this proposal, who would be the people assigned to this critical extreme high
priority task? I’d like to see names, lots and lots of names. Darran Lofthouse not among
them. ;)
> On Dec 2, 2016, at 8:59 AM, Darran Lofthouse <darran.lofthouse(a)jboss.com>
wrote:
>
> No the part we can't avoid is that we have two security implementations
> in the server and we have to have tests for both. The e-mail from Josef
> is a suggestion as to how this will be achieved.
>
> Other suggestions are welcome ;-)
>
> But we have two security implementations to test.
>
> One of these is deprecated and will hopefully be removed at some point
> so ideally we have something that allows testing to legacy to be removed
> without too much pain.
>
> Writing sufficient test coverage is not a small task so anything that
> gives us more than starting from scratch is a benefit.
>
> But maybe it would be worth summarising these additional scenarios that
> will be affected by changes.
>
> Regards,
> Darran Lofthouse.
>
> On 02/12/16 14:47, Tomaž Cerar wrote:
>> So if I understood all this correctly,
>> this mail here was more of a heads up what is happening
>> than a discussion on how could or should be best done.
>>
>> I just hope you (the team behind decisions) considered all scenarios how
>> will that impact everyone involved.
>>
>> --
>> tomaz
>>
>> On Fri, Dec 2, 2016 at 12:58 PM, Josef Cacek <jcacek(a)redhat.com
>> <mailto:jcacek@redhat.com>> wrote:
>>
>> We prefer duplication of all tests in the testsuite to keep test
>> coverage.
>>
>> To be sure we don't have regressions with introducing Elytron and at
>> the same time we are able to cover current features with Elytron, we
>> must run all tests with both security subsystems.
>> Even if a test is not security related on the first glance, the
>> tested component may use a security feature internally. By switching
>> to the new security subsystem the feature may stop work.
>>
>> Let's keep the current (legacy security based) tests alive until the
>> legacy security subsystem is fully removed.
>> Then we'll simply remove the *-legacy-security testsuite modules.
>>
>> -- Josef
>>
>> ----- Original Message -----
>>> From: "Tomaž Cerar" <tomaz.cerar(a)gmail.com
>> <mailto:tomaz.cerar@gmail.com>>
>>> To: "Darran Lofthouse" <darran.lofthouse(a)jboss.com
>> <mailto:darran.lofthouse@jboss.com>>
>>> Cc: wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
>>> Sent: Friday, December 2, 2016 12:37:22 PM
>>> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly
>> testsuite
>>>
>>> That is probably fine, but! it should be done differently.
>>>
>>> instead of duplicating whole testsuite (and adding extra hour to
>> execution
>>> and extra headaches with intermittent problems and duplication of
>>> maintenance)
>>> I would suggest that all security related tests get extracted to new
>>> "security" testsuite module and than only that part is duplicated.
>>>
>>> This way we will have all security related stuff in one place.
>>>
>>>
>>>
>>> On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse <
>>> darran.lofthouse(a)jboss.com <mailto:darran.lofthouse@jboss.com> >
>> wrote:
>>>
>>>
>>> Probably should add - any duplication should only be for security
>> tests
>>> - not everything else in there!
>>>
>>> On 02/12/16 11:08, Darran Lofthouse wrote:
>>>> On 02/12/16 11:03, Tomaž Cerar wrote:
>>>>>
>>>>> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek < jcacek(a)redhat.com
>> <mailto:jcacek@redhat.com>
>>>>> <mailto: jcacek(a)redhat.com <mailto:jcacek@redhat.com>
>> wrote:
>>>>>
>>>>> The modules would just live side by side - basic would use Elytron
>>>>> configuration, basic-legacy-security would use configuration
>> similar
>>>>> to (or same as) the current server configuration.
>>>>>
>>>>>
>>>>>
>>>>> What would this actually mean?
>>>>> we will have two copies basic tests suites one running with elytron
>>>>> another with legacy security subsystem?
>>>>>
>>>>> Do I read that right? Please say I am not.
>>>>
>>>> That is correct - we have two security implementations they both
>> need
>>>> testing.
>>>>
>>>> One needs testing for backwards compatibility and regressions,
>> the other
>>>> for equivalent behaviour and then new features and bugs.
>>>>
>>>> Needing to test both was discussed previously so this is more
>> about how
>>>> to separate both and also give the Elytron testing a good
>> foundation to
>>>> start from.
>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> wildfly-dev mailing list
>>>>> wildfly-dev(a)lists.jboss.org
<mailto:wildfly-dev@lists.jboss.org>
>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>> <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>> <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>>>
>>>
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>> <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>>
>>
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
--
Brian Stansberry
Manager, Senior Principal Software Engineer
JBoss by Red Hat
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
10:03 a.m.
Hi,
We still see the copying whole modules as the best way. Let's recap our view.
Tested components may use a security feature internally and without test coverage we would
not realize that the feature stopped working. Running all tests will help us to discover
regressions before they occur in application server.
Pros:
- we don't lose testcoverage, we can simply detect regressions. It also provides
possible early failure detection for dev.
- it would be useful for migration, it shows migration path - user can take a look on
tests with legacy solution and compare their configuration with tests which use Elytron
configuration
- by each testsuite run we see how many tests are skipped (i.e. TODO list for Elytron
integration)
- it should be simple to reconfigure the project (copy module; update metadata in pom.xml;
ignore failing tests)
- it's simple to remove legacy testsuite modules once the legacy security subsystem is
removed from WildFly
Cons and their solutions:
- double fixing of tests - we don't expect many such cases, the legacy should be
handled as frozen (rewriting/adding only on need-to-have basis). Only custormer ticket
related tests could be backported, it means only unit of tests per months.
- testsuite execution time - we don't need to run the legacy modules with each PR, it
can be resolved by using profiles
As a smoke test we tried to run testsuite/integration-tests/basic module with
standalone-elytron.xml and standalone-full-elytron.xml (merged standalone-full.xml and
standalone-elytron.xml). There is around 25% test failures/errors. Also some testcases,
which seem that are not directly related to security, failed.
Thanks,
-- Josef
----- Original Message -----
From: "Kabir Khan" <kabir.khan(a)jboss.com>
To: "Brian Stansberry" <brian.stansberry(a)redhat.com>
Cc: wildfly-dev(a)lists.jboss.org
Sent: Friday, December 2, 2016 4:28:07 PM
Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite
+1 doubling the testsuite size/time is not practical. Also, there is the
issue that if all tests are duplicated, when people come to enhance an
existing test they will need to do so in two places.
Could we not do what was suggested earlier and pull out tests which obviously
target security into two legacy/elytron modules, and leave the rest where
they are? Then if something later turns out to have 'hidden' implications,
that test gets pulled out into the two modules.
Regarding the deprecated legacy stuff, I think we will have to keep that
around for many years still due to it being used by the product (and in any
case in that branch, although removing it from WF might be possible sooner).
11:28 a.m.
Regarding that 25% failure - I would expect a lot of that to be picked
up anyway as we merge the *-elytron.xml configurations into the default
configurations and then drop the *-elytron configurations.
How about my reply to Stuart's e-mail and a 3-way split,
legacy-security, security, and basic where basic can have optional
profiles added to test different extension availability?
We could then do as Kabir suggests and pull out additional tests later
if we do find a reason to justify duplicating the test and in the
meantime have the profiles available to run the tests.
Regards,
Darran Lofthouse.
On 05/12/16 15:03, Josef Cacek wrote:
Hi,
We still see the copying whole modules as the best way. Let's recap our view.
Tested components may use a security feature internally and without test coverage we
would not realize that the feature stopped working. Running all tests will help us to
discover regressions before they occur in application server.
Pros:
- we don't lose testcoverage, we can simply detect regressions. It also provides
possible early failure detection for dev.
- it would be useful for migration, it shows migration path - user can take a look on
tests with legacy solution and compare their configuration with tests which use Elytron
configuration
- by each testsuite run we see how many tests are skipped (i.e. TODO list for Elytron
integration)
- it should be simple to reconfigure the project (copy module; update metadata in
pom.xml; ignore failing tests)
- it's simple to remove legacy testsuite modules once the legacy security subsystem
is removed from WildFly
Cons and their solutions:
- double fixing of tests - we don't expect many such cases, the legacy should be
handled as frozen (rewriting/adding only on need-to-have basis). Only custormer ticket
related tests could be backported, it means only unit of tests per months.
- testsuite execution time - we don't need to run the legacy modules with each PR, it
can be resolved by using profiles
As a smoke test we tried to run testsuite/integration-tests/basic module with
standalone-elytron.xml and standalone-full-elytron.xml (merged standalone-full.xml and
standalone-elytron.xml). There is around 25% test failures/errors. Also some testcases,
which seem that are not directly related to security, failed.
Thanks,
-- Josef
----- Original Message -----
> From: "Kabir Khan" <kabir.khan(a)jboss.com>
> To: "Brian Stansberry" <brian.stansberry(a)redhat.com>
> Cc: wildfly-dev(a)lists.jboss.org
> Sent: Friday, December 2, 2016 4:28:07 PM
> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite
>
> +1 doubling the testsuite size/time is not practical. Also, there is the
> issue that if all tests are duplicated, when people come to enhance an
> existing test they will need to do so in two places.
>
> Could we not do what was suggested earlier and pull out tests which obviously
> target security into two legacy/elytron modules, and leave the rest where
> they are? Then if something later turns out to have 'hidden' implications,
> that test gets pulled out into the two modules.
>
> Regarding the deprecated legacy stuff, I think we will have to keep that
> around for many years still due to it being used by the product (and in any
> case in that branch, although removing it from WF might be possible sooner).
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
2:40 a.m.
The 3-way split is a good idea, but it would only work if
- we are sure the non-security tests in shared modules stay really independent - i.e. no
new Ignores, Assumes, configuration based conditions;
- all shared modules have profiles for running with legacy security.
It's not clear how many new modules would be necessary with the 3 way split. Current
TS modules don't share server configuration and they (may) contain several surefire
executions with different server profiles.
-- Josef
----- Original Message -----
From: "Darran Lofthouse"
<darran.lofthouse(a)jboss.com>
To: wildfly-dev(a)lists.jboss.org
Sent: Monday, December 5, 2016 5:28:02 PM
Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite
Regarding that 25% failure - I would expect a lot of that to be picked
up anyway as we merge the *-elytron.xml configurations into the default
configurations and then drop the *-elytron configurations.
How about my reply to Stuart's e-mail and a 3-way split,
legacy-security, security, and basic where basic can have optional
profiles added to test different extension availability?
We could then do as Kabir suggests and pull out additional tests later
if we do find a reason to justify duplicating the test and in the
meantime have the profiles available to run the tests.
Regards,
Darran Lofthouse.
On 05/12/16 15:03, Josef Cacek wrote:
> Hi,
>
> We still see the copying whole modules as the best way. Let's recap our
> view.
>
> Tested components may use a security feature internally and without test
> coverage we would not realize that the feature stopped working. Running
> all tests will help us to discover regressions before they occur in
> application server.
>
> Pros:
> - we don't lose testcoverage, we can simply detect regressions. It also
> provides possible early failure detection for dev.
> - it would be useful for migration, it shows migration path - user can take
> a look on tests with legacy solution and compare their configuration with
> tests which use Elytron configuration
> - by each testsuite run we see how many tests are skipped (i.e. TODO list
> for Elytron integration)
> - it should be simple to reconfigure the project (copy module; update
> metadata in pom.xml; ignore failing tests)
> - it's simple to remove legacy testsuite modules once the legacy security
> subsystem is removed from WildFly
>
> Cons and their solutions:
> - double fixing of tests - we don't expect many such cases, the legacy
> should be handled as frozen (rewriting/adding only on need-to-have basis).
> Only custormer ticket related tests could be backported, it means only
> unit of tests per months.
> - testsuite execution time - we don't need to run the legacy modules with
> each PR, it can be resolved by using profiles
>
> As a smoke test we tried to run testsuite/integration-tests/basic module
> with standalone-elytron.xml and standalone-full-elytron.xml (merged
> standalone-full.xml and standalone-elytron.xml). There is around 25% test
> failures/errors. Also some testcases, which seem that are not directly
> related to security, failed.
>
> Thanks,
>
> -- Josef
>
>
> ----- Original Message -----
>> From: "Kabir Khan" <kabir.khan(a)jboss.com>
>> To: "Brian Stansberry" <brian.stansberry(a)redhat.com>
>> Cc: wildfly-dev(a)lists.jboss.org
>> Sent: Friday, December 2, 2016 4:28:07 PM
>> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite
>>
>> +1 doubling the testsuite size/time is not practical. Also, there is the
>> issue that if all tests are duplicated, when people come to enhance an
>> existing test they will need to do so in two places.
>>
>> Could we not do what was suggested earlier and pull out tests which
>> obviously
>> target security into two legacy/elytron modules, and leave the rest where
>> they are? Then if something later turns out to have 'hidden'
implications,
>> that test gets pulled out into the two modules.
>>
>> Regarding the deprecated legacy stuff, I think we will have to keep that
>> around for many years still due to it being used by the product (and in
>> any
>> case in that branch, although removing it from WF might be possible
>> sooner).
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
7:08 a.m.
As I think about this I am heading back to just a two way split.
If we go for the two way split, lets say the following about the default
WildFly configuration: -
- It will be entirely backed by the Elytron subsystem.
- Legacy security-realms will be removed.
- The legacy security subsystem will be removed.
- The legacy security extension will be removed.
Existing users forward porting their configuration are unaffected as
this only affects default new configuration.
We can offer documentation on how to re-enable those items where really
needed - anyone doing so hopefully submitting a Jira letting us know why
they are doing so.
This will mean the 900 "non-security" tests will certainly be verified
that they run without the legacy security extension being installed.
We then have a new legacy-security module where we add at the very least
the legacy security extension back in which we copy the 50 or so
security tests over.
The security specific tests left behind are all cleaned up and ported to
pure Elytron based testing.
Anything that breaks as we switch the configuration over in the
remaining 900 tests we will review on a case by case basis, also we will
keep an eye on other subsystems we have touched as they may want a
legacy test adding e.g. messaging is one we have been working on this week.
Regards,
Darran Lofthouse.
On 06/12/16 07:40, Josef Cacek wrote:
The 3-way split is a good idea, but it would only work if
- we are sure the non-security tests in shared modules stay really independent - i.e. no
new Ignores, Assumes, configuration based conditions;
- all shared modules have profiles for running with legacy security.
It's not clear how many new modules would be necessary with the 3 way split. Current
TS modules don't share server configuration and they (may) contain several surefire
executions with different server profiles.
-- Josef
----- Original Message -----
> From: "Darran Lofthouse" <darran.lofthouse(a)jboss.com>
> To: wildfly-dev(a)lists.jboss.org
> Sent: Monday, December 5, 2016 5:28:02 PM
> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite
>
> Regarding that 25% failure - I would expect a lot of that to be picked
> up anyway as we merge the *-elytron.xml configurations into the default
> configurations and then drop the *-elytron configurations.
>
> How about my reply to Stuart's e-mail and a 3-way split,
> legacy-security, security, and basic where basic can have optional
> profiles added to test different extension availability?
>
> We could then do as Kabir suggests and pull out additional tests later
> if we do find a reason to justify duplicating the test and in the
> meantime have the profiles available to run the tests.
>
> Regards,
> Darran Lofthouse.
>
> On 05/12/16 15:03, Josef Cacek wrote:
>> Hi,
>>
>> We still see the copying whole modules as the best way. Let's recap our
>> view.
>>
>> Tested components may use a security feature internally and without test
>> coverage we would not realize that the feature stopped working. Running
>> all tests will help us to discover regressions before they occur in
>> application server.
>>
>> Pros:
>> - we don't lose testcoverage, we can simply detect regressions. It also
>> provides possible early failure detection for dev.
>> - it would be useful for migration, it shows migration path - user can take
>> a look on tests with legacy solution and compare their configuration with
>> tests which use Elytron configuration
>> - by each testsuite run we see how many tests are skipped (i.e. TODO list
>> for Elytron integration)
>> - it should be simple to reconfigure the project (copy module; update
>> metadata in pom.xml; ignore failing tests)
>> - it's simple to remove legacy testsuite modules once the legacy security
>> subsystem is removed from WildFly
>>
>> Cons and their solutions:
>> - double fixing of tests - we don't expect many such cases, the legacy
>> should be handled as frozen (rewriting/adding only on need-to-have basis).
>> Only custormer ticket related tests could be backported, it means only
>> unit of tests per months.
>> - testsuite execution time - we don't need to run the legacy modules with
>> each PR, it can be resolved by using profiles
>>
>> As a smoke test we tried to run testsuite/integration-tests/basic module
>> with standalone-elytron.xml and standalone-full-elytron.xml (merged
>> standalone-full.xml and standalone-elytron.xml). There is around 25% test
>> failures/errors. Also some testcases, which seem that are not directly
>> related to security, failed.
>>
>> Thanks,
>>
>> -- Josef
>>
>>
>> ----- Original Message -----
>>> From: "Kabir Khan" <kabir.khan(a)jboss.com>
>>> To: "Brian Stansberry" <brian.stansberry(a)redhat.com>
>>> Cc: wildfly-dev(a)lists.jboss.org
>>> Sent: Friday, December 2, 2016 4:28:07 PM
>>> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite
>>>
>>> +1 doubling the testsuite size/time is not practical. Also, there is the
>>> issue that if all tests are duplicated, when people come to enhance an
>>> existing test they will need to do so in two places.
>>>
>>> Could we not do what was suggested earlier and pull out tests which
>>> obviously
>>> target security into two legacy/elytron modules, and leave the rest where
>>> they are? Then if something later turns out to have 'hidden'
implications,
>>> that test gets pulled out into the two modules.
>>>
>>> Regarding the deprecated legacy stuff, I think we will have to keep that
>>> around for many years still due to it being used by the product (and in
>>> any
>>> case in that branch, although removing it from WF might be possible
>>> sooner).
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
7:59 a.m.
Ok, we will go for this approach.
We will create a new testsuite module 'legacy-security', we will copy
the 50 or so tests that are easily identifiable as security tests into
this new module - as these are testing legacy security the only real
changes they should require will be to re-enable legacy security.
Remaining tests will be executed against configuration that favours
Elytron over legacy. Security tests in this area will be cleaned up and
then we can start adding new security tests.
We will monitor other tests and if security related regressions are
identified we will review on a case by case basis and if appropriate
fork into the legacy-security module.
Regards,
Darran Lofthouse.
On 06/12/16 12:08, Darran Lofthouse wrote:
As I think about this I am heading back to just a two way split.
If we go for the two way split, lets say the following about the default
WildFly configuration: -
- It will be entirely backed by the Elytron subsystem.
- Legacy security-realms will be removed.
- The legacy security subsystem will be removed.
- The legacy security extension will be removed.
Existing users forward porting their configuration are unaffected as
this only affects default new configuration.
We can offer documentation on how to re-enable those items where really
needed - anyone doing so hopefully submitting a Jira letting us know why
they are doing so.
This will mean the 900 "non-security" tests will certainly be verified
that they run without the legacy security extension being installed.
We then have a new legacy-security module where we add at the very least
the legacy security extension back in which we copy the 50 or so
security tests over.
The security specific tests left behind are all cleaned up and ported to
pure Elytron based testing.
Anything that breaks as we switch the configuration over in the
remaining 900 tests we will review on a case by case basis, also we will
keep an eye on other subsystems we have touched as they may want a
legacy test adding e.g. messaging is one we have been working on this week.
Regards,
Darran Lofthouse.
On 06/12/16 07:40, Josef Cacek wrote:
> The 3-way split is a good idea, but it would only work if
> - we are sure the non-security tests in shared modules stay really
> independent - i.e. no new Ignores, Assumes, configuration based
> conditions;
> - all shared modules have profiles for running with legacy security.
>
> It's not clear how many new modules would be necessary with the 3 way
> split. Current TS modules don't share server configuration and they
> (may) contain several surefire executions with different server profiles.
>
> -- Josef
>
> ----- Original Message -----
>> From: "Darran Lofthouse" <darran.lofthouse(a)jboss.com>
>> To: wildfly-dev(a)lists.jboss.org
>> Sent: Monday, December 5, 2016 5:28:02 PM
>> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly
>> testsuite
>>
>> Regarding that 25% failure - I would expect a lot of that to be picked
>> up anyway as we merge the *-elytron.xml configurations into the default
>> configurations and then drop the *-elytron configurations.
>>
>> How about my reply to Stuart's e-mail and a 3-way split,
>> legacy-security, security, and basic where basic can have optional
>> profiles added to test different extension availability?
>>
>> We could then do as Kabir suggests and pull out additional tests later
>> if we do find a reason to justify duplicating the test and in the
>> meantime have the profiles available to run the tests.
>>
>> Regards,
>> Darran Lofthouse.
>>
>> On 05/12/16 15:03, Josef Cacek wrote:
>>> Hi,
>>>
>>> We still see the copying whole modules as the best way. Let's recap our
>>> view.
>>>
>>> Tested components may use a security feature internally and without
>>> test
>>> coverage we would not realize that the feature stopped working. Running
>>> all tests will help us to discover regressions before they occur in
>>> application server.
>>>
>>> Pros:
>>> - we don't lose testcoverage, we can simply detect regressions. It also
>>> provides possible early failure detection for dev.
>>> - it would be useful for migration, it shows migration path - user
>>> can take
>>> a look on tests with legacy solution and compare their configuration
>>> with
>>> tests which use Elytron configuration
>>> - by each testsuite run we see how many tests are skipped (i.e. TODO
>>> list
>>> for Elytron integration)
>>> - it should be simple to reconfigure the project (copy module; update
>>> metadata in pom.xml; ignore failing tests)
>>> - it's simple to remove legacy testsuite modules once the legacy
>>> security
>>> subsystem is removed from WildFly
>>>
>>> Cons and their solutions:
>>> - double fixing of tests - we don't expect many such cases, the legacy
>>> should be handled as frozen (rewriting/adding only on need-to-have
>>> basis).
>>> Only custormer ticket related tests could be backported, it means only
>>> unit of tests per months.
>>> - testsuite execution time - we don't need to run the legacy modules
>>> with
>>> each PR, it can be resolved by using profiles
>>>
>>> As a smoke test we tried to run testsuite/integration-tests/basic
>>> module
>>> with standalone-elytron.xml and standalone-full-elytron.xml (merged
>>> standalone-full.xml and standalone-elytron.xml). There is around 25%
>>> test
>>> failures/errors. Also some testcases, which seem that are not directly
>>> related to security, failed.
>>>
>>> Thanks,
>>>
>>> -- Josef
>>>
>>>
>>> ----- Original Message -----
>>>> From: "Kabir Khan" <kabir.khan(a)jboss.com>
>>>> To: "Brian Stansberry" <brian.stansberry(a)redhat.com>
>>>> Cc: wildfly-dev(a)lists.jboss.org
>>>> Sent: Friday, December 2, 2016 4:28:07 PM
>>>> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly
>>>> testsuite
>>>>
>>>> +1 doubling the testsuite size/time is not practical. Also, there
>>>> is the
>>>> issue that if all tests are duplicated, when people come to enhance an
>>>> existing test they will need to do so in two places.
>>>>
>>>> Could we not do what was suggested earlier and pull out tests which
>>>> obviously
>>>> target security into two legacy/elytron modules, and leave the rest
>>>> where
>>>> they are? Then if something later turns out to have 'hidden'
>>>> implications,
>>>> that test gets pulled out into the two modules.
>>>>
>>>> Regarding the deprecated legacy stuff, I think we will have to keep
>>>> that
>>>> around for many years still due to it being used by the product
>>>> (and in
>>>> any
>>>> case in that branch, although removing it from WF might be possible
>>>> sooner).
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> wildfly-dev(a)lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
11:53 p.m.
On Tue, Dec 6, 2016 at 2:03 AM, Josef Cacek <jcacek(a)redhat.com> wrote:
Hi,
We still see the copying whole modules as the best way. Let's recap our
view.
Tested components may use a security feature internally and without test
coverage we would not realize that the feature stopped working. Running all
tests will help us to discover regressions before they occur in application
server.
They don't though. If the test is not set up to use security then security
won't be applied. The idea of adding an extra 900 test cases 'just in case'
is massive overkill, it adds a massive amount of technical debt and
decreases the overall quality of the test suite for very little gain.
Pros:
- we don't lose testcoverage, we can simply detect regressions. It also
provides possible early failure detection for dev.
- it would be useful for migration, it shows migration path - user can
take a look on tests with legacy solution and compare their configuration
with tests which use Elytron configuration
This is only applicable for tests that actually use Elytron. Just copying
the relevant tests has the same effect.
- by each testsuite run we see how many tests are skipped (i.e. TODO
list
for Elytron integration)
Once again, also accomplished by just copying Elytron tests.
- it should be simple to reconfigure the project (copy module;
update
metadata in pom.xml; ignore failing tests)
Once again, also accomplished by just copying Elytron tests. Even though it
is convenient at first IMHO the duplication of tests will greatly decrease
test suite maintainability, so overall I think this approach will use a lot
more man hours.
- it's simple to remove legacy testsuite modules once the legacy
security
subsystem is removed from WildFly
Cons and their solutions:
- double fixing of tests - we don't expect many such cases, the legacy
should be handled as frozen (rewriting/adding only on need-to-have basis).
Only custormer ticket related tests could be backported, it means only unit
of tests per months.
This is a major problem. We should not have to write a new test every time,
whenever possible existing tests should be enhanced (assuming similar tests
for similar functionality already exist). This has many advantages:
- The tests will execute faster, as deployment and set up is only performed
once
- It is very clear exactly what functionality is being tested, as all the
tests are in a single test case
- It reduces the number of classes in the test suite, which makes compile
times faster and increases maintainability
Freezing all these tests is not a practical idea IMHO.
- testsuite execution time - we don't need to run the legacy
modules with
each PR, it can be resolved by using profiles
Yes we do. If a test is not being run with every PR it might as well not be
there, as it means that is can be broken without anyone knowing.
As a smoke test we tried to run testsuite/integration-tests/basic module
with standalone-elytron.xml and standalone-full-elytron.xml (merged
standalone-full.xml and standalone-elytron.xml). There is around 25% test
failures/errors. Also some testcases, which seem that are not directly
related to security, failed.
This does not really mean much at the moment, it is most likely due to
configuration issues. Once the default config contains Elytron all these
tests will be running with the Elytron config anyway.
Stuart
Thanks,
-- Josef
----- Original Message -----
> From: "Kabir Khan" <kabir.khan(a)jboss.com>
> To: "Brian Stansberry" <brian.stansberry(a)redhat.com>
> Cc: wildfly-dev(a)lists.jboss.org
> Sent: Friday, December 2, 2016 4:28:07 PM
> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite
>
> +1 doubling the testsuite size/time is not practical. Also, there is the
> issue that if all tests are duplicated, when people come to enhance an
> existing test they will need to do so in two places.
>
> Could we not do what was suggested earlier and pull out tests which
obviously
> target security into two legacy/elytron modules, and leave the rest where
> they are? Then if something later turns out to have 'hidden'
implications,
> that test gets pulled out into the two modules.
>
> Regarding the deprecated legacy stuff, I think we will have to keep that
> around for many years still due to it being used by the product (and in
any
> case in that branch, although removing it from WF might be possible
sooner).
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
1:28 p.m.
Right we can’t just double the testsuite. Although I could understand someone doing this
as a temporary exercise. Not only is this bad from an overhead standpoint, but its also
not guaranteed to be effective, since a large portion of our tests either don’t use
security, or only the most basic functions. I think we need targeted testing for the new
functionality regardless.
On Dec 2, 2016, at 9:20 AM, Brian Stansberry
<brian.stansberry(a)redhat.com> wrote:
Duplicating the whole testsuite is basically a minimal effort way of getting coverage by
avoiding the task of test coverage analysis.
Only justification for that is expediency. That may be sufficient justification; I don’t
know.
But assuming for the sake of argument it’s sufficient, doubling the amount of time needed
to run the testsuite is a massive productivity loss and that means cutting that time down
to the necessary minimum and nothing more would need to be a critical extreme high
priority task.
So, with this proposal, who would be the people assigned to this critical extreme high
priority task? I’d like to see names, lots and lots of names. Darran Lofthouse not among
them. ;)
> On Dec 2, 2016, at 8:59 AM, Darran Lofthouse <darran.lofthouse(a)jboss.com>
wrote:
>
> No the part we can't avoid is that we have two security implementations
> in the server and we have to have tests for both. The e-mail from Josef
> is a suggestion as to how this will be achieved.
>
> Other suggestions are welcome ;-)
>
> But we have two security implementations to test.
>
> One of these is deprecated and will hopefully be removed at some point
> so ideally we have something that allows testing to legacy to be removed
> without too much pain.
>
> Writing sufficient test coverage is not a small task so anything that
> gives us more than starting from scratch is a benefit.
>
> But maybe it would be worth summarising these additional scenarios that
> will be affected by changes.
>
> Regards,
> Darran Lofthouse.
>
> On 02/12/16 14:47, Tomaž Cerar wrote:
>> So if I understood all this correctly,
>> this mail here was more of a heads up what is happening
>> than a discussion on how could or should be best done.
>>
>> I just hope you (the team behind decisions) considered all scenarios how
>> will that impact everyone involved.
>>
>> --
>> tomaz
>>
>> On Fri, Dec 2, 2016 at 12:58 PM, Josef Cacek <jcacek(a)redhat.com
>> <mailto:jcacek@redhat.com>> wrote:
>>
>> We prefer duplication of all tests in the testsuite to keep test
>> coverage.
>>
>> To be sure we don't have regressions with introducing Elytron and at
>> the same time we are able to cover current features with Elytron, we
>> must run all tests with both security subsystems.
>> Even if a test is not security related on the first glance, the
>> tested component may use a security feature internally. By switching
>> to the new security subsystem the feature may stop work.
>>
>> Let's keep the current (legacy security based) tests alive until the
>> legacy security subsystem is fully removed.
>> Then we'll simply remove the *-legacy-security testsuite modules.
>>
>> -- Josef
>>
>> ----- Original Message -----
>>> From: "Tomaž Cerar" <tomaz.cerar(a)gmail.com
>> <mailto:tomaz.cerar@gmail.com>>
>>> To: "Darran Lofthouse" <darran.lofthouse(a)jboss.com
>> <mailto:darran.lofthouse@jboss.com>>
>>> Cc: wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
>>> Sent: Friday, December 2, 2016 12:37:22 PM
>>> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly
>> testsuite
>>>
>>> That is probably fine, but! it should be done differently.
>>>
>>> instead of duplicating whole testsuite (and adding extra hour to
>> execution
>>> and extra headaches with intermittent problems and duplication of
>>> maintenance)
>>> I would suggest that all security related tests get extracted to new
>>> "security" testsuite module and than only that part is duplicated.
>>>
>>> This way we will have all security related stuff in one place.
>>>
>>>
>>>
>>> On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse <
>>> darran.lofthouse(a)jboss.com <mailto:darran.lofthouse@jboss.com> >
>> wrote:
>>>
>>>
>>> Probably should add - any duplication should only be for security
>> tests
>>> - not everything else in there!
>>>
>>> On 02/12/16 11:08, Darran Lofthouse wrote:
>>>> On 02/12/16 11:03, Tomaž Cerar wrote:
>>>>>
>>>>> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek < jcacek(a)redhat.com
>> <mailto:jcacek@redhat.com>
>>>>> <mailto: jcacek(a)redhat.com <mailto:jcacek@redhat.com>
>> wrote:
>>>>>
>>>>> The modules would just live side by side - basic would use Elytron
>>>>> configuration, basic-legacy-security would use configuration
>> similar
>>>>> to (or same as) the current server configuration.
>>>>>
>>>>>
>>>>>
>>>>> What would this actually mean?
>>>>> we will have two copies basic tests suites one running with elytron
>>>>> another with legacy security subsystem?
>>>>>
>>>>> Do I read that right? Please say I am not.
>>>>
>>>> That is correct - we have two security implementations they both
>> need
>>>> testing.
>>>>
>>>> One needs testing for backwards compatibility and regressions,
>> the other
>>>> for equivalent behaviour and then new features and bugs.
>>>>
>>>> Needing to test both was discussed previously so this is more
>> about how
>>>> to separate both and also give the Elytron testing a good
>> foundation to
>>>> start from.
>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> wildfly-dev mailing list
>>>>> wildfly-dev(a)lists.jboss.org
<mailto:wildfly-dev@lists.jboss.org>
>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>> <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>> <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>>>
>>>
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>> <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>>
>>
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
--
Brian Stansberry
Manager, Senior Principal Software Engineer
JBoss by Red Hat
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat
9:35 p.m.
On 2 Dec 2016 11:00 PM, "Josef Cacek" <jcacek(a)redhat.com> wrote:
I think this is massive overkill, and would severely reduce the quality and
usability of the testsuite (more tests != a better test suite, they need to
be useful test otherwise they just add technical debt for no gain).
I just did a quick search through the code base, and as far as I can tell
there is maybe ~50 test cases that test security related functionality
(i.e. use the SecurityDomainSetup functionality), out of ~950 test cases,
so this would amount to adding around 900 useless test cases.
The argument that Elytron may cause regressions in these cases does really
work either, because these tests are not linked to a security domain
Elytron should never actually be activated, and even if this was a concern
the correct approach would be to write some smoke tests to ensure that non
Elytron deployments still work correctly.
The expediency argument also does not really work, as the tests that
actually use security will need to be modified anyway so they use Elytron
instead of the old security subsystem, so if the whole test suite was
copied the only tests that worked 'out of the box' would be the tests that
don't actually test anything security related.
IMHO a better way to test this would be to move all security related tests
into their own module, and then create subclasses of these tests in Elytron
and legacy modules. Elytron should be added to the default config, so that
even non security related tests run with Elytron present (although we
should still have at least one test module that runs without it, just as a
smoke test to make sure that legacy configs without it will still work).
Stuart
> -- Josef
> ----- Original Message -----
> > From: "Tomaž Cerar" <tomaz.cerar(a)gmail.com > > To: "Darran Lofthouse"
<darran.lofthouse(a)jboss.com > > Cc:
wildfly-dev(a)lists.jboss.org
> > Sent: Friday, December 2, 2016 12:37:22 PM
> > Subject: Re: [wildfly-dev] Elytron integration tests in WildFly
testsuite
> > > That is probably fine, but! it should be done
differently.
> > > instead of duplicating whole testsuite (and adding
extra hour to
execution
> > and extra headaches with intermittent problems and duplication of
> > maintenance)
> > I would suggest that all security related tests get extracted to new
> > "security" testsuite module and than only that part is duplicated.
> > > This way we will have all security related stuff in
one place.
> > > > >
On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse <
> > darran.lofthouse(a)jboss.com > wrote:
> > > > Probably should add - any
duplication should only be for security tests
> > - not everything else in there!
> > > On 02/12/16 11:08, Darran Lofthouse wrote:
> > > On 02/12/16 11:03, Tomaž Cerar wrote:
> > > > > >> On Fri, Dec 2,
2016 at 9:11 AM, Josef Cacek < jcacek(a)redhat.com
> > >> <mailto: jcacek(a)redhat.com >> wrote:
> > > > > >> The modules would
just live side by side - basic would use Elytron
> > >> configuration, basic-legacy-security would use configuration similar
> > >> to (or same as) the current server configuration.
> > > > > > > > > > >
>> What would this actually mean?
> > >> we will have two copies basic tests suites one running with elytron
> > >> another with legacy security subsystem?
> > > > > >> Do I read that
right? Please say I am not.
> > > > > That is correct - we
have two security implementations they both need
> > > testing.
> > > > > One needs testing for
backwards compatibility and regressions, the
other
> > > for equivalent behaviour and then new features and bugs.
> > > > > Needing to test both
was discussed previously so this is more about
how
> > > > > > > >
>> _______________________________________________
> > >> wildfly-dev mailing list
> > >> wildfly-dev(a)lists.jboss.org
> > >> https://lists.jboss.org/mailman/listinfo/wildfly-dev
> > > > > > > >
_______________________________________________
> > wildfly-dev mailing list
> > wildfly-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/wildfly-dev
> > > >
_______________________________________________
> > wildfly-dev mailing list
> > wildfly-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/wildfly-dev
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
We prefer duplication of all tests in the testsuite to keep test coverage.
To be sure we don't have regressions with introducing Elytron and at the
same
time we are able to cover current features with Elytron, we must run
all tests with both security subsystems.
Even if a test is not security related on the first glance, the
tested
component may use a security feature internally. By switching to the new
security subsystem the feature may stop work.
Let's keep the current (legacy security based) tests alive until
the
legacy security subsystem is fully removed.
> Then we'll simply remove the *-legacy-security testsuite modules.
> > to separate both and also give the Elytron testing a good
foundation
to
> > > start from.
> >
7:30 a.m.
I am not a fan of the subclassing as at some point in the future we do
need to be able to delete the legacy tests so I would like to keep them
independent enough so we can do that without another testsuite refactoring.
But I do like the moving the 50 tests out, that fits more closely with
the three modules I described earlier.
So a new legacy-security module where we move the existing security
tests to. Clone this to a new security module and those tests will be
enhanced to test Elytron specifically, this new module will also be
where all new tests get added.
So that leaves 900 tests behind - I would suggest that remains by
default using the default configuration of the server only.
Then if we want I think as a side task the 900 tests could have some
alternative profiles to be defined so we can ensure the following
permutations are covered: -
1 - security extension installed.
2 - elytron extenstion installed.
3 - security and elytron extensions installed.
4 - neither extension installed.
I think the default config will either be #2 or #3 so the testing will
be more about verifying we don't have hidden dependencies on specific
extensions.
We probably don't want the overhead of those permutations in either
local builds or our current CI builds but maybe a separate CI build to
run those to watch for regressions in those areas.
I expect the *-elytron.xml configurations will be removed entirely from
the distribution.
Regards,
Darran Lofthouse.
On 04/12/16 02:35, Stuart Douglas wrote:
On 2 Dec 2016 11:00 PM, "Josef Cacek"
<jcacek(a)redhat.com
<mailto:jcacek@redhat.com>> wrote:
>
> We prefer duplication of all tests in the testsuite to keep test coverage.
>
> To be sure we don't have regressions with introducing Elytron and at
the same time we are able to cover current features with Elytron, we
must run all tests with both security subsystems.
> Even if a test is not security related on the first glance, the tested
component may use a security feature internally. By switching to the new
security subsystem the feature may stop work.
>
I think this is massive overkill, and would severely reduce the quality
and usability of the testsuite (more tests != a better test suite, they
need to be useful test otherwise they just add technical debt for no gain).
I just did a quick search through the code base, and as far as I can
tell there is maybe ~50 test cases that test security related
functionality (i.e. use the SecurityDomainSetup functionality), out of
~950 test cases, so this would amount to adding around 900 useless test
cases.
The argument that Elytron may cause regressions in these cases does
really work either, because these tests are not linked to a security
domain Elytron should never actually be activated, and even if this was
a concern the correct approach would be to write some smoke tests to
ensure that non Elytron deployments still work correctly.
The expediency argument also does not really work, as the tests that
actually use security will need to be modified anyway so they use
Elytron instead of the old security subsystem, so if the whole test
suite was copied the only tests that worked 'out of the box' would be
the tests that don't actually test anything security related.
IMHO a better way to test this would be to move all security related
tests into their own module, and then create subclasses of these tests
in Elytron and legacy modules. Elytron should be added to the default
config, so that even non security related tests run with Elytron present
(although we should still have at least one test module that runs
without it, just as a smoke test to make sure that legacy configs
without it will still work).
Stuart
> Let's keep the current (legacy security based) tests alive until the
legacy security subsystem is fully removed.
> Then we'll simply remove the *-legacy-security testsuite modules.
>
> -- Josef
>
> ----- Original Message -----
> > From: "Tomaž Cerar" <tomaz.cerar(a)gmail.com
<mailto:tomaz.cerar@gmail.com>>
> > To: "Darran Lofthouse" <darran.lofthouse(a)jboss.com
<mailto:darran.lofthouse@jboss.com>>
> > Cc: wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
> > Sent: Friday, December 2, 2016 12:37:22 PM
> > Subject: Re: [wildfly-dev] Elytron integration tests in WildFly
testsuite
> >
> > That is probably fine, but! it should be done differently.
> >
> > instead of duplicating whole testsuite (and adding extra hour to
execution
> > and extra headaches with intermittent problems and duplication of
> > maintenance)
> > I would suggest that all security related tests get extracted to new
> > "security" testsuite module and than only that part is duplicated.
> >
> > This way we will have all security related stuff in one place.
> >
> >
> >
> > On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse <
> > darran.lofthouse(a)jboss.com <mailto:darran.lofthouse@jboss.com> >
wrote:
> >
> >
> > Probably should add - any duplication should only be for security tests
> > - not everything else in there!
> >
> > On 02/12/16 11:08, Darran Lofthouse wrote:
> > > On 02/12/16 11:03, Tomaž Cerar wrote:
> > >>
> > >> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek < jcacek(a)redhat.com
<mailto:jcacek@redhat.com>
> > >> <mailto: jcacek(a)redhat.com <mailto:jcacek@redhat.com> >>
wrote:
> > >>
> > >> The modules would just live side by side - basic would use Elytron
> > >> configuration, basic-legacy-security would use configuration similar
> > >> to (or same as) the current server configuration.
> > >>
> > >>
> > >>
> > >> What would this actually mean?
> > >> we will have two copies basic tests suites one running with elytron
> > >> another with legacy security subsystem?
> > >>
> > >> Do I read that right? Please say I am not.
> > >
> > > That is correct - we have two security implementations they both need
> > > testing.
> > >
> > > One needs testing for backwards compatibility and regressions, the
other
> > > for equivalent behaviour and then new features and bugs.
> > >
> > > Needing to test both was discussed previously so this is more
about how
> > > to separate both and also give the Elytron testing a good
foundation to
> > > start from.
> > >
> > >>
> > >>
> > >> _______________________________________________
> > >> wildfly-dev mailing list
> > >> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
> > >> https://lists.jboss.org/mailman/listinfo/wildfly-dev
<https://lists.jboss.org/mailman/listinfo/wildfly-dev>
> > >>
> > >
> >
> > _______________________________________________
> > wildfly-dev mailing list
> > wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
> > https://lists.jboss.org/mailman/listinfo/wildfly-dev
<https://lists.jboss.org/mailman/listinfo/wildfly-dev>
> >
> >
> > _______________________________________________
> > wildfly-dev mailing list
> > wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
> > https://lists.jboss.org/mailman/listinfo/wildfly-dev
<https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
<https://lists.jboss.org/mailman/listinfo/wildfly-dev>
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
6:16 a.m.
On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek < jcacek(a)redhat.com
> wrote:
> The modules would just live side by side - basic would use
Elytron
> configuration, basic-legacy-security would use configuration similar to (or
> same as) the current server configuration.
What would this actually mean?
we will have two copies basic tests suites one running with elytron another
with legacy security subsystem?
Do I read that right? Please say I am not.
Not only the "basic". All testsuite modules in ideal case.
12:22 p.m.
Instead of duplicating the entire test suite can we just add a new CI job
that runs the tests with the *-elytron.xml configuration file? Duplicating
the entire testsuite will end in nothing but headaches IMO. Like Kabir
mentioned any bug in a test will need to be fixed in two locations.
On Fri, Dec 2, 2016 at 12:11 AM, Josef Cacek <jcacek(a)redhat.com> wrote:
Hi,
EAP QE and Elytron Dev teams would like to raise a discussion about how
the Elytron integration tests will be added to the WildFly/EAP testsuite.
We would like to agree on a unified approach across the testsuite, so we
don't end up in situation where each component comes with its own
"best-fitting" solution.
We see following requirements:
1. new features which come with Elytron need to be covered by integration
tests
2. existing scenarios have to stay working when Elytron is configured as
the default security subsystem
3. legacy security subsystem must stay working in parallel
Our teams came to agreement that the safest way would be to copy whole
existing testsuite modules into new ones. E.g.
cp -r testsuite/integration/basic testsuite/integration/basic-
legacy-security
# + Reconfigure testsuite/integration/basic to use Elytron
# + Add @Ignore to all tests which fail with Elytron or use directly the
old security configuration
The modules would just live side by side - basic would use Elytron
configuration, basic-legacy-security would use configuration similar to (or
same as) the current server configuration.
In the next step we will go through the Ignored tests (the TODO list) for
Elytron and rewrite them or report issues if necessary.
By this way we would keep all the test coverage for legacy security. We
would also have all existing scenarios covered (or at least present) for
Elytron configuration.
Are there any objections to this suggestion?
Thanks,
-- Josef
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
--
James R. Perkins
JBoss by Red Hat
1:22 p.m.
While it’s important that we maintain some legacy coverage, I think its important that our
testsuite prioritize coverage of the new vs the legacy, and this can be done organically.
New tests should use elytron, and old tests that are updated, which aren’t specifically
testing legacy compatibility should be converted over gradually, when convenient.
My main rationale hare is that code which is continually evolved is more likely to break
than that which is largely frozen. It’s really the interop pieces that interact with
legacy areas that are of most in need of continued coverage.
On Dec 2, 2016, at 11:22 AM, James Perkins
<jperkins(a)redhat.com> wrote:
Instead of duplicating the entire test suite can we just add a new CI job that runs the
tests with the *-elytron.xml configuration file? Duplicating the entire testsuite will end
in nothing but headaches IMO. Like Kabir mentioned any bug in a test will need to be fixed
in two locations.
On Fri, Dec 2, 2016 at 12:11 AM, Josef Cacek <jcacek(a)redhat.com
<mailto:jcacek@redhat.com>> wrote:
Hi,
EAP QE and Elytron Dev teams would like to raise a discussion about how the Elytron
integration tests will be added to the WildFly/EAP testsuite.
We would like to agree on a unified approach across the testsuite, so we don't end up
in situation where each component comes with its own "best-fitting" solution.
We see following requirements:
1. new features which come with Elytron need to be covered by integration tests
2. existing scenarios have to stay working when Elytron is configured as the default
security subsystem
3. legacy security subsystem must stay working in parallel
Our teams came to agreement that the safest way would be to copy whole existing testsuite
modules into new ones. E.g.
cp -r testsuite/integration/basic testsuite/integration/basic-legacy-security
# + Reconfigure testsuite/integration/basic to use Elytron
# + Add @Ignore to all tests which fail with Elytron or use directly the old security
configuration
The modules would just live side by side - basic would use Elytron configuration,
basic-legacy-security would use configuration similar to (or same as) the current server
configuration.
In the next step we will go through the Ignored tests (the TODO list) for Elytron and
rewrite them or report issues if necessary.
By this way we would keep all the test coverage for legacy security. We would also have
all existing scenarios covered (or at least present) for Elytron configuration.
Are there any objections to this suggestion?
Thanks,
-- Josef
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/wildfly-dev
<https://lists.jboss.org/mailman/listinfo/wildfly-dev>
--
James R. Perkins
JBoss by Red Hat
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat
2695
days inactive
2700
days old
23 comments
9 participants
participants (9)
-
Brian Stansberry -
Darran Lofthouse -
James Perkins -
Jan Kalina -
Jason Greene -
Josef Cacek -
Kabir Khan -
Stuart Douglas -
Tomaž Cerar