The modules would just live side by side - basic would use Elytron configuration, basic-legacy-security would use configuration similar to (or same as) the current server configuration.

On 02/12/16 11:03, Tomaž Cerar wrote: > > On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek <jcacek(a)redhat.com > <mailto:jcacek@redhat.com>> wrote: > > The modules would just live side by side - basic would use Elytron > configuration, basic-legacy-security would use configuration similar > to (or same as) the current server configuration. > > > > What would this actually mean? > we will have two copies basic tests suites one running with elytron > another with legacy security subsystem? > > Do I read that right? Please say I am not. That is correct - we have two security implementations they both need testing. One needs testing for backwards compatibility and regressions, the other for equivalent behaviour and then new features and bugs. Needing to test both was discussed previously so this is more about how to separate both and also give the Elytron testing a good foundation to start from. > > > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/wildfly-dev >

Probably should add - any duplication should only be for security tests - not everything else in there! On 02/12/16 11:08, Darran Lofthouse wrote: > On 02/12/16 11:03, Tomaž Cerar wrote: >> >> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek <jcacek(a)redhat.com >> <mailto:jcacek@redhat.com>> wrote: >> >> The modules would just live side by side - basic would use Elytron >> configuration, basic-legacy-security would use configuration similar >> to (or same as) the current server configuration. >> >> >> >> What would this actually mean? >> we will have two copies basic tests suites one running with elytron >> another with legacy security subsystem? >> >> Do I read that right? Please say I am not. > > That is correct - we have two security implementations they both need > testing. > > One needs testing for backwards compatibility and regressions, the other > for equivalent behaviour and then new features and bugs. > > Needing to test both was discussed previously so this is more about how > to separate both and also give the Elytron testing a good foundation to > start from. > >> >> >> _______________________________________________ >> wildfly-dev mailing list >> wildfly-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/wildfly-dev >> > _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

From: "Tomaž Cerar" <tomaz.cerar(a)gmail.com&gt; To: "Darran Lofthouse" <darran.lofthouse(a)jboss.com&gt; Cc: wildfly-dev(a)lists.jboss.org Sent: Friday, December 2, 2016 12:37:22 PM Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite That is probably fine, but! it should be done differently. instead of duplicating whole testsuite (and adding extra hour to execution and extra headaches with intermittent problems and duplication of maintenance) I would suggest that all security related tests get extracted to new "security" testsuite module and than only that part is duplicated. This way we will have all security related stuff in one place. On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse < darran.lofthouse(a)jboss.com > wrote: Probably should add - any duplication should only be for security tests - not everything else in there! On 02/12/16 11:08, Darran Lofthouse wrote: > On 02/12/16 11:03, Tomaž Cerar wrote: >> >> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek < jcacek(a)redhat.com >> <mailto: jcacek(a)redhat.com >> wrote: >> >> The modules would just live side by side - basic would use Elytron >> configuration, basic-legacy-security would use configuration similar >> to (or same as) the current server configuration. >> >> >> >> What would this actually mean? >> we will have two copies basic tests suites one running with elytron >> another with legacy security subsystem? >> >> Do I read that right? Please say I am not. > > That is correct - we have two security implementations they both need > testing. > > One needs testing for backwards compatibility and regressions, the other > for equivalent behaviour and then new features and bugs. > > Needing to test both was discussed previously so this is more about how > to separate both and also give the Elytron testing a good foundation to > start from. > >> >> >> _______________________________________________ >> wildfly-dev mailing list >> wildfly-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/wildfly-dev >> > _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

We prefer duplication of all tests in the testsuite to keep test coverage. To be sure we don't have regressions with introducing Elytron and at the same time we are able to cover current features with Elytron, we must run all tests with both security subsystems. Even if a test is not security related on the first glance, the tested component may use a security feature internally. By switching to the new security subsystem the feature may stop work. Let's keep the current (legacy security based) tests alive until the legacy security subsystem is fully removed. Then we'll simply remove the *-legacy-security testsuite modules. -- Josef ----- Original Message ----- > From: "Tomaž Cerar" <tomaz.cerar(a)gmail.com&gt; > To: "Darran Lofthouse" <darran.lofthouse(a)jboss.com&gt; > Cc: wildfly-dev(a)lists.jboss.org > Sent: Friday, December 2, 2016 12:37:22 PM > Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite > > That is probably fine, but! it should be done differently. > > instead of duplicating whole testsuite (and adding extra hour to execution > and extra headaches with intermittent problems and duplication of > maintenance) > I would suggest that all security related tests get extracted to new > "security" testsuite module and than only that part is duplicated. > > This way we will have all security related stuff in one place. > > > > On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse < > darran.lofthouse(a)jboss.com > wrote: > > > Probably should add - any duplication should only be for security tests > - not everything else in there! > > On 02/12/16 11:08, Darran Lofthouse wrote: > > On 02/12/16 11:03, Tomaž Cerar wrote: > >> > >> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek < jcacek(a)redhat.com > >> <mailto: jcacek(a)redhat.com >> wrote: > >> > >> The modules would just live side by side - basic would use Elytron > >> configuration, basic-legacy-security would use configuration similar > >> to (or same as) the current server configuration. > >> > >> > >> > >> What would this actually mean? > >> we will have two copies basic tests suites one running with elytron > >> another with legacy security subsystem? > >> > >> Do I read that right? Please say I am not. > > > > That is correct - we have two security implementations they both need > > testing. > > > > One needs testing for backwards compatibility and regressions, the other > > for equivalent behaviour and then new features and bugs. > > > > Needing to test both was discussed previously so this is more about how > > to separate both and also give the Elytron testing a good foundation to > > start from. > > > >> > >> > >> _______________________________________________ > >> wildfly-dev mailing list > >> wildfly-dev(a)lists.jboss.org > >> https://lists.jboss.org/mailman/listinfo/wildfly-dev > >> > > > > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/wildfly-dev > > > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/wildfly-dev

On Dec 2, 2016, at 8:59 AM, Darran Lofthouse <darran.lofthouse(a)jboss.com&gt; wrote: No the part we can't avoid is that we have two security implementations in the server and we have to have tests for both. The e-mail from Josef is a suggestion as to how this will be achieved. Other suggestions are welcome ;-) But we have two security implementations to test. One of these is deprecated and will hopefully be removed at some point so ideally we have something that allows testing to legacy to be removed without too much pain. Writing sufficient test coverage is not a small task so anything that gives us more than starting from scratch is a benefit. But maybe it would be worth summarising these additional scenarios that will be affected by changes. Regards, Darran Lofthouse. On 02/12/16 14:47, Tomaž Cerar wrote: > So if I understood all this correctly, > this mail here was more of a heads up what is happening > than a discussion on how could or should be best done. > > I just hope you (the team behind decisions) considered all scenarios how > will that impact everyone involved. > > -- > tomaz > > On Fri, Dec 2, 2016 at 12:58 PM, Josef Cacek <jcacek(a)redhat.com > <mailto:jcacek@redhat.com>> wrote: > > We prefer duplication of all tests in the testsuite to keep test > coverage. > > To be sure we don't have regressions with introducing Elytron and at > the same time we are able to cover current features with Elytron, we > must run all tests with both security subsystems. > Even if a test is not security related on the first glance, the > tested component may use a security feature internally. By switching > to the new security subsystem the feature may stop work. > > Let's keep the current (legacy security based) tests alive until the > legacy security subsystem is fully removed. > Then we'll simply remove the *-legacy-security testsuite modules. > > -- Josef > > ----- Original Message ----- >> From: "Tomaž Cerar" <tomaz.cerar(a)gmail.com > <mailto:tomaz.cerar@gmail.com>> >> To: "Darran Lofthouse" <darran.lofthouse(a)jboss.com > <mailto:darran.lofthouse@jboss.com>> >> Cc: wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org> >> Sent: Friday, December 2, 2016 12:37:22 PM >> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly > testsuite >> >> That is probably fine, but! it should be done differently. >> >> instead of duplicating whole testsuite (and adding extra hour to > execution >> and extra headaches with intermittent problems and duplication of >> maintenance) >> I would suggest that all security related tests get extracted to new >> "security" testsuite module and than only that part is duplicated. >> >> This way we will have all security related stuff in one place. >> >> >> >> On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse < >> darran.lofthouse(a)jboss.com <mailto:darran.lofthouse@jboss.com> > > wrote: >> >> >> Probably should add - any duplication should only be for security > tests >> - not everything else in there! >> >> On 02/12/16 11:08, Darran Lofthouse wrote: >>> On 02/12/16 11:03, Tomaž Cerar wrote: >>>> >>>> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek < jcacek(a)redhat.com > <mailto:jcacek@redhat.com> >>>> <mailto: jcacek(a)redhat.com <mailto:jcacek@redhat.com> >> wrote: >>>> >>>> The modules would just live side by side - basic would use Elytron >>>> configuration, basic-legacy-security would use configuration > similar >>>> to (or same as) the current server configuration. >>>> >>>> >>>> >>>> What would this actually mean? >>>> we will have two copies basic tests suites one running with elytron >>>> another with legacy security subsystem? >>>> >>>> Do I read that right? Please say I am not. >>> >>> That is correct - we have two security implementations they both > need >>> testing. >>> >>> One needs testing for backwards compatibility and regressions, > the other >>> for equivalent behaviour and then new features and bugs. >>> >>> Needing to test both was discussed previously so this is more > about how >>> to separate both and also give the Elytron testing a good > foundation to >>> start from. >>> >>>> >>>> >>>> _______________________________________________ >>>> wildfly-dev mailing list >>>> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org> >>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev > <https://lists.jboss.org/mailman/listinfo/wildfly-dev> >>>> >>> >> >> _______________________________________________ >> wildfly-dev mailing list >> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org> >> https://lists.jboss.org/mailman/listinfo/wildfly-dev > <https://lists.jboss.org/mailman/listinfo/wildfly-dev> >> >> >> _______________________________________________ >> wildfly-dev mailing list >> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org> >> https://lists.jboss.org/mailman/listinfo/wildfly-dev > <https://lists.jboss.org/mailman/listinfo/wildfly-dev> > > _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

Hi, EAP QE and Elytron Dev teams would like to raise a discussion about how the Elytron integration tests will be added to the WildFly/EAP testsuite. We would like to agree on a unified approach across the testsuite, so we don't end up in situation where each component comes with its own "best-fitting" solution. We see following requirements: 1. new features which come with Elytron need to be covered by integration tests 2. existing scenarios have to stay working when Elytron is configured as the default security subsystem 3. legacy security subsystem must stay working in parallel Our teams came to agreement that the safest way would be to copy whole existing testsuite modules into new ones. E.g. cp -r testsuite/integration/basic testsuite/integration/basic- legacy-security # + Reconfigure testsuite/integration/basic to use Elytron # + Add @Ignore to all tests which fail with Elytron or use directly the old security configuration The modules would just live side by side - basic would use Elytron configuration, basic-legacy-security would use configuration similar to (or same as) the current server configuration. In the next step we will go through the Ignored tests (the TODO list) for Elytron and rewrite them or report issues if necessary. By this way we would keep all the test coverage for legacy security. We would also have all existing scenarios covered (or at least present) for Elytron configuration. Are there any objections to this suggestion? Thanks, -- Josef _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

On Dec 2, 2016, at 9:20 AM, Brian Stansberry <brian.stansberry(a)redhat.com&gt; wrote: Duplicating the whole testsuite is basically a minimal effort way of getting coverage by avoiding the task of test coverage analysis. Only justification for that is expediency. That may be sufficient justification; I don’t know. But assuming for the sake of argument it’s sufficient, doubling the amount of time needed to run the testsuite is a massive productivity loss and that means cutting that time down to the necessary minimum and nothing more would need to be a critical extreme high priority task. So, with this proposal, who would be the people assigned to this critical extreme high priority task? I’d like to see names, lots and lots of names. Darran Lofthouse not among them. ;) > On Dec 2, 2016, at 8:59 AM, Darran Lofthouse <darran.lofthouse(a)jboss.com&gt; wrote: > > No the part we can't avoid is that we have two security implementations > in the server and we have to have tests for both. The e-mail from Josef > is a suggestion as to how this will be achieved. > > Other suggestions are welcome ;-) > > But we have two security implementations to test. > > One of these is deprecated and will hopefully be removed at some point > so ideally we have something that allows testing to legacy to be removed > without too much pain. > > Writing sufficient test coverage is not a small task so anything that > gives us more than starting from scratch is a benefit. > > But maybe it would be worth summarising these additional scenarios that > will be affected by changes. > > Regards, > Darran Lofthouse. > > On 02/12/16 14:47, Tomaž Cerar wrote: >> So if I understood all this correctly, >> this mail here was more of a heads up what is happening >> than a discussion on how could or should be best done. >> >> I just hope you (the team behind decisions) considered all scenarios how >> will that impact everyone involved. >> >> -- >> tomaz >> >> On Fri, Dec 2, 2016 at 12:58 PM, Josef Cacek <jcacek(a)redhat.com >> <mailto:jcacek@redhat.com>> wrote: >> >> We prefer duplication of all tests in the testsuite to keep test >> coverage. >> >> To be sure we don't have regressions with introducing Elytron and at >> the same time we are able to cover current features with Elytron, we >> must run all tests with both security subsystems. >> Even if a test is not security related on the first glance, the >> tested component may use a security feature internally. By switching >> to the new security subsystem the feature may stop work. >> >> Let's keep the current (legacy security based) tests alive until the >> legacy security subsystem is fully removed. >> Then we'll simply remove the *-legacy-security testsuite modules. >> >> -- Josef >> >> ----- Original Message ----- >>> From: "Tomaž Cerar" <tomaz.cerar(a)gmail.com >> <mailto:tomaz.cerar@gmail.com>> >>> To: "Darran Lofthouse" <darran.lofthouse(a)jboss.com >> <mailto:darran.lofthouse@jboss.com>> >>> Cc: wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org> >>> Sent: Friday, December 2, 2016 12:37:22 PM >>> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly >> testsuite >>> >>> That is probably fine, but! it should be done differently. >>> >>> instead of duplicating whole testsuite (and adding extra hour to >> execution >>> and extra headaches with intermittent problems and duplication of >>> maintenance) >>> I would suggest that all security related tests get extracted to new >>> "security" testsuite module and than only that part is duplicated. >>> >>> This way we will have all security related stuff in one place. >>> >>> >>> >>> On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse < >>> darran.lofthouse(a)jboss.com <mailto:darran.lofthouse@jboss.com> > >> wrote: >>> >>> >>> Probably should add - any duplication should only be for security >> tests >>> - not everything else in there! >>> >>> On 02/12/16 11:08, Darran Lofthouse wrote: >>>> On 02/12/16 11:03, Tomaž Cerar wrote: >>>>> >>>>> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek < jcacek(a)redhat.com >> <mailto:jcacek@redhat.com> >>>>> <mailto: jcacek(a)redhat.com <mailto:jcacek@redhat.com> >> wrote: >>>>> >>>>> The modules would just live side by side - basic would use Elytron >>>>> configuration, basic-legacy-security would use configuration >> similar >>>>> to (or same as) the current server configuration. >>>>> >>>>> >>>>> >>>>> What would this actually mean? >>>>> we will have two copies basic tests suites one running with elytron >>>>> another with legacy security subsystem? >>>>> >>>>> Do I read that right? Please say I am not. >>>> >>>> That is correct - we have two security implementations they both >> need >>>> testing. >>>> >>>> One needs testing for backwards compatibility and regressions, >> the other >>>> for equivalent behaviour and then new features and bugs. >>>> >>>> Needing to test both was discussed previously so this is more >> about how >>>> to separate both and also give the Elytron testing a good >> foundation to >>>> start from. >>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> wildfly-dev mailing list >>>>> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org> >>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >> <https://lists.jboss.org/mailman/listinfo/wildfly-dev> >>>>> >>>> >>> >>> _______________________________________________ >>> wildfly-dev mailing list >>> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org> >>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >> <https://lists.jboss.org/mailman/listinfo/wildfly-dev> >>> >>> >>> _______________________________________________ >>> wildfly-dev mailing list >>> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org> >>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >> <https://lists.jboss.org/mailman/listinfo/wildfly-dev> >> >> > > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/wildfly-dev -- Brian Stansberry Manager, Senior Principal Software Engineer JBoss by Red Hat _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

On 2 Dec 2016 11:00 PM, "Josef Cacek" <jcacek(a)redhat.com <mailto:jcacek@redhat.com>> wrote: > > We prefer duplication of all tests in the testsuite to keep test coverage. > > To be sure we don't have regressions with introducing Elytron and at the same time we are able to cover current features with Elytron, we must run all tests with both security subsystems. > Even if a test is not security related on the first glance, the tested component may use a security feature internally. By switching to the new security subsystem the feature may stop work. > I think this is massive overkill, and would severely reduce the quality and usability of the testsuite (more tests != a better test suite, they need to be useful test otherwise they just add technical debt for no gain). I just did a quick search through the code base, and as far as I can tell there is maybe ~50 test cases that test security related functionality (i.e. use the SecurityDomainSetup functionality), out of ~950 test cases, so this would amount to adding around 900 useless test cases. The argument that Elytron may cause regressions in these cases does really work either, because these tests are not linked to a security domain Elytron should never actually be activated, and even if this was a concern the correct approach would be to write some smoke tests to ensure that non Elytron deployments still work correctly. The expediency argument also does not really work, as the tests that actually use security will need to be modified anyway so they use Elytron instead of the old security subsystem, so if the whole test suite was copied the only tests that worked 'out of the box' would be the tests that don't actually test anything security related. IMHO a better way to test this would be to move all security related tests into their own module, and then create subclasses of these tests in Elytron and legacy modules. Elytron should be added to the default config, so that even non security related tests run with Elytron present (although we should still have at least one test module that runs without it, just as a smoke test to make sure that legacy configs without it will still work). Stuart > Let's keep the current (legacy security based) tests alive until the legacy security subsystem is fully removed. > Then we'll simply remove the *-legacy-security testsuite modules. > > -- Josef > > ----- Original Message ----- > > From: "Tomaž Cerar" <tomaz.cerar(a)gmail.com <mailto:tomaz.cerar@gmail.com>> > > To: "Darran Lofthouse" <darran.lofthouse(a)jboss.com <mailto:darran.lofthouse@jboss.com>> > > Cc: wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org> > > Sent: Friday, December 2, 2016 12:37:22 PM > > Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite > > > > That is probably fine, but! it should be done differently. > > > > instead of duplicating whole testsuite (and adding extra hour to execution > > and extra headaches with intermittent problems and duplication of > > maintenance) > > I would suggest that all security related tests get extracted to new > > "security" testsuite module and than only that part is duplicated. > > > > This way we will have all security related stuff in one place. > > > > > > > > On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse < > > darran.lofthouse(a)jboss.com <mailto:darran.lofthouse@jboss.com> > wrote: > > > > > > Probably should add - any duplication should only be for security tests > > - not everything else in there! > > > > On 02/12/16 11:08, Darran Lofthouse wrote: > > > On 02/12/16 11:03, Tomaž Cerar wrote: > > >> > > >> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek < jcacek(a)redhat.com <mailto:jcacek@redhat.com> > > >> <mailto: jcacek(a)redhat.com <mailto:jcacek@redhat.com> >> wrote: > > >> > > >> The modules would just live side by side - basic would use Elytron > > >> configuration, basic-legacy-security would use configuration similar > > >> to (or same as) the current server configuration. > > >> > > >> > > >> > > >> What would this actually mean? > > >> we will have two copies basic tests suites one running with elytron > > >> another with legacy security subsystem? > > >> > > >> Do I read that right? Please say I am not. > > > > > > That is correct - we have two security implementations they both need > > > testing. > > > > > > One needs testing for backwards compatibility and regressions, the other > > > for equivalent behaviour and then new features and bugs. > > > > > > Needing to test both was discussed previously so this is more about how > > > to separate both and also give the Elytron testing a good foundation to > > > start from. > > > > > >> > > >> > > >> _______________________________________________ > > >> wildfly-dev mailing list > > >> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org> > > >> https://lists.jboss.org/mailman/listinfo/wildfly-dev <https://lists.jboss.org/mailman/listinfo/wildfly-dev> > > >> > > > > > > > _______________________________________________ > > wildfly-dev mailing list > > wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org> > > https://lists.jboss.org/mailman/listinfo/wildfly-dev <https://lists.jboss.org/mailman/listinfo/wildfly-dev> > > > > > > _______________________________________________ > > wildfly-dev mailing list > > wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org> > > https://lists.jboss.org/mailman/listinfo/wildfly-dev <https://lists.jboss.org/mailman/listinfo/wildfly-dev> > > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org> > https://lists.jboss.org/mailman/listinfo/wildfly-dev <https://lists.jboss.org/mailman/listinfo/wildfly-dev> _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

Hi, We still see the copying whole modules as the best way. Let's recap our view. Tested components may use a security feature internally and without test coverage we would not realize that the feature stopped working. Running all tests will help us to discover regressions before they occur in application server. Pros: - we don't lose testcoverage, we can simply detect regressions. It also provides possible early failure detection for dev. - it would be useful for migration, it shows migration path - user can take a look on tests with legacy solution and compare their configuration with tests which use Elytron configuration - by each testsuite run we see how many tests are skipped (i.e. TODO list for Elytron integration) - it should be simple to reconfigure the project (copy module; update metadata in pom.xml; ignore failing tests) - it's simple to remove legacy testsuite modules once the legacy security subsystem is removed from WildFly Cons and their solutions: - double fixing of tests - we don't expect many such cases, the legacy should be handled as frozen (rewriting/adding only on need-to-have basis). Only custormer ticket related tests could be backported, it means only unit of tests per months. - testsuite execution time - we don't need to run the legacy modules with each PR, it can be resolved by using profiles As a smoke test we tried to run testsuite/integration-tests/basic module with standalone-elytron.xml and standalone-full-elytron.xml (merged standalone-full.xml and standalone-elytron.xml). There is around 25% test failures/errors. Also some testcases, which seem that are not directly related to security, failed. Thanks, -- Josef ----- Original Message ----- > From: "Kabir Khan" <kabir.khan(a)jboss.com&gt; > To: "Brian Stansberry" <brian.stansberry(a)redhat.com&gt; > Cc: wildfly-dev(a)lists.jboss.org > Sent: Friday, December 2, 2016 4:28:07 PM > Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite > > +1 doubling the testsuite size/time is not practical. Also, there is the > issue that if all tests are duplicated, when people come to enhance an > existing test they will need to do so in two places. > > Could we not do what was suggested earlier and pull out tests which obviously > target security into two legacy/elytron modules, and leave the rest where > they are? Then if something later turns out to have 'hidden' implications, > that test gets pulled out into the two modules. > > Regarding the deprecated legacy stuff, I think we will have to keep that > around for many years still due to it being used by the product (and in any > case in that branch, although removing it from WF might be possible sooner). _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

From: "Darran Lofthouse" <darran.lofthouse(a)jboss.com&gt; To: wildfly-dev(a)lists.jboss.org Sent: Monday, December 5, 2016 5:28:02 PM Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite Regarding that 25% failure - I would expect a lot of that to be picked up anyway as we merge the *-elytron.xml configurations into the default configurations and then drop the *-elytron configurations. How about my reply to Stuart's e-mail and a 3-way split, legacy-security, security, and basic where basic can have optional profiles added to test different extension availability? We could then do as Kabir suggests and pull out additional tests later if we do find a reason to justify duplicating the test and in the meantime have the profiles available to run the tests. Regards, Darran Lofthouse. On 05/12/16 15:03, Josef Cacek wrote: > Hi, > > We still see the copying whole modules as the best way. Let's recap our > view. > > Tested components may use a security feature internally and without test > coverage we would not realize that the feature stopped working. Running > all tests will help us to discover regressions before they occur in > application server. > > Pros: > - we don't lose testcoverage, we can simply detect regressions. It also > provides possible early failure detection for dev. > - it would be useful for migration, it shows migration path - user can take > a look on tests with legacy solution and compare their configuration with > tests which use Elytron configuration > - by each testsuite run we see how many tests are skipped (i.e. TODO list > for Elytron integration) > - it should be simple to reconfigure the project (copy module; update > metadata in pom.xml; ignore failing tests) > - it's simple to remove legacy testsuite modules once the legacy security > subsystem is removed from WildFly > > Cons and their solutions: > - double fixing of tests - we don't expect many such cases, the legacy > should be handled as frozen (rewriting/adding only on need-to-have basis). > Only custormer ticket related tests could be backported, it means only > unit of tests per months. > - testsuite execution time - we don't need to run the legacy modules with > each PR, it can be resolved by using profiles > > As a smoke test we tried to run testsuite/integration-tests/basic module > with standalone-elytron.xml and standalone-full-elytron.xml (merged > standalone-full.xml and standalone-elytron.xml). There is around 25% test > failures/errors. Also some testcases, which seem that are not directly > related to security, failed. > > Thanks, > > -- Josef > > > ----- Original Message ----- >> From: "Kabir Khan" <kabir.khan(a)jboss.com&gt; >> To: "Brian Stansberry" <brian.stansberry(a)redhat.com&gt; >> Cc: wildfly-dev(a)lists.jboss.org >> Sent: Friday, December 2, 2016 4:28:07 PM >> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite >> >> +1 doubling the testsuite size/time is not practical. Also, there is the >> issue that if all tests are duplicated, when people come to enhance an >> existing test they will need to do so in two places. >> >> Could we not do what was suggested earlier and pull out tests which >> obviously >> target security into two legacy/elytron modules, and leave the rest where >> they are? Then if something later turns out to have 'hidden' implications, >> that test gets pulled out into the two modules. >> >> Regarding the deprecated legacy stuff, I think we will have to keep that >> around for many years still due to it being used by the product (and in >> any >> case in that branch, although removing it from WF might be possible >> sooner). > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/wildfly-dev > _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

As I think about this I am heading back to just a two way split. If we go for the two way split, lets say the following about the default WildFly configuration: - - It will be entirely backed by the Elytron subsystem. - Legacy security-realms will be removed. - The legacy security subsystem will be removed. - The legacy security extension will be removed. Existing users forward porting their configuration are unaffected as this only affects default new configuration. We can offer documentation on how to re-enable those items where really needed - anyone doing so hopefully submitting a Jira letting us know why they are doing so. This will mean the 900 "non-security" tests will certainly be verified that they run without the legacy security extension being installed. We then have a new legacy-security module where we add at the very least the legacy security extension back in which we copy the 50 or so security tests over. The security specific tests left behind are all cleaned up and ported to pure Elytron based testing. Anything that breaks as we switch the configuration over in the remaining 900 tests we will review on a case by case basis, also we will keep an eye on other subsystems we have touched as they may want a legacy test adding e.g. messaging is one we have been working on this week. Regards, Darran Lofthouse. On 06/12/16 07:40, Josef Cacek wrote: > The 3-way split is a good idea, but it would only work if > - we are sure the non-security tests in shared modules stay really > independent - i.e. no new Ignores, Assumes, configuration based > conditions; > - all shared modules have profiles for running with legacy security. > > It's not clear how many new modules would be necessary with the 3 way > split. Current TS modules don't share server configuration and they > (may) contain several surefire executions with different server profiles. > > -- Josef > > ----- Original Message ----- >> From: "Darran Lofthouse" <darran.lofthouse(a)jboss.com&gt; >> To: wildfly-dev(a)lists.jboss.org >> Sent: Monday, December 5, 2016 5:28:02 PM >> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly >> testsuite >> >> Regarding that 25% failure - I would expect a lot of that to be picked >> up anyway as we merge the *-elytron.xml configurations into the default >> configurations and then drop the *-elytron configurations. >> >> How about my reply to Stuart's e-mail and a 3-way split, >> legacy-security, security, and basic where basic can have optional >> profiles added to test different extension availability? >> >> We could then do as Kabir suggests and pull out additional tests later >> if we do find a reason to justify duplicating the test and in the >> meantime have the profiles available to run the tests. >> >> Regards, >> Darran Lofthouse. >> >> On 05/12/16 15:03, Josef Cacek wrote: >>> Hi, >>> >>> We still see the copying whole modules as the best way. Let's recap our >>> view. >>> >>> Tested components may use a security feature internally and without >>> test >>> coverage we would not realize that the feature stopped working. Running >>> all tests will help us to discover regressions before they occur in >>> application server. >>> >>> Pros: >>> - we don't lose testcoverage, we can simply detect regressions. It also >>> provides possible early failure detection for dev. >>> - it would be useful for migration, it shows migration path - user >>> can take >>> a look on tests with legacy solution and compare their configuration >>> with >>> tests which use Elytron configuration >>> - by each testsuite run we see how many tests are skipped (i.e. TODO >>> list >>> for Elytron integration) >>> - it should be simple to reconfigure the project (copy module; update >>> metadata in pom.xml; ignore failing tests) >>> - it's simple to remove legacy testsuite modules once the legacy >>> security >>> subsystem is removed from WildFly >>> >>> Cons and their solutions: >>> - double fixing of tests - we don't expect many such cases, the legacy >>> should be handled as frozen (rewriting/adding only on need-to-have >>> basis). >>> Only custormer ticket related tests could be backported, it means only >>> unit of tests per months. >>> - testsuite execution time - we don't need to run the legacy modules >>> with >>> each PR, it can be resolved by using profiles >>> >>> As a smoke test we tried to run testsuite/integration-tests/basic >>> module >>> with standalone-elytron.xml and standalone-full-elytron.xml (merged >>> standalone-full.xml and standalone-elytron.xml). There is around 25% >>> test >>> failures/errors. Also some testcases, which seem that are not directly >>> related to security, failed. >>> >>> Thanks, >>> >>> -- Josef >>> >>> >>> ----- Original Message ----- >>>> From: "Kabir Khan" <kabir.khan(a)jboss.com&gt; >>>> To: "Brian Stansberry" <brian.stansberry(a)redhat.com&gt; >>>> Cc: wildfly-dev(a)lists.jboss.org >>>> Sent: Friday, December 2, 2016 4:28:07 PM >>>> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly >>>> testsuite >>>> >>>> +1 doubling the testsuite size/time is not practical. Also, there >>>> is the >>>> issue that if all tests are duplicated, when people come to enhance an >>>> existing test they will need to do so in two places. >>>> >>>> Could we not do what was suggested earlier and pull out tests which >>>> obviously >>>> target security into two legacy/elytron modules, and leave the rest >>>> where >>>> they are? Then if something later turns out to have 'hidden' >>>> implications, >>>> that test gets pulled out into the two modules. >>>> >>>> Regarding the deprecated legacy stuff, I think we will have to keep >>>> that >>>> around for many years still due to it being used by the product >>>> (and in >>>> any >>>> case in that branch, although removing it from WF might be possible >>>> sooner). >>> _______________________________________________ >>> wildfly-dev mailing list >>> wildfly-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>> >> _______________________________________________ >> wildfly-dev mailing list >> wildfly-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>