[wildfly-dev] Elytron integration tests in WildFly testsuite

Mon Dec 5 07:30:35 EST 2016

I am not a fan of the subclassing as at some point in the future we do 
need to be able to delete the legacy tests so I would like to keep them 
independent enough so we can do that without another testsuite refactoring.

But I do like the moving the 50 tests out, that fits more closely with 
the three modules I described earlier.

So a new legacy-security module where we move the existing security 
tests to.  Clone this to a new security module and those tests will be 
enhanced to test Elytron specifically, this new module will also be 
where all new tests get added.

So that leaves 900 tests behind - I would suggest that remains by 
default using the default configuration of the server only.

Then if we want I think as a side task the 900 tests could have some 
alternative profiles to be defined so we can ensure the following 
permutations are covered: -

  1 - security extension installed.
  2 - elytron extenstion installed.
  3 - security and elytron extensions installed.
  4 - neither extension installed.

I think the default config will either be #2 or #3 so the testing will 
be more about verifying we don't have hidden dependencies on specific 
extensions.

We probably don't want the overhead of those permutations in either 
local builds or our current CI builds but maybe a separate CI build to 
run those to watch for regressions in those areas.

I expect the *-elytron.xml configurations will be removed entirely from 
the distribution.

Regards,
Darran Lofthouse.

On 04/12/16 02:35, Stuart Douglas wrote:
> On 2 Dec 2016 11:00 PM, "Josef Cacek" <jcacek at redhat.com
> <mailto:jcacek at redhat.com>> wrote:
>>
>> We prefer duplication of all tests in the testsuite to keep test coverage.
>>
>> To be sure we don't have regressions with introducing Elytron and at
> the same time we are able to cover current features with Elytron, we
> must run all tests with both security subsystems.
>> Even if a test is not security related on the first glance, the tested
> component may use a security feature internally. By switching to the new
> security subsystem the feature may stop work.
>>
>
> I think this is massive overkill, and would severely reduce the quality
> and usability of the testsuite (more tests != a better test suite, they
> need to be useful test otherwise they just add technical debt for no gain).
>
> I just did a quick search through the code base, and as far as I can
> tell there is maybe ~50 test cases that test security related
> functionality (i.e. use the SecurityDomainSetup functionality), out of
> ~950 test cases, so this would amount to adding around 900 useless test
> cases.
>
> The argument that Elytron may cause regressions in these cases does
> really work either, because these tests are not linked to a security
> domain Elytron should never actually be activated, and even if this was
> a concern the correct approach would be to write some smoke tests to
> ensure that non Elytron deployments still work correctly.
>
> The expediency argument also does not really work, as the tests that
> actually use security will need to be modified anyway so they use
> Elytron instead of the old security subsystem, so if the whole test
> suite was copied the only tests that worked 'out of the box' would be
> the tests that don't actually test anything security related.
>
> IMHO a better way to test this would be to move all security related
> tests into their own module, and then create subclasses of these tests
> in Elytron and legacy modules. Elytron should be added to the default
> config, so that even non security related tests run with Elytron present
> (although we should still have at least one test module that runs
> without it, just as a smoke test to make sure that legacy configs
> without it will still work).
>
> Stuart
>
>
>> Let's keep the current (legacy security based) tests alive until the
> legacy security subsystem is fully removed.
>> Then we'll simply remove the *-legacy-security testsuite modules.
>>
>> -- Josef
>>
>> ----- Original Message -----
>> > From: "Tomaž Cerar" <tomaz.cerar at gmail.com
> <mailto:tomaz.cerar at gmail.com>>
>> > To: "Darran Lofthouse" <darran.lofthouse at jboss.com
> <mailto:darran.lofthouse at jboss.com>>
>> > Cc: wildfly-dev at lists.jboss.org <mailto:wildfly-dev at lists.jboss.org>
>> > Sent: Friday, December 2, 2016 12:37:22 PM
>> > Subject: Re: [wildfly-dev] Elytron integration tests in WildFly
> testsuite
>> >
>> > That is probably fine, but! it should be done differently.
>> >
>> > instead of duplicating whole testsuite (and adding extra hour to
> execution
>> > and extra headaches with intermittent problems and duplication of
>> > maintenance)
>> > I would suggest that all security related tests get extracted to new
>> > "security" testsuite module and than only that part is duplicated.
>> >
>> > This way we will have all security related stuff in one place.
>> >
>> >
>> >
>> > On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse <
>> > darran.lofthouse at jboss.com <mailto:darran.lofthouse at jboss.com> > wrote:
>> >
>> >
>> > Probably should add - any duplication should only be for security tests
>> > - not everything else in there!
>> >
>> > On 02/12/16 11:08, Darran Lofthouse wrote:
>> > > On 02/12/16 11:03, Tomaž Cerar wrote:
>> > >>
>> > >> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek < jcacek at redhat.com
> <mailto:jcacek at redhat.com>
>> > >> <mailto: jcacek at redhat.com <mailto:jcacek at redhat.com> >> wrote:
>> > >>
>> > >> The modules would just live side by side - basic would use Elytron
>> > >> configuration, basic-legacy-security would use configuration similar
>> > >> to (or same as) the current server configuration.
>> > >>
>> > >>
>> > >>
>> > >> What would this actually mean?
>> > >> we will have two copies basic tests suites one running with elytron
>> > >> another with legacy security subsystem?
>> > >>
>> > >> Do I read that right? Please say I am not.
>> > >
>> > > That is correct - we have two security implementations they both need
>> > > testing.
>> > >
>> > > One needs testing for backwards compatibility and regressions, the
> other
>> > > for equivalent behaviour and then new features and bugs.
>> > >
>> > > Needing to test both was discussed previously so this is more
> about how
>> > > to separate both and also give the Elytron testing a good
> foundation to
>> > > start from.
>> > >
>> > >>
>> > >>
>> > >> _______________________________________________
>> > >> wildfly-dev mailing list
>> > >> wildfly-dev at lists.jboss.org <mailto:wildfly-dev at lists.jboss.org>
>> > >> https://lists.jboss.org/mailman/listinfo/wildfly-dev
> <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>> > >>
>> > >
>> >
>> > _______________________________________________
>> > wildfly-dev mailing list
>> > wildfly-dev at lists.jboss.org <mailto:wildfly-dev at lists.jboss.org>
>> > https://lists.jboss.org/mailman/listinfo/wildfly-dev
> <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>> >
>> >
>> > _______________________________________________
>> > wildfly-dev mailing list
>> > wildfly-dev at lists.jboss.org <mailto:wildfly-dev at lists.jboss.org>
>> > https://lists.jboss.org/mailman/listinfo/wildfly-dev
> <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev at lists.jboss.org <mailto:wildfly-dev at lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
> <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>
>
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>