[wildfly-dev] Elytron integration tests in WildFly testsuite

[Josef Cacek]

The 3-way split is a good idea, but it would only work if
- we are sure the non-security tests in shared modules stay really independent - i.e. no new Ignores, Assumes, configuration based conditions;
- all shared modules have profiles for running with legacy security.

It's not clear how many new modules would be necessary with the 3 way split. Current TS modules don't share server configuration and they (may) contain several surefire executions with different server profiles.

-- Josef

----- Original Message -----
> From: "Darran Lofthouse" <darran.lofthouse at jboss.com>
> To: wildfly-dev at lists.jboss.org
> Sent: Monday, December 5, 2016 5:28:02 PM
> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite
> 
> Regarding that 25% failure - I would expect a lot of that to be picked
> up anyway as we merge the *-elytron.xml configurations into the default
> configurations and then drop the *-elytron configurations.
> 
> How about my reply to Stuart's e-mail and a 3-way split,
> legacy-security, security, and basic where basic can have optional
> profiles added to test different extension availability?
> 
> We could then do as Kabir suggests and pull out additional tests later
> if we do find a reason to justify duplicating the test and in the
> meantime have the profiles available to run the tests.
> 
> Regards,
> Darran Lofthouse.
> 
> On 05/12/16 15:03, Josef Cacek wrote:
> > Hi,
> >
> > We still see the copying whole modules as the best way. Let's recap our
> > view.
> >
> > Tested components may use a security feature internally and without test
> > coverage we would not realize that the feature stopped working. Running
> > all tests will help us to discover regressions before they occur in
> > application server.
> >
> > Pros:
> > - we don't lose testcoverage, we can simply detect regressions. It also
> > provides possible early failure detection for dev.
> > - it would be useful for migration, it shows migration path - user can take
> > a look on tests with legacy solution and compare their configuration with
> > tests which use Elytron configuration
> > - by each testsuite run we see how many tests are skipped (i.e. TODO list
> > for Elytron integration)
> > - it should be simple to reconfigure the project (copy module; update
> > metadata in pom.xml; ignore failing tests)
> > - it's simple to remove legacy testsuite modules once the legacy security
> > subsystem is removed from WildFly
> >
> > Cons and their solutions:
> > - double fixing of tests - we don't expect many such cases, the legacy
> > should be handled as frozen (rewriting/adding only on need-to-have basis).
> > Only custormer ticket related tests could be backported, it means only
> > unit of tests per months.
> > - testsuite execution time - we don't need to run the legacy modules with
> > each PR, it can be resolved by using profiles
> >
> > As a smoke test we tried to run testsuite/integration-tests/basic module
> > with standalone-elytron.xml and standalone-full-elytron.xml (merged
> > standalone-full.xml and standalone-elytron.xml). There is around 25% test
> > failures/errors. Also some testcases, which seem that are not directly
> > related to security, failed.
> >
> > Thanks,
> >
> > -- Josef
> >
> >
> > ----- Original Message -----
> >> From: "Kabir Khan" <kabir.khan at jboss.com>
> >> To: "Brian Stansberry" <brian.stansberry at redhat.com>
> >> Cc: wildfly-dev at lists.jboss.org
> >> Sent: Friday, December 2, 2016 4:28:07 PM
> >> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite
> >>
> >> +1 doubling the testsuite size/time is not practical. Also, there is the
> >> issue that if all tests are duplicated, when people come to enhance an
> >> existing test they will need to do so in two places.
> >>
> >> Could we not do what was suggested earlier and pull out tests which
> >> obviously
> >> target security into two legacy/elytron modules, and leave the rest where
> >> they are? Then if something later turns out to have 'hidden' implications,
> >> that test gets pulled out into the two modules.
> >>
> >> Regarding the deprecated legacy stuff, I think we will have to keep that
> >> around for many years still due to it being used by the product (and in
> >> any
> >> case in that branch, although removing it from WF might be possible
> >> sooner).
> > _______________________________________________
> > wildfly-dev mailing list
> > wildfly-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/wildfly-dev
> >
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>