Hi,
We still see the copying whole modules as the best way. Let's recap our view.
Tested components may use a security feature internally and without test coverage we would
not realize that the feature stopped working. Running all tests will help us to discover
regressions before they occur in application server.
Pros:
- we don't lose testcoverage, we can simply detect regressions. It also provides
possible early failure detection for dev.
- it would be useful for migration, it shows migration path - user can take a look on
tests with legacy solution and compare their configuration with tests which use Elytron
configuration
- by each testsuite run we see how many tests are skipped (i.e. TODO list for Elytron
integration)
- it should be simple to reconfigure the project (copy module; update metadata in pom.xml;
ignore failing tests)
- it's simple to remove legacy testsuite modules once the legacy security subsystem is
removed from WildFly
Cons and their solutions:
- double fixing of tests - we don't expect many such cases, the legacy should be
handled as frozen (rewriting/adding only on need-to-have basis). Only custormer ticket
related tests could be backported, it means only unit of tests per months.
- testsuite execution time - we don't need to run the legacy modules with each PR, it
can be resolved by using profiles
As a smoke test we tried to run testsuite/integration-tests/basic module with
standalone-elytron.xml and standalone-full-elytron.xml (merged standalone-full.xml and
standalone-elytron.xml). There is around 25% test failures/errors. Also some testcases,
which seem that are not directly related to security, failed.
Thanks,
-- Josef
----- Original Message -----
From: "Kabir Khan" <kabir.khan(a)jboss.com>
To: "Brian Stansberry" <brian.stansberry(a)redhat.com>
Cc: wildfly-dev(a)lists.jboss.org
Sent: Friday, December 2, 2016 4:28:07 PM
Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite
+1 doubling the testsuite size/time is not practical. Also, there is the
issue that if all tests are duplicated, when people come to enhance an
existing test they will need to do so in two places.
Could we not do what was suggested earlier and pull out tests which obviously
target security into two legacy/elytron modules, and leave the rest where
they are? Then if something later turns out to have 'hidden' implications,
that test gets pulled out into the two modules.
Regarding the deprecated legacy stuff, I think we will have to keep that
around for many years still due to it being used by the product (and in any
case in that branch, although removing it from WF might be possible sooner).