Re: [wildfly-dev] Elytron integration tests in WildFly testsuite

From: "Darran Lofthouse" <darran.lofthouse(a)jboss.com&gt; To: wildfly-dev(a)lists.jboss.org Sent: Monday, December 5, 2016 5:28:02 PM Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite Regarding that 25% failure - I would expect a lot of that to be picked up anyway as we merge the *-elytron.xml configurations into the default configurations and then drop the *-elytron configurations. How about my reply to Stuart's e-mail and a 3-way split, legacy-security, security, and basic where basic can have optional profiles added to test different extension availability? We could then do as Kabir suggests and pull out additional tests later if we do find a reason to justify duplicating the test and in the meantime have the profiles available to run the tests. Regards, Darran Lofthouse. On 05/12/16 15:03, Josef Cacek wrote: > Hi, > > We still see the copying whole modules as the best way. Let's recap our > view. > > Tested components may use a security feature internally and without test > coverage we would not realize that the feature stopped working. Running > all tests will help us to discover regressions before they occur in > application server. > > Pros: > - we don't lose testcoverage, we can simply detect regressions. It also > provides possible early failure detection for dev. > - it would be useful for migration, it shows migration path - user can take > a look on tests with legacy solution and compare their configuration with > tests which use Elytron configuration > - by each testsuite run we see how many tests are skipped (i.e. TODO list > for Elytron integration) > - it should be simple to reconfigure the project (copy module; update > metadata in pom.xml; ignore failing tests) > - it's simple to remove legacy testsuite modules once the legacy security > subsystem is removed from WildFly > > Cons and their solutions: > - double fixing of tests - we don't expect many such cases, the legacy > should be handled as frozen (rewriting/adding only on need-to-have basis). > Only custormer ticket related tests could be backported, it means only > unit of tests per months. > - testsuite execution time - we don't need to run the legacy modules with > each PR, it can be resolved by using profiles > > As a smoke test we tried to run testsuite/integration-tests/basic module > with standalone-elytron.xml and standalone-full-elytron.xml (merged > standalone-full.xml and standalone-elytron.xml). There is around 25% test > failures/errors. Also some testcases, which seem that are not directly > related to security, failed. > > Thanks, > > -- Josef > > > ----- Original Message ----- >> From: "Kabir Khan" <kabir.khan(a)jboss.com&gt; >> To: "Brian Stansberry" <brian.stansberry(a)redhat.com&gt; >> Cc: wildfly-dev(a)lists.jboss.org >> Sent: Friday, December 2, 2016 4:28:07 PM >> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite >> >> +1 doubling the testsuite size/time is not practical. Also, there is the >> issue that if all tests are duplicated, when people come to enhance an >> existing test they will need to do so in two places. >> >> Could we not do what was suggested earlier and pull out tests which >> obviously >> target security into two legacy/elytron modules, and leave the rest where >> they are? Then if something later turns out to have 'hidden' implications, >> that test gets pulled out into the two modules. >> >> Regarding the deprecated legacy stuff, I think we will have to keep that >> around for many years still due to it being used by the product (and in >> any >> case in that branch, although removing it from WF might be possible >> sooner). > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/wildfly-dev > _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev