<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Dec 6, 2016 at 2:03 AM, Josef Cacek <span dir="ltr"><<a href="mailto:jcacek@redhat.com" target="_blank">jcacek@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
We still see the copying whole modules as the best way. Let's recap our view.<br>
<br>
Tested components may use a security feature internally and without test coverage we would not realize that the feature stopped working. Running all tests will help us to discover regressions before they occur in application server.<br></blockquote><div><br></div><div>They don't though. If the test is not set up to use security then security won't be applied. The idea of adding an extra 900 test cases 'just in case' is massive overkill, it adds a massive amount of technical debt and decreases the overall quality of the test suite for very little gain.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Pros:<br>
- we don't lose testcoverage, we can simply detect regressions. It also provides possible early failure detection for dev.<br>
- it would be useful for migration, it shows migration path - user can take a look on tests with legacy solution and compare their configuration with tests which use Elytron configuration<br></blockquote><div><br></div><div>This is only applicable for tests that actually use Elytron. Just copying the relevant tests has the same effect.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
- by each testsuite run we see how many tests are skipped (i.e. TODO list for Elytron integration)<br></blockquote><div><br></div><div>Once again, also accomplished by just copying Elytron tests.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
- it should be simple to reconfigure the project (copy module; update metadata in pom.xml; ignore failing tests)<br></blockquote><div><br></div>Once again, also accomplished by just copying Elytron tests. Even though it is convenient at first IMHO the duplication of tests will greatly decrease test suite maintainability, so overall I think this approach will use a lot more man hours.<br><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
- it's simple to remove legacy testsuite modules once the legacy security subsystem is removed from WildFly<br>
<br>
Cons and their solutions:<br>
- double fixing of tests - we don't expect many such cases, the legacy should be handled as frozen (rewriting/adding only on need-to-have basis). Only custormer ticket related tests could be backported, it means only unit of tests per months.<br></blockquote><div><br></div><div>This is a major problem. We should not have to write a new test every time, whenever possible existing tests should be enhanced (assuming similar tests for similar functionality already exist). This has many advantages:<br><br></div><div>- The tests will execute faster, as deployment and set up is only performed once<br></div><div>- It is very clear exactly what functionality is being tested, as all the tests are in a single test case<br></div><div>- It reduces the number of classes in the test suite, which makes compile times faster and increases maintainability<br><br></div><div>Freezing all these tests is not a practical idea IMHO.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
- testsuite execution time - we don't need to run the legacy modules with each PR, it can be resolved by using profiles<br></blockquote><div><br></div><div>Yes we do. If a test is not being run with every PR it might as well not be there, as it means that is can be broken without anyone knowing. <br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
As a smoke test we tried to run testsuite/integration-tests/<wbr>basic module with standalone-elytron.xml and standalone-full-elytron.xml (merged standalone-full.xml and standalone-elytron.xml). There is around 25% test failures/errors. Also some testcases, which seem that are not directly related to security, failed.<br></blockquote><div><br></div><div>This does not really mean much at the moment, it is most likely due to configuration issues. Once the default config contains Elytron all these tests will be running with the Elytron config anyway. <br></div><div><br></div><div>Stuart<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Thanks,<br>
<span class="gmail-im gmail-HOEnZb"><br>
-- Josef<br>
<br>
<br>
----- Original Message -----<br>
</span><span class="gmail-im gmail-HOEnZb">> From: "Kabir Khan" <<a href="mailto:kabir.khan@jboss.com">kabir.khan@jboss.com</a>><br>
> To: "Brian Stansberry" <<a href="mailto:brian.stansberry@redhat.com">brian.stansberry@redhat.com</a>><br>
> Cc: <a href="mailto:wildfly-dev@lists.jboss.org">wildfly-dev@lists.jboss.org</a><br>
> Sent: Friday, December 2, 2016 4:28:07 PM<br>
> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite<br>
><br>
</span><span class="gmail-im gmail-HOEnZb">> +1 doubling the testsuite size/time is not practical. Also, there is the<br>
> issue that if all tests are duplicated, when people come to enhance an<br>
> existing test they will need to do so in two places.<br>
><br>
> Could we not do what was suggested earlier and pull out tests which obviously<br>
> target security into two legacy/elytron modules, and leave the rest where<br>
> they are? Then if something later turns out to have 'hidden' implications,<br>
> that test gets pulled out into the two modules.<br>
><br>
> Regarding the deprecated legacy stuff, I think we will have to keep that<br>
> around for many years still due to it being used by the product (and in any<br>
> case in that branch, although removing it from WF might be possible sooner).<br>
</span><div class="gmail-HOEnZb"><div class="gmail-h5">______________________________<wbr>_________________<br>
wildfly-dev mailing list<br>
<a href="mailto:wildfly-dev@lists.jboss.org">wildfly-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/wildfly-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/wildfly-dev</a><br>
</div></div></blockquote></div><br></div></div>