[wildfly-dev] Elytron integration tests in WildFly testsuite

Wed Dec 7 07:59:19 EST 2016

Ok, we will go for this approach.

We will create a new testsuite module 'legacy-security', we will copy 
the 50 or so tests that are easily identifiable as security tests into 
this new module - as these are testing legacy security the only real 
changes they should require will be to re-enable legacy security.

Remaining tests will be executed against configuration that favours 
Elytron over legacy.  Security tests in this area will be cleaned up and 
then we can start adding new security tests.

We will monitor other tests and if security related regressions are 
identified we will review on a case by case basis and if appropriate 
fork into the legacy-security module.

Regards,
Darran Lofthouse.

On 06/12/16 12:08, Darran Lofthouse wrote:
> As I think about this I am heading back to just a two way split.
>
> If we go for the two way split, lets say the following about the default
> WildFly configuration: -
>
>  - It will be entirely backed by the Elytron subsystem.
>  - Legacy security-realms will be removed.
>  - The legacy security subsystem will be removed.
>  - The legacy security extension will be removed.
>
> Existing users forward porting their configuration are unaffected as
> this only affects default new configuration.
>
> We can offer documentation on how to re-enable those items where really
> needed - anyone doing so hopefully submitting a Jira letting us know why
> they are doing so.
>
> This will mean the 900 "non-security" tests will certainly be verified
> that they run without the legacy security extension being installed.
>
> We then have a new legacy-security module where we add at the very least
> the legacy security extension back in which we copy the 50 or so
> security tests over.
>
> The security specific tests left behind are all cleaned up and ported to
> pure Elytron based testing.
>
> Anything that breaks as we switch the configuration over in the
> remaining 900 tests we will review on a case by case basis, also we will
> keep an eye on other subsystems we have touched as they may want a
> legacy test adding e.g. messaging is one we have been working on this week.
>
> Regards,
> Darran Lofthouse.
>
>
> On 06/12/16 07:40, Josef Cacek wrote:
>> The 3-way split is a good idea, but it would only work if
>> - we are sure the non-security tests in shared modules stay really
>> independent - i.e. no new Ignores, Assumes, configuration based
>> conditions;
>> - all shared modules have profiles for running with legacy security.
>>
>> It's not clear how many new modules would be necessary with the 3 way
>> split. Current TS modules don't share server configuration and they
>> (may) contain several surefire executions with different server profiles.
>>
>> -- Josef
>>
>> ----- Original Message -----
>>> From: "Darran Lofthouse" <darran.lofthouse at jboss.com>
>>> To: wildfly-dev at lists.jboss.org
>>> Sent: Monday, December 5, 2016 5:28:02 PM
>>> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly
>>> testsuite
>>>
>>> Regarding that 25% failure - I would expect a lot of that to be picked
>>> up anyway as we merge the *-elytron.xml configurations into the default
>>> configurations and then drop the *-elytron configurations.
>>>
>>> How about my reply to Stuart's e-mail and a 3-way split,
>>> legacy-security, security, and basic where basic can have optional
>>> profiles added to test different extension availability?
>>>
>>> We could then do as Kabir suggests and pull out additional tests later
>>> if we do find a reason to justify duplicating the test and in the
>>> meantime have the profiles available to run the tests.
>>>
>>> Regards,
>>> Darran Lofthouse.
>>>
>>> On 05/12/16 15:03, Josef Cacek wrote:
>>>> Hi,
>>>>
>>>> We still see the copying whole modules as the best way. Let's recap our
>>>> view.
>>>>
>>>> Tested components may use a security feature internally and without
>>>> test
>>>> coverage we would not realize that the feature stopped working. Running
>>>> all tests will help us to discover regressions before they occur in
>>>> application server.
>>>>
>>>> Pros:
>>>> - we don't lose testcoverage, we can simply detect regressions. It also
>>>> provides possible early failure detection for dev.
>>>> - it would be useful for migration, it shows migration path - user
>>>> can take
>>>> a look on tests with legacy solution and compare their configuration
>>>> with
>>>> tests which use Elytron configuration
>>>> - by each testsuite run we see how many tests are skipped (i.e. TODO
>>>> list
>>>> for Elytron integration)
>>>> - it should be simple to reconfigure the project (copy module; update
>>>> metadata in pom.xml; ignore failing tests)
>>>> - it's simple to remove legacy testsuite modules once the legacy
>>>> security
>>>> subsystem is removed from WildFly
>>>>
>>>> Cons and their solutions:
>>>> - double fixing of tests - we don't expect many such cases, the legacy
>>>> should be handled as frozen (rewriting/adding only on need-to-have
>>>> basis).
>>>> Only custormer ticket related tests could be backported, it means only
>>>> unit of tests per months.
>>>> - testsuite execution time - we don't need to run the legacy modules
>>>> with
>>>> each PR, it can be resolved by using profiles
>>>>
>>>> As a smoke test we tried to run testsuite/integration-tests/basic
>>>> module
>>>> with standalone-elytron.xml and standalone-full-elytron.xml (merged
>>>> standalone-full.xml and standalone-elytron.xml). There is around 25%
>>>> test
>>>> failures/errors. Also some testcases, which seem that are not directly
>>>> related to security, failed.
>>>>
>>>> Thanks,
>>>>
>>>> -- Josef
>>>>
>>>>
>>>> ----- Original Message -----
>>>>> From: "Kabir Khan" <kabir.khan at jboss.com>
>>>>> To: "Brian Stansberry" <brian.stansberry at redhat.com>
>>>>> Cc: wildfly-dev at lists.jboss.org
>>>>> Sent: Friday, December 2, 2016 4:28:07 PM
>>>>> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly
>>>>> testsuite
>>>>>
>>>>> +1 doubling the testsuite size/time is not practical. Also, there
>>>>> is the
>>>>> issue that if all tests are duplicated, when people come to enhance an
>>>>> existing test they will need to do so in two places.
>>>>>
>>>>> Could we not do what was suggested earlier and pull out tests which
>>>>> obviously
>>>>> target security into two legacy/elytron modules, and leave the rest
>>>>> where
>>>>> they are? Then if something later turns out to have 'hidden'
>>>>> implications,
>>>>> that test gets pulled out into the two modules.
>>>>>
>>>>> Regarding the deprecated legacy stuff, I think we will have to keep
>>>>> that
>>>>> around for many years still due to it being used by the product
>>>>> (and in
>>>>> any
>>>>> case in that branch, although removing it from WF might be possible
>>>>> sooner).
>>>> _______________________________________________
>>>> wildfly-dev mailing list
>>>> wildfly-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>>
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> wildfly-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>