[wildfly-dev] Elytron integration tests in WildFly testsuite

Kabir Khan kabir.khan at jboss.com
Fri Dec 2 10:28:07 EST 2016 

+1 doubling the testsuite size/time is not practical. Also, there is the issue that if all tests are duplicated, when people come to enhance an existing test they will need to do so in two places.

Could we not do what was suggested earlier and pull out tests which obviously target security into two legacy/elytron modules, and leave the rest where they are? Then if something later turns out to have 'hidden' implications, that test gets pulled out into the two modules. 

Regarding the deprecated legacy stuff, I think we will have to keep that around for many years still due to it being used by the product (and in any case in that branch, although removing it from WF might be possible sooner).


> On 2 Dec 2016, at 15:20, Brian Stansberry <brian.stansberry at redhat.com> wrote:
> 
> Duplicating the whole testsuite is basically a minimal effort way of getting coverage by avoiding the task of test coverage analysis.
> 
> Only justification for that is expediency. That may be sufficient justification; I don’t know.
> 
> But assuming for the sake of argument it’s sufficient, doubling the amount of time needed to run the testsuite is a massive productivity loss and that means cutting that time down to the necessary minimum and nothing more would need to be a critical extreme high priority task.
> 
> So, with this proposal, who would be the people assigned to this critical extreme high priority task? I’d like to see names, lots and lots of names. Darran Lofthouse not among them. ;)
> 
> 
>> On Dec 2, 2016, at 8:59 AM, Darran Lofthouse <darran.lofthouse at jboss.com> wrote:
>> 
>> No the part we can't avoid is that we have two security implementations 
>> in the server and we have to have tests for both.  The e-mail from Josef 
>> is a suggestion as to how this will be achieved.
>> 
>> Other suggestions are welcome ;-)
>> 
>> But we have two security implementations to test.
>> 
>> One of these is deprecated and will hopefully be removed at some point 
>> so ideally we have something that allows testing to legacy to be removed 
>> without too much pain.
>> 
>> Writing sufficient test coverage is not a small task so anything that 
>> gives us more than starting from scratch is a benefit.
>> 
>> But maybe it would be worth summarising these additional scenarios that 
>> will be affected by changes.
>> 
>> Regards,
>> Darran Lofthouse.
>> 
>> On 02/12/16 14:47, Tomaž Cerar wrote:
>>> So if I understood all this correctly,
>>> this mail here was more of a heads up what is happening
>>> than a discussion on how could or should be best done.
>>> 
>>> I just hope you (the team behind decisions) considered all scenarios how
>>> will that impact everyone involved.
>>> 
>>> --
>>> tomaz
>>> 
>>> On Fri, Dec 2, 2016 at 12:58 PM, Josef Cacek <jcacek at redhat.com
>>> <mailto:jcacek at redhat.com>> wrote:
>>> 
>>>   We prefer duplication of all tests in the testsuite to keep test
>>>   coverage.
>>> 
>>>   To be sure we don't have regressions with introducing Elytron and at
>>>   the same time we are able to cover current features with Elytron, we
>>>   must run all tests with both security subsystems.
>>>   Even if a test is not security related on the first glance, the
>>>   tested component may use a security feature internally. By switching
>>>   to the new security subsystem the feature may stop work.
>>> 
>>>   Let's keep the current (legacy security based) tests alive until the
>>>   legacy security subsystem is fully removed.
>>>   Then we'll simply remove the *-legacy-security testsuite modules.
>>> 
>>>   -- Josef
>>> 
>>>   ----- Original Message -----
>>>> From: "Tomaž Cerar" <tomaz.cerar at gmail.com
>>>   <mailto:tomaz.cerar at gmail.com>>
>>>> To: "Darran Lofthouse" <darran.lofthouse at jboss.com
>>>   <mailto:darran.lofthouse at jboss.com>>
>>>> Cc: wildfly-dev at lists.jboss.org <mailto:wildfly-dev at lists.jboss.org>
>>>> Sent: Friday, December 2, 2016 12:37:22 PM
>>>> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly
>>>   testsuite
>>>> 
>>>> That is probably fine, but! it should be done differently.
>>>> 
>>>> instead of duplicating whole testsuite (and adding extra hour to
>>>   execution
>>>> and extra headaches with intermittent problems and duplication of
>>>> maintenance)
>>>> I would suggest that all security related tests get extracted to new
>>>> "security" testsuite module and than only that part is duplicated.
>>>> 
>>>> This way we will have all security related stuff in one place.
>>>> 
>>>> 
>>>> 
>>>> On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse <
>>>> darran.lofthouse at jboss.com <mailto:darran.lofthouse at jboss.com> >
>>>   wrote:
>>>> 
>>>> 
>>>> Probably should add - any duplication should only be for security
>>>   tests
>>>> - not everything else in there!
>>>> 
>>>> On 02/12/16 11:08, Darran Lofthouse wrote:
>>>>> On 02/12/16 11:03, Tomaž Cerar wrote:
>>>>>> 
>>>>>> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek < jcacek at redhat.com
>>>   <mailto:jcacek at redhat.com>
>>>>>> <mailto: jcacek at redhat.com <mailto:jcacek at redhat.com> >> wrote:
>>>>>> 
>>>>>> The modules would just live side by side - basic would use Elytron
>>>>>> configuration, basic-legacy-security would use configuration
>>>   similar
>>>>>> to (or same as) the current server configuration.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> What would this actually mean?
>>>>>> we will have two copies basic tests suites one running with elytron
>>>>>> another with legacy security subsystem?
>>>>>> 
>>>>>> Do I read that right? Please say I am not.
>>>>> 
>>>>> That is correct - we have two security implementations they both
>>>   need
>>>>> testing.
>>>>> 
>>>>> One needs testing for backwards compatibility and regressions,
>>>   the other
>>>>> for equivalent behaviour and then new features and bugs.
>>>>> 
>>>>> Needing to test both was discussed previously so this is more
>>>   about how
>>>>> to separate both and also give the Elytron testing a good
>>>   foundation to
>>>>> start from.
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> wildfly-dev mailing list
>>>>>> wildfly-dev at lists.jboss.org <mailto:wildfly-dev at lists.jboss.org>
>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>   <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>>>>>> 
>>>>> 
>>>> 
>>>> _______________________________________________
>>>> wildfly-dev mailing list
>>>> wildfly-dev at lists.jboss.org <mailto:wildfly-dev at lists.jboss.org>
>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>   <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>>>> 
>>>> 
>>>> _______________________________________________
>>>> wildfly-dev mailing list
>>>> wildfly-dev at lists.jboss.org <mailto:wildfly-dev at lists.jboss.org>
>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>   <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>>> 
>>> 
>> 
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
> 
> -- 
> Brian Stansberry
> Manager, Senior Principal Software Engineer
> JBoss by Red Hat
> 
> 
> 
> 
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev

More information about the wildfly-dev mailing list