[keycloak-dev] Testing with mocking libraries?

[Anil Saldanha]

Agree. I had forgotten about Arquillian.

> On May 31, 2018, at 2:09 PM, Stian Thorgersen <sthorger at redhat.com> wrote:
> 
> 
> 
>> On 31 May 2018 at 20:48, Anil Saldanha <asaldanha1947 at gmail.com> wrote:
>> You need both sets of testing to maintain the long term quality of a security project like Keycloak. :-)
> 
> By using Arquillian we are actually able to test Keycloak very extensively without the need for Mocking through functional testing. I would suggest you review how extensive coverage we already have completely without any use of mocking and almost no basic unit testing at all.
> 
> The problem I see is really only down to the difficulty of writing integration testing in some cases, especially when there are many configuration options. That's where it may be useful to mock the system being integrated with, but through mocking http calls, not Java method calls.
>  
>> 
>> I agree on the mocked IDP approach being more effective.
>> 
>>> On Thu, May 31, 2018 at 1:20 PM, Stian Thorgersen <sthorger at redhat.com> wrote:
>>> With regards to Identity Brokering and Social Login one potentially better way to do it would be to have a mocked IdP. At least then you are testing the actual HTTP calls and it's much closer to a real scenario than doing junit tests with mocks and only testing parts of the code base.
>>> 
>>>> On 31 May 2018 at 18:08, Anil Saldanha <asaldanha1947 at gmail.com> wrote:
>>>> I am sorry I have not looked at your integration tests.  So I may be wrong here. <namaste/>
>>>> 
>>>> One area that I feel that you need lots of integration tests is in the "Identity Brokering" feature.  You have way too many permutations and combinations (with authentication flows) that without proper integration tests/mocks, you will not do proper justice to the value of KC/RH-SSO.
>>>> 
>>>>> On Thu, May 31, 2018 at 10:17 AM, Bill Burke <bburke at redhat.com> wrote:
>>>>> Never liked mocks either as they are just opinionated warped
>>>>> perceptions of reality.   But there are cases where I think we need
>>>>> them off the top of my head:
>>>>> 
>>>>> * Openshift integration
>>>>> * Social logins.
>>>>> 
>>>>> Openshift we just won't have the infrastructure for it.  Social login
>>>>> tests require accounts which we won't want to share the logins for in
>>>>> a public CI.  We need something in public CI so that we can catch at
>>>>> least some potential problems and that we can easily try to reproduce
>>>>> in local IDE environments.  Mocks wouldn't be a replacement for real
>>>>> integration testing, just a stop gap.
>>>>> 
>>>>> 
>>>>> 
>>>>> On Thu, May 31, 2018 at 10:44 AM, Anil Saldanha <asaldanha1947 at gmail.com> wrote:
>>>>> > You really have to test your core logic (with no server set up) using
>>>>> > integration tests with mocking libraries. One way to test out all the
>>>>> > possible preconditions to your logic.
>>>>> >
>>>>> > You will be doing a service to your customers and users by shielding them
>>>>> > from those nasty exceptions (NPEs) that happen in an integration platform
>>>>> > such as KeyCloak/RH-SSO. :-)
>>>>> >
>>>>> > /me wearing a user/customer hat
>>>>> >
>>>>> > On Thu, May 31, 2018 at 1:38 AM, Marek Posolda <mposolda at redhat.com> wrote:
>>>>> >
>>>>> >> Not sure. I am not strongly against it, but I afraid that if we
>>>>> >> introduce some mocks support, people (especially from community) will
>>>>> >> start to test with mocks everywhere and add them to all PRs (even to
>>>>> >> those which could be easily tested properly). Not sure if it's better
>>>>> >> alternative to skip automated test at all instead of test with mocks?
>>>>> >>
>>>>> >> But for this one, I guess we can likely extend our social test with an
>>>>> >> account from hosted domain and hence test it properly?
>>>>> >>
>>>>> >> Marek
>>>>> >>
>>>>> >>
>>>>> >> On 30/05/18 09:19, Stian Thorgersen wrote:
>>>>> >> > At the moment our testing strategy is only with functional or integration
>>>>> >> > level tests. Both with the full server up and running and primarily
>>>>> >> testing
>>>>> >> > through public APIs.
>>>>> >> >
>>>>> >> > Now here comes the question should we also allow testing through a
>>>>> >> mocking
>>>>> >> > library like Mockito?
>>>>> >> >
>>>>> >> > In general I'm against mocking libraries. At best you end up with
>>>>> >> something
>>>>> >> > that may work, but you're not guaranteed that it actually works. They
>>>>> >> also
>>>>> >> > have a very big maintenance cost when any changes are made to the
>>>>> >> codebase.
>>>>> >> >
>>>>> >> > However, take a look at https://github.com/keycloak/keycloak/pull/5215
>>>>> >> for
>>>>> >> > example. It is a contribution to add support for the hd param to the
>>>>> >> Google
>>>>> >> > login. Not sure how else we could test this without a mocking library.
>>>>> >> > _______________________________________________
>>>>> >> > keycloak-dev mailing list
>>>>> >> > keycloak-dev at lists.jboss.org
>>>>> >> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>> >>
>>>>> >>
>>>>> >> _______________________________________________
>>>>> >> keycloak-dev mailing list
>>>>> >> keycloak-dev at lists.jboss.org
>>>>> >> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>> >>
>>>>> > _______________________________________________
>>>>> > keycloak-dev mailing list
>>>>> > keycloak-dev at lists.jboss.org
>>>>> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> Bill Burke
>>>>> Red Hat
>>>> 
>>> 
>> 
>