Re: [keycloak-dev] Testing with mocking libraries?

Agree. I had forgotten about Arquillian. On May 31, 2018, at 2:09 PM, Stian Thorgersen <sthorger(a)redhat.com <mailto:sthorger@redhat.com>> wrote: > > > On 31 May 2018 at 20:48, Anil Saldanha <asaldanha1947(a)gmail.com > <mailto:asaldanha1947@gmail.com>> wrote: > > You need both sets of testing to maintain the long term quality > of a security project like Keycloak. :-) > > > By using Arquillian we are actually able to test Keycloak very > extensively without the need for Mocking through functional testing. > I would suggest you review how extensive coverage we already have > completely without any use of mocking and almost no basic unit > testing at all. > > The problem I see is really only down to the difficulty of writing > integration testing in some cases, especially when there are many > configuration options. That's where it may be useful to mock the > system being integrated with, but through mocking http calls, not > Java method calls. > > > I agree on the mocked IDP approach being more effective. > > On Thu, May 31, 2018 at 1:20 PM, Stian Thorgersen > <sthorger(a)redhat.com <mailto:sthorger@redhat.com>> wrote: > > With regards to Identity Brokering and Social Login one > potentially better way to do it would be to have a mocked > IdP. At least then you are testing the actual HTTP calls and > it's much closer to a real scenario than doing junit tests > with mocks and only testing parts of the code base. > > On 31 May 2018 at 18:08, Anil Saldanha > <asaldanha1947(a)gmail.com <mailto:asaldanha1947@gmail.com>> wrote: > > I am sorry I have not looked at your integration tests.  > So I may be wrong here. <namaste/> > > One area that I feel that you need lots of integration > tests is in the "Identity Brokering" feature.  You have > way too many permutations and combinations (with > authentication flows) that without proper integration > tests/mocks, you will not do proper justice to the value > of KC/RH-SSO. > > On Thu, May 31, 2018 at 10:17 AM, Bill Burke > <bburke(a)redhat.com <mailto:bburke@redhat.com>> wrote: > > Never liked mocks either as they are just opinionated > warped > perceptions of reality.  But there are cases where I > think we need > them off the top of my head: > > * Openshift integration > * Social logins. > > Openshift we just won't have the infrastructure for > it.  Social login > tests require accounts which we won't want to share > the logins for in > a public CI.  We need something in public CI so that > we can catch at > least some potential problems and that we can easily > try to reproduce > in local IDE environments.  Mocks wouldn't be a > replacement for real > integration testing, just a stop gap. > > > > On Thu, May 31, 2018 at 10:44 AM, Anil Saldanha > <asaldanha1947(a)gmail.com > <mailto:asaldanha1947@gmail.com>> wrote: > > You really have to test your core logic (with no > server set up) using > > integration tests with mocking libraries. One way > to test out all the > > possible preconditions to your logic. > > > > You will be doing a service to your customers and > users by shielding them > > from those nasty exceptions (NPEs) that happen in > an integration platform > > such as KeyCloak/RH-SSO. :-) > > > > /me wearing a user/customer hat > > > > On Thu, May 31, 2018 at 1:38 AM, Marek Posolda > <mposolda(a)redhat.com <mailto:mposolda@redhat.com>> wrote: > > > >> Not sure. I am not strongly against it, but I > afraid that if we > >> introduce some mocks support, people (especially > from community) will > >> start to test with mocks everywhere and add them > to all PRs (even to > >> those which could be easily tested properly). Not > sure if it's better > >> alternative to skip automated test at all instead > of test with mocks? > >> > >> But for this one, I guess we can likely extend our > social test with an > >> account from hosted domain and hence test it properly? > >> > >> Marek > >> > >> > >> On 30/05/18 09:19, Stian Thorgersen wrote: > >> > At the moment our testing strategy is only with > functional or integration > >> > level tests. Both with the full server up and > running and primarily > >> testing > >> > through public APIs. > >> > > >> > Now here comes the question should we also allow > testing through a > >> mocking > >> > library like Mockito? > >> > > >> > In general I'm against mocking libraries. At > best you end up with > >> something > >> > that may work, but you're not guaranteed that it > actually works. They > >> also > >> > have a very big maintenance cost when any > changes are made to the > >> codebase. > >> > > >> > However, take a look at > https://github.com/keycloak/keycloak/pull/5215 > <https://github.com/keycloak/keycloak/pull/5215> > >> for > >> > example. It is a contribution to add support for > the hd param to the > >> Google > >> > login. Not sure how else we could test this > without a mocking library. > >> > _______________________________________________ > >> > keycloak-dev mailing list > >> > keycloak-dev(a)lists.jboss.org > <mailto:keycloak-dev@lists.jboss.org> > >> > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > <https://lists.jboss.org/mailman/listinfo/keycloak-dev> > >> > >> > >> _______________________________________________ > >> keycloak-dev mailing list > >> keycloak-dev(a)lists.jboss.org > <mailto:keycloak-dev@lists.jboss.org> > >> > https://lists.jboss.org/mailman/listinfo/keycloak-dev > <https://lists.jboss.org/mailman/listinfo/keycloak-dev> > >> > > _______________________________________________ > > keycloak-dev mailing list > > keycloak-dev(a)lists.jboss.org > <mailto:keycloak-dev@lists.jboss.org> > > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > <https://lists.jboss.org/mailman/listinfo/keycloak-dev> > > > > -- > Bill Burke > Red Hat > > > > >