2:19 a.m.
At the moment our testing strategy is only with functional or integration
level tests. Both with the full server up and running and primarily testing
through public APIs.
Now here comes the question should we also allow testing through a mocking
library like Mockito?
In general I'm against mocking libraries. At best you end up with something
that may work, but you're not guaranteed that it actually works. They also
have a very big maintenance cost when any changes are made to the codebase.
However, take a look at https://github.com/keycloak/keycloak/pull/5215 for
example. It is a contribution to add support for the hd param to the Google
login. Not sure how else we could test this without a mocking library.
1:38 a.m.
Not sure. I am not strongly against it, but I afraid that if we
introduce some mocks support, people (especially from community) will
start to test with mocks everywhere and add them to all PRs (even to
those which could be easily tested properly). Not sure if it's better
alternative to skip automated test at all instead of test with mocks?
But for this one, I guess we can likely extend our social test with an
account from hosted domain and hence test it properly?
Marek
On 30/05/18 09:19, Stian Thorgersen wrote:
At the moment our testing strategy is only with functional or
integration
level tests. Both with the full server up and running and primarily testing
through public APIs.
Now here comes the question should we also allow testing through a mocking
library like Mockito?
In general I'm against mocking libraries. At best you end up with something
that may work, but you're not guaranteed that it actually works. They also
have a very big maintenance cost when any changes are made to the codebase.
However, take a look at https://github.com/keycloak/keycloak/pull/5215 for
example. It is a contribution to add support for the hd param to the Google
login. Not sure how else we could test this without a mocking library.
_______________________________________________
keycloak-dev mailing list
keycloak-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-dev
9:44 a.m.
You really have to test your core logic (with no server set up) using
integration tests with mocking libraries. One way to test out all the
possible preconditions to your logic.
You will be doing a service to your customers and users by shielding them
from those nasty exceptions (NPEs) that happen in an integration platform
such as KeyCloak/RH-SSO. :-)
/me wearing a user/customer hat
On Thu, May 31, 2018 at 1:38 AM, Marek Posolda <mposolda(a)redhat.com> wrote:
Not sure. I am not strongly against it, but I afraid that if we
introduce some mocks support, people (especially from community) will
start to test with mocks everywhere and add them to all PRs (even to
those which could be easily tested properly). Not sure if it's better
alternative to skip automated test at all instead of test with mocks?
But for this one, I guess we can likely extend our social test with an
account from hosted domain and hence test it properly?
Marek
On 30/05/18 09:19, Stian Thorgersen wrote:
> At the moment our testing strategy is only with functional or integration
> level tests. Both with the full server up and running and primarily
testing
> through public APIs.
>
> Now here comes the question should we also allow testing through a
mocking
> library like Mockito?
>
> In general I'm against mocking libraries. At best you end up with
something
> that may work, but you're not guaranteed that it actually works. They
also
> have a very big maintenance cost when any changes are made to the
codebase.
>
> However, take a look at https://github.com/keycloak/keycloak/pull/5215
for
> example. It is a contribution to add support for the hd param to the
Google
> login. Not sure how else we could test this without a mocking library.
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
_______________________________________________
keycloak-dev mailing list
keycloak-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-dev
10:17 a.m.
Never liked mocks either as they are just opinionated warped
perceptions of reality. But there are cases where I think we need
them off the top of my head:
* Openshift integration
* Social logins.
Openshift we just won't have the infrastructure for it. Social login
tests require accounts which we won't want to share the logins for in
a public CI. We need something in public CI so that we can catch at
least some potential problems and that we can easily try to reproduce
in local IDE environments. Mocks wouldn't be a replacement for real
integration testing, just a stop gap.
On Thu, May 31, 2018 at 10:44 AM, Anil Saldanha <asaldanha1947(a)gmail.com> wrote:
You really have to test your core logic (with no server set up)
using
integration tests with mocking libraries. One way to test out all the
possible preconditions to your logic.
You will be doing a service to your customers and users by shielding them
from those nasty exceptions (NPEs) that happen in an integration platform
such as KeyCloak/RH-SSO. :-)
/me wearing a user/customer hat
On Thu, May 31, 2018 at 1:38 AM, Marek Posolda <mposolda(a)redhat.com> wrote:
> Not sure. I am not strongly against it, but I afraid that if we
> introduce some mocks support, people (especially from community) will
> start to test with mocks everywhere and add them to all PRs (even to
> those which could be easily tested properly). Not sure if it's better
> alternative to skip automated test at all instead of test with mocks?
>
> But for this one, I guess we can likely extend our social test with an
> account from hosted domain and hence test it properly?
>
> Marek
>
>
> On 30/05/18 09:19, Stian Thorgersen wrote:
> > At the moment our testing strategy is only with functional or integration
> > level tests. Both with the full server up and running and primarily
> testing
> > through public APIs.
> >
> > Now here comes the question should we also allow testing through a
> mocking
> > library like Mockito?
> >
> > In general I'm against mocking libraries. At best you end up with
> something
> > that may work, but you're not guaranteed that it actually works. They
> also
> > have a very big maintenance cost when any changes are made to the
> codebase.
> >
> > However, take a look at https://github.com/keycloak/keycloak/pull/5215
> for
> > example. It is a contribution to add support for the hd param to the
> Google
> > login. Not sure how else we could test this without a mocking library.
> > _______________________________________________
> > keycloak-dev mailing list
> > keycloak-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
>
>
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>
_______________________________________________
keycloak-dev mailing list
keycloak-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-dev
--
Bill Burke
Red Hat
11:08 a.m.
I am sorry I have not looked at your integration tests. So I may be wrong
here. <namaste/>
One area that I feel that you need lots of integration tests is in the
"Identity Brokering" feature. You have way too many permutations and
combinations (with authentication flows) that without proper integration
tests/mocks, you will not do proper justice to the value of KC/RH-SSO.
On Thu, May 31, 2018 at 10:17 AM, Bill Burke <bburke(a)redhat.com> wrote:
Never liked mocks either as they are just opinionated warped
perceptions of reality. But there are cases where I think we need
them off the top of my head:
* Openshift integration
* Social logins.
Openshift we just won't have the infrastructure for it. Social login
tests require accounts which we won't want to share the logins for in
a public CI. We need something in public CI so that we can catch at
least some potential problems and that we can easily try to reproduce
in local IDE environments. Mocks wouldn't be a replacement for real
integration testing, just a stop gap.
On Thu, May 31, 2018 at 10:44 AM, Anil Saldanha <asaldanha1947(a)gmail.com>
wrote:
> You really have to test your core logic (with no server set up) using
> integration tests with mocking libraries. One way to test out all the
> possible preconditions to your logic.
>
> You will be doing a service to your customers and users by shielding them
> from those nasty exceptions (NPEs) that happen in an integration platform
> such as KeyCloak/RH-SSO. :-)
>
> /me wearing a user/customer hat
>
> On Thu, May 31, 2018 at 1:38 AM, Marek Posolda <mposolda(a)redhat.com>
wrote:
>
>> Not sure. I am not strongly against it, but I afraid that if we
>> introduce some mocks support, people (especially from community) will
>> start to test with mocks everywhere and add them to all PRs (even to
>> those which could be easily tested properly). Not sure if it's better
>> alternative to skip automated test at all instead of test with mocks?
>>
>> But for this one, I guess we can likely extend our social test with an
>> account from hosted domain and hence test it properly?
>>
>> Marek
>>
>>
>> On 30/05/18 09:19, Stian Thorgersen wrote:
>> > At the moment our testing strategy is only with functional or
integration
>> > level tests. Both with the full server up and running and primarily
>> testing
>> > through public APIs.
>> >
>> > Now here comes the question should we also allow testing through a
>> mocking
>> > library like Mockito?
>> >
>> > In general I'm against mocking libraries. At best you end up with
>> something
>> > that may work, but you're not guaranteed that it actually works. They
>> also
>> > have a very big maintenance cost when any changes are made to the
>> codebase.
>> >
>> > However, take a look at https://github.com/keycloak/
keycloak/pull/5215
>> for
>> > example. It is a contribution to add support for the hd param to the
>> Google
>> > login. Not sure how else we could test this without a mocking library.
>> > _______________________________________________
>> > keycloak-dev mailing list
>> > keycloak-dev(a)lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>
>>
>> _______________________________________________
>> keycloak-dev mailing list
>> keycloak-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
--
Bill Burke
Red Hat
1:20 p.m.
With regards to Identity Brokering and Social Login one potentially better
way to do it would be to have a mocked IdP. At least then you are testing
the actual HTTP calls and it's much closer to a real scenario than doing
junit tests with mocks and only testing parts of the code base.
On 31 May 2018 at 18:08, Anil Saldanha <asaldanha1947(a)gmail.com> wrote:
I am sorry I have not looked at your integration tests. So I may be
wrong
here. <namaste/>
One area that I feel that you need lots of integration tests is in the
"Identity Brokering" feature. You have way too many permutations and
combinations (with authentication flows) that without proper integration
tests/mocks, you will not do proper justice to the value of KC/RH-SSO.
On Thu, May 31, 2018 at 10:17 AM, Bill Burke <bburke(a)redhat.com> wrote:
> Never liked mocks either as they are just opinionated warped
> perceptions of reality. But there are cases where I think we need
> them off the top of my head:
>
> * Openshift integration
> * Social logins.
>
> Openshift we just won't have the infrastructure for it. Social login
> tests require accounts which we won't want to share the logins for in
> a public CI. We need something in public CI so that we can catch at
> least some potential problems and that we can easily try to reproduce
> in local IDE environments. Mocks wouldn't be a replacement for real
> integration testing, just a stop gap.
>
>
>
> On Thu, May 31, 2018 at 10:44 AM, Anil Saldanha <asaldanha1947(a)gmail.com>
> wrote:
> > You really have to test your core logic (with no server set up) using
> > integration tests with mocking libraries. One way to test out all the
> > possible preconditions to your logic.
> >
> > You will be doing a service to your customers and users by shielding
> them
> > from those nasty exceptions (NPEs) that happen in an integration
> platform
> > such as KeyCloak/RH-SSO. :-)
> >
> > /me wearing a user/customer hat
> >
> > On Thu, May 31, 2018 at 1:38 AM, Marek Posolda <mposolda(a)redhat.com>
> wrote:
> >
> >> Not sure. I am not strongly against it, but I afraid that if we
> >> introduce some mocks support, people (especially from community) will
> >> start to test with mocks everywhere and add them to all PRs (even to
> >> those which could be easily tested properly). Not sure if it's better
> >> alternative to skip automated test at all instead of test with mocks?
> >>
> >> But for this one, I guess we can likely extend our social test with an
> >> account from hosted domain and hence test it properly?
> >>
> >> Marek
> >>
> >>
> >> On 30/05/18 09:19, Stian Thorgersen wrote:
> >> > At the moment our testing strategy is only with functional or
> integration
> >> > level tests. Both with the full server up and running and primarily
> >> testing
> >> > through public APIs.
> >> >
> >> > Now here comes the question should we also allow testing through a
> >> mocking
> >> > library like Mockito?
> >> >
> >> > In general I'm against mocking libraries. At best you end up with
> >> something
> >> > that may work, but you're not guaranteed that it actually works.
They
> >> also
> >> > have a very big maintenance cost when any changes are made to the
> >> codebase.
> >> >
> >> > However, take a look at https://github.com/keycloak/ke
> ycloak/pull/5215
> >> for
> >> > example. It is a contribution to add support for the hd param to the
> >> Google
> >> > login. Not sure how else we could test this without a mocking
> library.
> >> > _______________________________________________
> >> > keycloak-dev mailing list
> >> > keycloak-dev(a)lists.jboss.org
> >> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
> >>
> >>
> >> _______________________________________________
> >> keycloak-dev mailing list
> >> keycloak-dev(a)lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/keycloak-dev
> >>
> > _______________________________________________
> > keycloak-dev mailing list
> > keycloak-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
>
>
>
> --
> Bill Burke
> Red Hat
>
1:48 p.m.
You need both sets of testing to maintain the long term quality of a
security project like Keycloak. :-)
I agree on the mocked IDP approach being more effective.
On Thu, May 31, 2018 at 1:20 PM, Stian Thorgersen <sthorger(a)redhat.com>
wrote:
With regards to Identity Brokering and Social Login one potentially
better
way to do it would be to have a mocked IdP. At least then you are testing
the actual HTTP calls and it's much closer to a real scenario than doing
junit tests with mocks and only testing parts of the code base.
On 31 May 2018 at 18:08, Anil Saldanha <asaldanha1947(a)gmail.com> wrote:
> I am sorry I have not looked at your integration tests. So I may be
> wrong here. <namaste/>
>
> One area that I feel that you need lots of integration tests is in the
> "Identity Brokering" feature. You have way too many permutations and
> combinations (with authentication flows) that without proper integration
> tests/mocks, you will not do proper justice to the value of KC/RH-SSO.
>
> On Thu, May 31, 2018 at 10:17 AM, Bill Burke <bburke(a)redhat.com> wrote:
>
>> Never liked mocks either as they are just opinionated warped
>> perceptions of reality. But there are cases where I think we need
>> them off the top of my head:
>>
>> * Openshift integration
>> * Social logins.
>>
>> Openshift we just won't have the infrastructure for it. Social login
>> tests require accounts which we won't want to share the logins for in
>> a public CI. We need something in public CI so that we can catch at
>> least some potential problems and that we can easily try to reproduce
>> in local IDE environments. Mocks wouldn't be a replacement for real
>> integration testing, just a stop gap.
>>
>>
>>
>> On Thu, May 31, 2018 at 10:44 AM, Anil Saldanha <asaldanha1947(a)gmail.com>
>> wrote:
>> > You really have to test your core logic (with no server set up) using
>> > integration tests with mocking libraries. One way to test out all the
>> > possible preconditions to your logic.
>> >
>> > You will be doing a service to your customers and users by shielding
>> them
>> > from those nasty exceptions (NPEs) that happen in an integration
>> platform
>> > such as KeyCloak/RH-SSO. :-)
>> >
>> > /me wearing a user/customer hat
>> >
>> > On Thu, May 31, 2018 at 1:38 AM, Marek Posolda <mposolda(a)redhat.com>
>> wrote:
>> >
>> >> Not sure. I am not strongly against it, but I afraid that if we
>> >> introduce some mocks support, people (especially from community) will
>> >> start to test with mocks everywhere and add them to all PRs (even to
>> >> those which could be easily tested properly). Not sure if it's
better
>> >> alternative to skip automated test at all instead of test with mocks?
>> >>
>> >> But for this one, I guess we can likely extend our social test with an
>> >> account from hosted domain and hence test it properly?
>> >>
>> >> Marek
>> >>
>> >>
>> >> On 30/05/18 09:19, Stian Thorgersen wrote:
>> >> > At the moment our testing strategy is only with functional or
>> integration
>> >> > level tests. Both with the full server up and running and
primarily
>> >> testing
>> >> > through public APIs.
>> >> >
>> >> > Now here comes the question should we also allow testing through a
>> >> mocking
>> >> > library like Mockito?
>> >> >
>> >> > In general I'm against mocking libraries. At best you end up
with
>> >> something
>> >> > that may work, but you're not guaranteed that it actually
works.
>> They
>> >> also
>> >> > have a very big maintenance cost when any changes are made to the
>> >> codebase.
>> >> >
>> >> > However, take a look at https://github.com/keycloak/ke
>> ycloak/pull/5215
>> >> for
>> >> > example. It is a contribution to add support for the hd param to
the
>> >> Google
>> >> > login. Not sure how else we could test this without a mocking
>> library.
>> >> > _______________________________________________
>> >> > keycloak-dev mailing list
>> >> > keycloak-dev(a)lists.jboss.org
>> >> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
>> >>
>> >>
>> >> _______________________________________________
>> >> keycloak-dev mailing list
>> >> keycloak-dev(a)lists.jboss.org
>> >> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>> >>
>> > _______________________________________________
>> > keycloak-dev mailing list
>> > keycloak-dev(a)lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>
>>
>>
>> --
>> Bill Burke
>> Red Hat
>>
>
>
2:09 p.m.
On 31 May 2018 at 20:48, Anil Saldanha <asaldanha1947(a)gmail.com> wrote:
You need both sets of testing to maintain the long term quality of a
security project like Keycloak. :-)
By using Arquillian we are actually able to test Keycloak very extensively
without the need for Mocking through functional testing. I would suggest
you review how extensive coverage we already have completely without any
use of mocking and almost no basic unit testing at all.
The problem I see is really only down to the difficulty of writing
integration testing in some cases, especially when there are many
configuration options. That's where it may be useful to mock the system
being integrated with, but through mocking http calls, not Java method
calls.
I agree on the mocked IDP approach being more effective.
On Thu, May 31, 2018 at 1:20 PM, Stian Thorgersen <sthorger(a)redhat.com>
wrote:
> With regards to Identity Brokering and Social Login one potentially
> better way to do it would be to have a mocked IdP. At least then you are
> testing the actual HTTP calls and it's much closer to a real scenario than
> doing junit tests with mocks and only testing parts of the code base.
>
> On 31 May 2018 at 18:08, Anil Saldanha <asaldanha1947(a)gmail.com> wrote:
>
>> I am sorry I have not looked at your integration tests. So I may be
>> wrong here. <namaste/>
>>
>> One area that I feel that you need lots of integration tests is in the
>> "Identity Brokering" feature. You have way too many permutations and
>> combinations (with authentication flows) that without proper integration
>> tests/mocks, you will not do proper justice to the value of KC/RH-SSO.
>>
>> On Thu, May 31, 2018 at 10:17 AM, Bill Burke <bburke(a)redhat.com> wrote:
>>
>>> Never liked mocks either as they are just opinionated warped
>>> perceptions of reality. But there are cases where I think we need
>>> them off the top of my head:
>>>
>>> * Openshift integration
>>> * Social logins.
>>>
>>> Openshift we just won't have the infrastructure for it. Social login
>>> tests require accounts which we won't want to share the logins for in
>>> a public CI. We need something in public CI so that we can catch at
>>> least some potential problems and that we can easily try to reproduce
>>> in local IDE environments. Mocks wouldn't be a replacement for real
>>> integration testing, just a stop gap.
>>>
>>>
>>>
>>> On Thu, May 31, 2018 at 10:44 AM, Anil Saldanha <
>>> asaldanha1947(a)gmail.com> wrote:
>>> > You really have to test your core logic (with no server set up) using
>>> > integration tests with mocking libraries. One way to test out all the
>>> > possible preconditions to your logic.
>>> >
>>> > You will be doing a service to your customers and users by shielding
>>> them
>>> > from those nasty exceptions (NPEs) that happen in an integration
>>> platform
>>> > such as KeyCloak/RH-SSO. :-)
>>> >
>>> > /me wearing a user/customer hat
>>> >
>>> > On Thu, May 31, 2018 at 1:38 AM, Marek Posolda
<mposolda(a)redhat.com>
>>> wrote:
>>> >
>>> >> Not sure. I am not strongly against it, but I afraid that if we
>>> >> introduce some mocks support, people (especially from community)
will
>>> >> start to test with mocks everywhere and add them to all PRs (even
to
>>> >> those which could be easily tested properly). Not sure if it's
better
>>> >> alternative to skip automated test at all instead of test with
mocks?
>>> >>
>>> >> But for this one, I guess we can likely extend our social test with
>>> an
>>> >> account from hosted domain and hence test it properly?
>>> >>
>>> >> Marek
>>> >>
>>> >>
>>> >> On 30/05/18 09:19, Stian Thorgersen wrote:
>>> >> > At the moment our testing strategy is only with functional or
>>> integration
>>> >> > level tests. Both with the full server up and running and
primarily
>>> >> testing
>>> >> > through public APIs.
>>> >> >
>>> >> > Now here comes the question should we also allow testing
through a
>>> >> mocking
>>> >> > library like Mockito?
>>> >> >
>>> >> > In general I'm against mocking libraries. At best you end
up with
>>> >> something
>>> >> > that may work, but you're not guaranteed that it actually
works.
>>> They
>>> >> also
>>> >> > have a very big maintenance cost when any changes are made to
the
>>> >> codebase.
>>> >> >
>>> >> > However, take a look at https://github.com/keycloak/ke
>>> ycloak/pull/5215
>>> >> for
>>> >> > example. It is a contribution to add support for the hd param
to
>>> the
>>> >> Google
>>> >> > login. Not sure how else we could test this without a mocking
>>> library.
>>> >> > _______________________________________________
>>> >> > keycloak-dev mailing list
>>> >> > keycloak-dev(a)lists.jboss.org
>>> >> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> keycloak-dev mailing list
>>> >> keycloak-dev(a)lists.jboss.org
>>> >> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>> >>
>>> > _______________________________________________
>>> > keycloak-dev mailing list
>>> > keycloak-dev(a)lists.jboss.org
>>> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>
>>>
>>>
>>> --
>>> Bill Burke
>>> Red Hat
>>>
>>
>>
>
2:46 p.m.
Agree. I had forgotten about Arquillian.
On May 31, 2018, at 2:09 PM, Stian Thorgersen
<sthorger(a)redhat.com> wrote:
> On 31 May 2018 at 20:48, Anil Saldanha <asaldanha1947(a)gmail.com> wrote:
> You need both sets of testing to maintain the long term quality of a security project
like Keycloak. :-)
By using Arquillian we are actually able to test Keycloak very extensively without the
need for Mocking through functional testing. I would suggest you review how extensive
coverage we already have completely without any use of mocking and almost no basic unit
testing at all.
The problem I see is really only down to the difficulty of writing integration testing in
some cases, especially when there are many configuration options. That's where it may
be useful to mock the system being integrated with, but through mocking http calls, not
Java method calls.
>
> I agree on the mocked IDP approach being more effective.
>
>> On Thu, May 31, 2018 at 1:20 PM, Stian Thorgersen <sthorger(a)redhat.com>
wrote:
>> With regards to Identity Brokering and Social Login one potentially better way to
do it would be to have a mocked IdP. At least then you are testing the actual HTTP calls
and it's much closer to a real scenario than doing junit tests with mocks and only
testing parts of the code base.
>>
>>> On 31 May 2018 at 18:08, Anil Saldanha <asaldanha1947(a)gmail.com>
wrote:
>>> I am sorry I have not looked at your integration tests. So I may be wrong
here. <namaste/>
>>>
>>> One area that I feel that you need lots of integration tests is in the
"Identity Brokering" feature. You have way too many permutations and
combinations (with authentication flows) that without proper integration tests/mocks, you
will not do proper justice to the value of KC/RH-SSO.
>>>
>>>> On Thu, May 31, 2018 at 10:17 AM, Bill Burke <bburke(a)redhat.com>
wrote:
>>>> Never liked mocks either as they are just opinionated warped
>>>> perceptions of reality. But there are cases where I think we need
>>>> them off the top of my head:
>>>>
>>>> * Openshift integration
>>>> * Social logins.
>>>>
>>>> Openshift we just won't have the infrastructure for it. Social
login
>>>> tests require accounts which we won't want to share the logins for
in
>>>> a public CI. We need something in public CI so that we can catch at
>>>> least some potential problems and that we can easily try to reproduce
>>>> in local IDE environments. Mocks wouldn't be a replacement for real
>>>> integration testing, just a stop gap.
>>>>
>>>>
>>>>
>>>> On Thu, May 31, 2018 at 10:44 AM, Anil Saldanha
<asaldanha1947(a)gmail.com> wrote:
>>>> > You really have to test your core logic (with no server set up)
using
>>>> > integration tests with mocking libraries. One way to test out all
the
>>>> > possible preconditions to your logic.
>>>> >
>>>> > You will be doing a service to your customers and users by shielding
them
>>>> > from those nasty exceptions (NPEs) that happen in an integration
platform
>>>> > such as KeyCloak/RH-SSO. :-)
>>>> >
>>>> > /me wearing a user/customer hat
>>>> >
>>>> > On Thu, May 31, 2018 at 1:38 AM, Marek Posolda
<mposolda(a)redhat.com> wrote:
>>>> >
>>>> >> Not sure. I am not strongly against it, but I afraid that if we
>>>> >> introduce some mocks support, people (especially from community)
will
>>>> >> start to test with mocks everywhere and add them to all PRs
(even to
>>>> >> those which could be easily tested properly). Not sure if
it's better
>>>> >> alternative to skip automated test at all instead of test with
mocks?
>>>> >>
>>>> >> But for this one, I guess we can likely extend our social test
with an
>>>> >> account from hosted domain and hence test it properly?
>>>> >>
>>>> >> Marek
>>>> >>
>>>> >>
>>>> >> On 30/05/18 09:19, Stian Thorgersen wrote:
>>>> >> > At the moment our testing strategy is only with functional
or integration
>>>> >> > level tests. Both with the full server up and running and
primarily
>>>> >> testing
>>>> >> > through public APIs.
>>>> >> >
>>>> >> > Now here comes the question should we also allow testing
through a
>>>> >> mocking
>>>> >> > library like Mockito?
>>>> >> >
>>>> >> > In general I'm against mocking libraries. At best you
end up with
>>>> >> something
>>>> >> > that may work, but you're not guaranteed that it
actually works. They
>>>> >> also
>>>> >> > have a very big maintenance cost when any changes are made
to the
>>>> >> codebase.
>>>> >> >
>>>> >> > However, take a look at
https://github.com/keycloak/keycloak/pull/5215
>>>> >> for
>>>> >> > example. It is a contribution to add support for the hd
param to the
>>>> >> Google
>>>> >> > login. Not sure how else we could test this without a
mocking library.
>>>> >> > _______________________________________________
>>>> >> > keycloak-dev mailing list
>>>> >> > keycloak-dev(a)lists.jboss.org
>>>> >> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>> >>
>>>> >>
>>>> >> _______________________________________________
>>>> >> keycloak-dev mailing list
>>>> >> keycloak-dev(a)lists.jboss.org
>>>> >> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>> >>
>>>> > _______________________________________________
>>>> > keycloak-dev mailing list
>>>> > keycloak-dev(a)lists.jboss.org
>>>> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>
>>>>
>>>>
>>>> --
>>>> Bill Burke
>>>> Red Hat
>>>
>>
>
4:18 a.m.
For IDP, I am not sure how would Mocked IDP help? We already have lots
of test coverage for Identity brokering for OIDC and SAML. The social
providers usually implements OAuth/OIDC and there are just small
differences between them. But those small differences and the fact that
social providers are out of our control, is exactly the reason why we
need to test with real providers instead of mocks IMO.
For example Facebook can do some backwards incompatible change (EG.
require new parameter in their login flow), which will break the social
login with Facebook. And if we test with mock instead of real facebook,
we won't detect it. For this PR, Google can also decide to change the
semantics of "hd" and send different value of it (or stop sending "hd"
at all) and we won't detect it if we test it with mocks.
Marek
Dne 31.5.2018 v 21:46 Anil Saldanha napsal(a):
Agree. I had forgotten about Arquillian.
On May 31, 2018, at 2:09 PM, Stian Thorgersen <sthorger(a)redhat.com
<mailto:sthorger@redhat.com>> wrote:
>
>
> On 31 May 2018 at 20:48, Anil Saldanha <asaldanha1947(a)gmail.com
> <mailto:asaldanha1947@gmail.com>> wrote:
>
> You need both sets of testing to maintain the long term quality
> of a security project like Keycloak. :-)
>
>
> By using Arquillian we are actually able to test Keycloak very
> extensively without the need for Mocking through functional testing.
> I would suggest you review how extensive coverage we already have
> completely without any use of mocking and almost no basic unit
> testing at all.
>
> The problem I see is really only down to the difficulty of writing
> integration testing in some cases, especially when there are many
> configuration options. That's where it may be useful to mock the
> system being integrated with, but through mocking http calls, not
> Java method calls.
>
>
> I agree on the mocked IDP approach being more effective.
>
> On Thu, May 31, 2018 at 1:20 PM, Stian Thorgersen
> <sthorger(a)redhat.com <mailto:sthorger@redhat.com>> wrote:
>
> With regards to Identity Brokering and Social Login one
> potentially better way to do it would be to have a mocked
> IdP. At least then you are testing the actual HTTP calls and
> it's much closer to a real scenario than doing junit tests
> with mocks and only testing parts of the code base.
>
> On 31 May 2018 at 18:08, Anil Saldanha
> <asaldanha1947(a)gmail.com <mailto:asaldanha1947@gmail.com>>
wrote:
>
> I am sorry I have not looked at your integration tests.
> So I may be wrong here. <namaste/>
>
> One area that I feel that you need lots of integration
> tests is in the "Identity Brokering" feature. You have
> way too many permutations and combinations (with
> authentication flows) that without proper integration
> tests/mocks, you will not do proper justice to the value
> of KC/RH-SSO.
>
> On Thu, May 31, 2018 at 10:17 AM, Bill Burke
> <bburke(a)redhat.com <mailto:bburke@redhat.com>> wrote:
>
> Never liked mocks either as they are just opinionated
> warped
> perceptions of reality. But there are cases where I
> think we need
> them off the top of my head:
>
> * Openshift integration
> * Social logins.
>
> Openshift we just won't have the infrastructure for
> it. Social login
> tests require accounts which we won't want to share
> the logins for in
> a public CI. We need something in public CI so that
> we can catch at
> least some potential problems and that we can easily
> try to reproduce
> in local IDE environments. Mocks wouldn't be a
> replacement for real
> integration testing, just a stop gap.
>
>
>
> On Thu, May 31, 2018 at 10:44 AM, Anil Saldanha
> <asaldanha1947(a)gmail.com
> <mailto:asaldanha1947@gmail.com>> wrote:
> > You really have to test your core logic (with no
> server set up) using
> > integration tests with mocking libraries. One way
> to test out all the
> > possible preconditions to your logic.
> >
> > You will be doing a service to your customers and
> users by shielding them
> > from those nasty exceptions (NPEs) that happen in
> an integration platform
> > such as KeyCloak/RH-SSO. :-)
> >
> > /me wearing a user/customer hat
> >
> > On Thu, May 31, 2018 at 1:38 AM, Marek Posolda
> <mposolda(a)redhat.com <mailto:mposolda@redhat.com>>
wrote:
> >
> >> Not sure. I am not strongly against it, but I
> afraid that if we
> >> introduce some mocks support, people (especially
> from community) will
> >> start to test with mocks everywhere and add them
> to all PRs (even to
> >> those which could be easily tested properly). Not
> sure if it's better
> >> alternative to skip automated test at all instead
> of test with mocks?
> >>
> >> But for this one, I guess we can likely extend our
> social test with an
> >> account from hosted domain and hence test it properly?
> >>
> >> Marek
> >>
> >>
> >> On 30/05/18 09:19, Stian Thorgersen wrote:
> >> > At the moment our testing strategy is only with
> functional or integration
> >> > level tests. Both with the full server up and
> running and primarily
> >> testing
> >> > through public APIs.
> >> >
> >> > Now here comes the question should we also allow
> testing through a
> >> mocking
> >> > library like Mockito?
> >> >
> >> > In general I'm against mocking libraries. At
> best you end up with
> >> something
> >> > that may work, but you're not guaranteed that it
> actually works. They
> >> also
> >> > have a very big maintenance cost when any
> changes are made to the
> >> codebase.
> >> >
> >> > However, take a look at
> https://github.com/keycloak/keycloak/pull/5215
> <https://github.com/keycloak/keycloak/pull/5215>
> >> for
> >> > example. It is a contribution to add support for
> the hd param to the
> >> Google
> >> > login. Not sure how else we could test this
> without a mocking library.
> >> > _______________________________________________
> >> > keycloak-dev mailing list
> >> > keycloak-dev(a)lists.jboss.org
> <mailto:keycloak-dev@lists.jboss.org>
> >> >
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
> >>
> >>
> >> _______________________________________________
> >> keycloak-dev mailing list
> >> keycloak-dev(a)lists.jboss.org
> <mailto:keycloak-dev@lists.jboss.org>
> >>
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
> >>
> > _______________________________________________
> > keycloak-dev mailing list
> > keycloak-dev(a)lists.jboss.org
> <mailto:keycloak-dev@lists.jboss.org>
> >
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
>
>
>
> --
> Bill Burke
> Red Hat
>
>
>
>
>
3:33 a.m.
I don't like mocks for many reasons, but this PR seems to me reasonable.
It's hard to setup and test the social IDPs in general. But if we allow
using mocks I have same concern as Marek, I'm afraid that the mocks will
be used everywhere.
If we will be able to limit mocks usage for specific areas like social
logins, it's acceptable for me.
Pavel
Dne 31.5.2018 v 08:38 Marek Posolda napsal(a):
Not sure. I am not strongly against it, but I afraid that if we
introduce some mocks support, people (especially from community) will
start to test with mocks everywhere and add them to all PRs (even to
those which could be easily tested properly). Not sure if it's better
alternative to skip automated test at all instead of test with mocks?
But for this one, I guess we can likely extend our social test with an
account from hosted domain and hence test it properly?
Marek
On 30/05/18 09:19, Stian Thorgersen wrote:
> At the moment our testing strategy is only with functional or integration
> level tests. Both with the full server up and running and primarily testing
> through public APIs.
>
> Now here comes the question should we also allow testing through a mocking
> library like Mockito?
>
> In general I'm against mocking libraries. At best you end up with something
> that may work, but you're not guaranteed that it actually works. They also
> have a very big maintenance cost when any changes are made to the codebase.
>
> However, take a look at https://github.com/keycloak/keycloak/pull/5215 for
> example. It is a contribution to add support for the hd param to the Google
> login. Not sure how else we could test this without a mocking library.
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
_______________________________________________
keycloak-dev mailing list
keycloak-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-dev
3:53 a.m.
+1
On 06/01/2018 10:33 AM, Pavel Drozd wrote:
I don't like mocks for many reasons, but this PR seems to me
reasonable.
It's hard to setup and test the social IDPs in general. But if we allow
using mocks I have same concern as Marek, I'm afraid that the mocks will
be used everywhere.
If we will be able to limit mocks usage for specific areas like social
logins, it's acceptable for me.
Pavel
Dne 31.5.2018 v 08:38 Marek Posolda napsal(a):
> Not sure. I am not strongly against it, but I afraid that if we
> introduce some mocks support, people (especially from community) will
> start to test with mocks everywhere and add them to all PRs (even to
> those which could be easily tested properly). Not sure if it's better
> alternative to skip automated test at all instead of test with mocks?
>
> But for this one, I guess we can likely extend our social test with an
> account from hosted domain and hence test it properly?
>
> Marek
>
>
> On 30/05/18 09:19, Stian Thorgersen wrote:
>> At the moment our testing strategy is only with functional or integration
>> level tests. Both with the full server up and running and primarily testing
>> through public APIs.
>>
>> Now here comes the question should we also allow testing through a mocking
>> library like Mockito?
>>
>> In general I'm against mocking libraries. At best you end up with something
>> that may work, but you're not guaranteed that it actually works. They also
>> have a very big maintenance cost when any changes are made to the codebase.
>>
>> However, take a look at https://github.com/keycloak/keycloak/pull/5215 for
>> example. It is a contribution to add support for the hd param to the Google
>> login. Not sure how else we could test this without a mocking library.
>> _______________________________________________
>> keycloak-dev mailing list
>> keycloak-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
_______________________________________________
keycloak-dev mailing list
keycloak-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-dev
2400
days inactive
2402
days old
11 comments
6 participants
participants (6)
-
Anil Saldanha -
Bill Burke -
Marek Posolda -
Pavel Drozd -
Stian Thorgersen -
Vlasta Ramik