[keycloak-dev] Why aren't tomcat and jetty tests run!?

[Stian Thorgersen]

Let's put this on hold for after 7.1 GA and make it a priority after that.

We are going to use Arquillian bases tests and removing the old tests as
they have less value from a QE perspective. This has been discussed many
times now and we have agreed that this will be done a long time ago so
please let's not reiterate this discussion again and again. If dev and QE
are going to collaborate on the tests we need to share the approach and QE
has to have full functional tests.

We need to resolve this issue though so we can get access to the
KeycloakSession from within certain tests. Ability to run adapter tests
from within the IDE is also a nice to have, but that may be harder to
achieve (and the effort may not be worth it, at least not for all adapters).

On 9 December 2016 at 08:51, Marek Posolda <mposolda at redhat.com> wrote:

> Ahoj Vlasto,
>
> On 09/12/16 08:28, Vlasta Ramik wrote:
> >
> > Ahoj Marku,
> >
> >
> > On 12/08/2016 05:50 PM, Marek Posolda wrote:
> >> IMO ideal is, if we can have both possibilities. Test with embedded
> >> container during development, but also has possibility to run the same
> >> test against external container in different JVM. Arquillian testsuite
> >> can support both approaches, but the old testsuite doesn't support the
> >> external...
> >>
> >> I agree some tests shouldn't use REST endpoints. Like the model tests
> >> from the old testsuite. But they still need to be able to run on
> >> external environment though. I guess we can have something like the
> >> "bridge" test, which will just deploy the model test to the REST
> >> endpoint and you will be able to test things inside this REST endpoint
> >> on server side? So you will be able to use all the stuff like model
> >> classes etc.
> > There is https://issues.jboss.org/browse/KEYCLOAK-3729 it seems to me
> > that your proposal with "bridge" test could resolve this. It would be
> > great to have a meeting to discuss details or if you will have time to
> > make PoC. I'd love to take a look at it.
> ah, so you already tried different approach with the expose
> KeycloakSessionFactory to JNDI and remote lookup it from tests. This
> seems much easier then my approach based on deploying test on REST
> endpoint. But for the JNDI, I have no clue about the things like
> serialization etc. Maybe there is some easy solution to it, but I don't
> know ATM... I am sure that approach based on REST endpoints will work,
> but it's more work.
>
> +1 for the meeting. Maybe next week or after Christmas? I personally
> won't have time to do any PoC or something around this before Christmas :(
>
> Marek
> >> Marek
> >>
> >> On 08/12/16 17:47, Marek Posolda wrote:
> >>> Arquillian can support that. All we need is just different
> >>> implementations of arquillian DeployableContainer interface to
> >>> represent various containers. I've managed to have adapter tests
> >>> already running on embedded undertow and everything is running in
> >>> single JVM. Development and debugging is fine with that.
> >>>
> >>> Having the test+server+adapter in single JVM just rocks during
> >>> development. Problem is, that it won't handle all type of errors. For
> >>> example if Jetty adapter ZIP distribution is broken (eg. some JAR file
> >>> is missing in it), the Jetty adapter tests, which uses embedded Jetty
> >>> won't catch that. For those reasons, also QA team needs to test with
> >>> "real" containers, not with the embedded ones.
> >>>
> >>>
> >>>
> >>> On 08/12/16 15:05, Bill Burke wrote:
> >>>> On 12/8/16 6:48 AM, Marek Posolda wrote:
> >>>>> They are not included in the script running travis
> >>>>> https://github.com/keycloak/keycloak/blob/master/travis-
> run-tests.sh#L4
> >>>>>
> >>>>> It seems wee should add them here.
> >>>>>
> >>>>>  From the long run, it will be better to remove them (as they rely on
> >>>>> the old testsuite) and rather have them running on the
> >>>>> integration-arquillian testsuite. I know it sucks during development
> >>>>> that Jetty itself doesn't run in same JVM like the test (debugging
> >>>>> etc). But I hope we can have the solution, which will allow to
> >>>>> choose whether Jetty (or hopefully some other servers) are running
> >>>>> in same JVM or different JVM.
> >>>>>
> >>>> Unless arquillian can support running these adapter tests within an
> >>>> IDE, they need to stay.
> >>> Arquillian can support that. All we need is just different
> >>> implementations of arquillian DeployableContainer interface to
> >>> represent various containers. I've managed to have adapter tests
> >>> already running on embedded undertow and everything is running in
> >>> single JVM. Development and debugging is fine with that.
> >>>> In general, More often than not, I don't like the arquillian style
> >>>> we've enforced because it takes a lot longer to write tests, is often
> >>>> nearly impossible to do everything through a REST interface, and is
> >>>> often really hard to debug.  Maybe its just the types of things I'm
> >>>> testing.  Some of the tests should not have been ported to
> >>>> arquillian, or, at least shouldn't have been ported to using a REST
> >>>> interface for the entire test.
> >>> Yeah, having the test+server+adapter in single JVM just rocks during
> >>> development. Problem is, that it won't handle all type of errors. For
> >>> example if Jetty adapter ZIP distribution is broken (eg. some JAR file
> >>> is missing), the Jetty adapter tests, which uses embedded Jetty, won't
> >>> catch that. For those and similar reasons, also QA team needs to test
> >>> with "real" containers, not with the embedded ones.
> >>>
> >>> IMO ideal is, if we can have both possibilities. Test with embedded
> >>> container during development, but also has possibility to run the same
> >>> test against external container in different JVM. Arquillian testsuite
> >>> can support both approaches, but the old testsuite doesn't support the
> >>> external...
> >>>
> >>> I agree some tests shouldn't use REST endpoints. Like the model tests
> >>> from the old testsuite. But they still need to be able to run on
> >>> external environment though. I guess we can have something like the
> >>> "bridge" test, which will just deploy the model test to the REST
> >>> endpoint and you will be able to test things inside this REST endpoint
> >>> on server side? So you will be able to use all the stuff like model
> >>> classes etc.
> >>>
> >>> Marek
> >>>> Bill
> >> _______________________________________________
> >> keycloak-dev mailing list
> >> keycloak-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/keycloak-dev
> >
>
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>