[keycloak-dev] Microservice-oriented Integration Testing

Mon Mar 11 04:27:25 EDT 2019

Hello,

TLDR: I think end to end testing using selenium and web driver to test in
different browser is a waste of time.

fyi: Facebook run unit tests for thousands of test files and tens of
thousands of modules directly in node. They don’t run those tests in
different browsers:

> After living with this problem for a while we discovered that Node tests
cover our internal logic really well, and browser tests, even if we had
them, don’t give us enough confidence. Many browser issues and regressions
that we have had are unobservable from the code, and need actual human
interaction. They are also very tedious to write and understand.
> I think at this point we won’t be investing into running our unit tests
in the browser. The combination of Node unit tests and manual DOM tests
seems to be working well, and further work might include making DOM
fixtures easier to access. Dan Abramov
<https://github.com/facebook/react/issues/5703#issuecomment-314770489>

I don't want to transform this thread in ui integration testing debate.
Integration tests are always difficult to understand and write.
You probably need to verify that those tests give you enough confidence,
and catch bugs present on one browser specifically.

> We need each browser to have a separate server since the UI tests are
changing it's configuration etc. Selenium Grid would be good for testing
some read-only app where concurrency is not a problem.

> if you have some other ideas/solutions related to this topic.

An idea is to make integration tests idempotent, to make tests parallels
and speed up test suite.

Cheers

On Fri, Mar 8, 2019 at 9:47 PM Václav Muzikář <vmuzikar at redhat.com> wrote:

> I quite like the idea of running those three entities separately,
> especially in context of UI tests with desktop browsers (mobile browsers
> are run in VMs in any case). However, I can see two problems with that.
>
> 1) We'd need to maintain both the current all-in-one approach and the
> decoupled approach. Those two are quite different and it won't work "just
> like that" over time - maintenance will be required. Do we have capacity
> for this?
>
> 2) In order to make sense (as real environment as possible), the images
> would need to be based on a fully-fledged OS (whole RHEL/Windows Server
> distribution) - not just some bare minimum like Alpine. We'd need to
> either:
> 2.1) Use docker, i.e. basically a nested virtualization (we've got a
> provisioned VM where the tests are run - same like now - and this VM would
> host several more VMs with server/service/client). I'm afraid this could
> bring potential problems (there's a difference in virtualizing some small
> service in Docker and e.g. Windows Server in Docker) and a significant
> overhead as well.
> 2.2) Use openstack/shift. This would mean running 3 large VMs for each part
> of testsuite that's ran in parallel (in contrast to 1 like we have now). Do
> we have resources for something like this? As an alternative, we could run
> less tests in parallel, but I don't think this is something we want - to
> run tests much more slower.
>
> Regarding the Selenium Grid. I've researched it a bit in the past, and if I
> understand it correctly, it's not usable for us. It's focused on parallel
> test execution, i.e. running multiple tests in multiple browsers against 1
> source (auth/app server). We need each browser to have a separate server
> since the UI tests are changing it's configuration etc. Selenium Grid would
> be good for testing some read-only app where concurrency is not a problem.
> Not mentioning the fact that we already run UI tests in parallel (each
> browser independently from others).
>
> Remote Webdriver wouldn't be a problem. In fact, we are already using it!
> FF/Chrome/IE drivers are technically remote drivers - it's just using
> localhost. It might not seem so because the browser's execution is managed
> by Arquillian Drone so everything seems seamless. But Drone doesn't need to
> be the one who's executing the browser - it can be something completely
> different on a different machine. And it wouldn't even require much changes
> to the testsuite.
>
> V.
>
> On Fri, Mar 8, 2019 at 8:47 PM Tomas Kyjovsky <tkyjovsk at redhat.com> wrote:
>
> > I have some ideas about a possible future direction of the testsuite. I
> > wanted to kick off a discussion around this topic and see what others
> think.
> >
> > In context of the microservice paradigm our current approach to
> > integration testing seems a bit monolithic.
> > We run the whole testsuite on a single "all-in-one" host, while a typical
> > SSO use case always involves interaction of at least 3 separate entities
> > spread across different hosts, talking to each other via network.
> >
> > 1) SSO server
> >   + server-side integrations: JPA/cache server, LDAP server, external
> IDP,
> > etc.
> >   + possible clustering/scaling/failover/upgrade scenarios
> > 2) SSO client (secured service)
> >   + multiple different services, different runtimes
> >   + possibly clustered
> > 3) SSO user (delegated by user agent / browser)
> >
> > The all-in-one approach still works, and it's perhaps better for local
> > development-testing loop, but it's just a bit weird in proper integration
> > testing that for example browser (in UI compatibility tests) runs on the
> > same machine with the server and all the other services. I've been
> > wondering if it's worth pushing for a more microservice-oriented approach
> > with proper service decomposition.
> >
> > More detailed service decomposition here, feel free to add comments on
> it:
> > [1].
> >
> > Advatanges:
> > - more realistic setup
> > - issues which don't appear when testing on a single host can be
> discovered
> > - different setups on server / client / user side can be provisioned and
> > combined independently
> > - the test machine itself could be reduced to git + one JDK + Maven +
> > cloud client tools (e.g. docker, novaclient, etc.)
> > - ... anything else?
> >
> > Disadvantages:
> > - additional work, unclear how much
> > - increased complexity, a different set of challenges related to
> > provisioning (but with local docker not that much)
> > - needs preparation/configuration of VM images for all tested services
> > - not sure right now how we would handle some corner cases like service
> > restarts/reconfigs which are needed for some tests
> > - some tests might have to be rewritten/adapted to the non-localhost
> > environment
> > - ... anything else?
> >
> > Options:
> > - docker / podman
> > - openshift / kubernetes
> > - openstack, aws or similar cloud
> > - a combination of the above?
> >
> > In general the process would look like this: build project --> build
> > distro zips --> build VM images --> run tests while the testsuite itself
> > would be able to provision and teardown the tested system (or its
> > individual components) from the pre-built images. We already use
> something
> > like this in the performance testsuite (docker + docker compose). Maybe
> it
> > could be generalized to the whole testsuite.
> >
> > Since the integration testsuite already uses Arquillian I think that the
> > Arquillian Cube extension [2] would be an obvious candidate because it
> > supports both Docker (incl. the Compose orchestration format) and
> > Kubernetes/Openshift cluster (need to investigate extent of support).
> While
> > also keeping the default JVM-embedded test mode for development.
> >
> > For additional separation, we could also start using the remote Webdriver
> > mode instead being tied to the browser installed locally on the test
> > machine. Arquillian is able send commands to a remote Selenium Grid [3]
> > which has a bunch of independent browser nodes and relay the commands.
> The
> > grid is provisioned separately, and there is already some automation for
> it
> > in docker [4]. (I already did a quick test a while ago. There were some
> > minor issues but with some effort I think it could work.)
> >
> > Some ideas to think about. Let me  know what you think, or if you have
> > some other ideas/solutions related to this topic.
> >
> >
> > Tomas
> >
> > [1]
> >
> https://docs.google.com/spreadsheets/d/1PbsSfU8R6CEz4yYCm6Mld1pMNXxeb19JEgcmaR3AiTQ/
> > [2] http://arquillian.org/arquillian-cube/
> > [3] https://www.seleniumhq.org/docs/07_selenium_grid.jsp
> > [4] https://github.com/SeleniumHQ/docker-selenium
> > _______________________________________________
> > keycloak-dev mailing list
> > keycloak-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
> >
>
>
> --
> Václav Muzikář
> Quality Engineer
> Keycloak / Red Hat Single Sign-On
> Red Hat Czech s.r.o.
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev

-- 
Guillaume Vincent
Senior Software Engineer