1:45 p.m.
I have some ideas about a possible future direction of the testsuite. I wanted to kick off
a discussion around this topic and see what others think.
In context of the microservice paradigm our current approach to integration testing seems
a bit monolithic.
We run the whole testsuite on a single "all-in-one" host, while a typical SSO
use case always involves interaction of at least 3 separate entities spread across
different hosts, talking to each other via network.
1) SSO server
+ server-side integrations: JPA/cache server, LDAP server, external IDP, etc.
+ possible clustering/scaling/failover/upgrade scenarios
2) SSO client (secured service)
+ multiple different services, different runtimes
+ possibly clustered
3) SSO user (delegated by user agent / browser)
The all-in-one approach still works, and it's perhaps better for local
development-testing loop, but it's just a bit weird in proper integration testing that
for example browser (in UI compatibility tests) runs on the same machine with the server
and all the other services. I've been wondering if it's worth pushing for a more
microservice-oriented approach with proper service decomposition.
More detailed service decomposition here, feel free to add comments on it: [1].
Advatanges:
- more realistic setup
- issues which don't appear when testing on a single host can be discovered
- different setups on server / client / user side can be provisioned and combined
independently
- the test machine itself could be reduced to git + one JDK + Maven + cloud client tools
(e.g. docker, novaclient, etc.)
- ... anything else?
Disadvantages:
- additional work, unclear how much
- increased complexity, a different set of challenges related to provisioning (but with
local docker not that much)
- needs preparation/configuration of VM images for all tested services
- not sure right now how we would handle some corner cases like service restarts/reconfigs
which are needed for some tests
- some tests might have to be rewritten/adapted to the non-localhost environment
- ... anything else?
Options:
- docker / podman
- openshift / kubernetes
- openstack, aws or similar cloud
- a combination of the above?
In general the process would look like this: build project --> build distro zips -->
build VM images --> run tests while the testsuite itself would be able to provision and
teardown the tested system (or its individual components) from the pre-built images. We
already use something like this in the performance testsuite (docker + docker compose).
Maybe it could be generalized to the whole testsuite.
Since the integration testsuite already uses Arquillian I think that the Arquillian Cube
extension [2] would be an obvious candidate because it supports both Docker (incl. the
Compose orchestration format) and Kubernetes/Openshift cluster (need to investigate extent
of support). While also keeping the default JVM-embedded test mode for development.
For additional separation, we could also start using the remote Webdriver mode instead
being tied to the browser installed locally on the test machine. Arquillian is able send
commands to a remote Selenium Grid [3] which has a bunch of independent browser nodes and
relay the commands. The grid is provisioned separately, and there is already some
automation for it in docker [4]. (I already did a quick test a while ago. There were some
minor issues but with some effort I think it could work.)
Some ideas to think about. Let me know what you think, or if you have some other
ideas/solutions related to this topic.
Tomas
[1] https://docs.google.com/spreadsheets/d/1PbsSfU8R6CEz4yYCm6Mld1pMNXxeb19JE...
[2] http://arquillian.org/arquillian-cube/
[3] https://www.seleniumhq.org/docs/07_selenium_grid.jsp
[4] https://github.com/SeleniumHQ/docker-selenium