[keycloak-dev] Microservice-oriented Integration Testing

Tue Mar 19 11:30:25 EDT 2019

I can share my own thoughts on this as I been working on custom tests on
top of Keycloak and it turned out to be a non trivial task.

Few thoughts on this:
- Keycloak testsuite is not available in separate JAR/module, meaning it
can not be included in 3rd party projects as test framework. From maven
perspective my project would need to depend on keycloak test suite
"test-jar" which is not published in maven central. It is also kind of
anti-pattern.
- Setting up Arquillian separatelly with all its dependencies is
nightmare. Yes, there are several components which bring incompatible
core and 2-3 extra version placeholders to manage.

These are my findings which I suffered. I would really love at least
move of testsuite framework to its own module so it can be references as
first class citizen.
AbstractKeycloakTest should be part of src/main/java somewhere and get
imported into arquillian tests as test scoped dependency. Separating a
test framework will help keeping tests separated from specifics needed
to setup things.

I had an attempt with Arquillian Cube and Docker, however it has severe
issues when dependencies include a jaxrs provider other than Jersey.
Later one is used by docker client. Pulling in resteasy together with
Keycloak dependencies simply blows ups internals. I believe there is a
future in it, but couldn't pay for it just yet.

Kind regards,
Łukasz

On 11.03.2019 09:39, Stian Thorgersen wrote:
> In general it's a good idea and it can probably be done in stages rather
> than as one big task.
> 
> Could be something like:
> 
> 1. Run browser on a separate node. This could allow testing multiple
> browsers concurrently with the same Keycloak server. Perhaps we could
> create separate realms for each browser in order to allow them to run
> concurrently.
> 2. Adapter tests. These should just provision a Keycloak server somehow in
> a similar fashion to how we consume a DB from the server tests as we are
> testing the adapter not the server here. Running two side-by-side in
> Arquillian doesn't make all that much sense.
> 3. Allow tests to run separately to server. This will be required to allow
> using the testsuite on Keycloak running on Docker or OpenShift.
> 
> On Fri, 8 Mar 2019 at 20:47, Tomas Kyjovsky <tkyjovsk at redhat.com> wrote:
> 
>> I have some ideas about a possible future direction of the testsuite. I
>> wanted to kick off a discussion around this topic and see what others think.
>>
>> In context of the microservice paradigm our current approach to
>> integration testing seems a bit monolithic.
>> We run the whole testsuite on a single "all-in-one" host, while a typical
>> SSO use case always involves interaction of at least 3 separate entities
>> spread across different hosts, talking to each other via network.
>>
>> 1) SSO server
>>   + server-side integrations: JPA/cache server, LDAP server, external IDP,
>> etc.
>>   + possible clustering/scaling/failover/upgrade scenarios
>> 2) SSO client (secured service)
>>   + multiple different services, different runtimes
>>   + possibly clustered
>> 3) SSO user (delegated by user agent / browser)
>>
>> The all-in-one approach still works, and it's perhaps better for local
>> development-testing loop, but it's just a bit weird in proper integration
>> testing that for example browser (in UI compatibility tests) runs on the
>> same machine with the server and all the other services. I've been
>> wondering if it's worth pushing for a more microservice-oriented approach
>> with proper service decomposition.
>>
>> More detailed service decomposition here, feel free to add comments on it:
>> [1].
>>
>> Advatanges:
>> - more realistic setup
>> - issues which don't appear when testing on a single host can be discovered
>> - different setups on server / client / user side can be provisioned and
>> combined independently
>> - the test machine itself could be reduced to git + one JDK + Maven +
>> cloud client tools (e.g. docker, novaclient, etc.)
>> - ... anything else?
>>
>> Disadvantages:
>> - additional work, unclear how much
>> - increased complexity, a different set of challenges related to
>> provisioning (but with local docker not that much)
>> - needs preparation/configuration of VM images for all tested services
>> - not sure right now how we would handle some corner cases like service
>> restarts/reconfigs which are needed for some tests
>> - some tests might have to be rewritten/adapted to the non-localhost
>> environment
>> - ... anything else?
>>
>> Options:
>> - docker / podman
>> - openshift / kubernetes
>> - openstack, aws or similar cloud
>> - a combination of the above?
>>
>> In general the process would look like this: build project --> build
>> distro zips --> build VM images --> run tests while the testsuite itself
>> would be able to provision and teardown the tested system (or its
>> individual components) from the pre-built images. We already use something
>> like this in the performance testsuite (docker + docker compose). Maybe it
>> could be generalized to the whole testsuite.
>>
>> Since the integration testsuite already uses Arquillian I think that the
>> Arquillian Cube extension [2] would be an obvious candidate because it
>> supports both Docker (incl. the Compose orchestration format) and
>> Kubernetes/Openshift cluster (need to investigate extent of support). While
>> also keeping the default JVM-embedded test mode for development.
>>
>> For additional separation, we could also start using the remote Webdriver
>> mode instead being tied to the browser installed locally on the test
>> machine. Arquillian is able send commands to a remote Selenium Grid [3]
>> which has a bunch of independent browser nodes and relay the commands. The
>> grid is provisioned separately, and there is already some automation for it
>> in docker [4]. (I already did a quick test a while ago. There were some
>> minor issues but with some effort I think it could work.)
>>
>> Some ideas to think about. Let me  know what you think, or if you have
>> some other ideas/solutions related to this topic.
>>
>>
>> Tomas
>>
>> [1]
>> https://docs.google.com/spreadsheets/d/1PbsSfU8R6CEz4yYCm6Mld1pMNXxeb19JEgcmaR3AiTQ/
>> [2] http://arquillian.org/arquillian-cube/
>> [3] https://www.seleniumhq.org/docs/07_selenium_grid.jsp
>> [4] https://github.com/SeleniumHQ/docker-selenium
>> _______________________________________________
>> keycloak-dev mailing list
>> keycloak-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
> 