TLDR; I've setup a PoC [1] for an alternative way to test the HAL management console. The PoC is based on Testcontainers [2], Arquillian [3] and JUnit 5. Although this is primarily intended for UI testing, the usage of Testcontainers could also be interesting for the WildFly test suite. --- The existing HAL test suite [4] is a rich test suite for the HAL management console. It contains 300+ UI tests based on Arquillian [3]. The UI tests require a browser and a running WildFly instance as an execution environment. This makes is hard to run the tests on a CI server. Another issue is that the tests are not very stable and often run into timeouts. To execute the complete test suite you need a stable environment. Recently I came across Testcontainers [2], which provide a nice API to start arbitrary containers before / after (all) unit tests. The library also provides an elegant way to start and use browsers in remote web containers. Therefore, I decided to implement a PoC which provides an alternative and more stable way to test the HAL management console. Tests are self contained and can run in CI environments such as GH Actions or Teamcity. At the same time the new approach has to be compatible with the existing test suite. Tests should be runnable w/o major code changes using the new approach. The result is [1]. If you're interested, feel free to take a look and let me know what you think! --- [1] https://github.com/hpehl/manatoko [2] https://www.testcontainers.org/ [3] http://arquillian.org/arquillian-graphene/ [4] https://github.com/hal/testsuite.next _______________________________________________ wildfly-dev mailing list -- wildfly-dev(a)lists.jboss.org To unsubscribe send an email to wildfly-dev-leave(a)lists.jboss.org %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s

Hi Harald, a couple of years ago we experimented with containerizing a browser for HAL testsuite execution but these days we just increased the complexity a bit and didn't improve stability so we abandoned this approach but your current solution could be better. I like that this new way HAL testsuite can be easier to execute in various environments. Is your idea to enhance the current testsuite with all tests coverage it already provides to be executed using testcontainers or to replace the current testsuite with new one and rewrite test coverage from scratch? BTW my understanding was there are already some plans to use testcontainers in WildFly testsuite but maybe I am wrong?

On Tue, Feb 1, 2022 at 8:44 AM Jan Kašík <jkasik(a)redhat.com&gt; wrote: > Hello! Thanks for this effort Harald. I really like the motivation. > > I took a look at the PoC and tried to run it with Podman (IIRC officially > unsupported by TestContainers so mileage may vary). It took up to 30 > seconds to initialize containers already downloaded from quay.io for a > test class. I'm also not a big fan of a common parent class for each test > class and I would prefer to have an additional few lines of code in each > test class. Otherwise it runs fine. > > Thanks to having multiple Maven modules with tests it is currently > possible to parallelize the run of individual modules in Jenkins matrix > configuration. Is it possible to download all needed containers in a > preparation step to not have to do this in every matrix configuration? > Initialization like this would add some execution time to the test run. > > IMHO it is hard to say if stability will improve in the usual large scale > HAL tests are running in. Jenkins nodes we use are virtualized. I can > imagine that using TestContainers may improve stability in terms of > dependencies - now we would be sure browser and VNC are always present and > properly configured but that's not a common problem from my experience. How > would using TestContainers improve stability? > > If all the changes you mentioned are aimed only at upstream (since you > mention GH actions and TeamCity), would it be possible to still run tests > without TestContainers? > > Does the TestContainers solution suppose to replace existing testsuite? > > Thanks! > > On Mon, Jan 31, 2022 at 7:43 PM Harald Pehl <hpehl(a)redhat.com&gt; wrote: > >> TLDR; >> >> I've setup a PoC [1] for an alternative way to test the HAL management >> console. >> The PoC is based on Testcontainers [2], Arquillian [3] and JUnit 5. >> Although >> this is primarily intended for UI testing, the usage of Testcontainers >> could >> also be interesting for the WildFly test suite. >> >> --- >> >> The existing HAL test suite [4] is a rich test suite for the HAL >> management >> console. It contains 300+ UI tests based on Arquillian [3]. >> >> The UI tests require a browser and a running WildFly instance as an >> execution >> environment. This makes is hard to run the tests on a CI server. Another >> issue >> is that the tests are not very stable and often run into timeouts. To >> execute >> the complete test suite you need a stable environment. >> >> Recently I came across Testcontainers [2], which provide a nice API to >> start >> arbitrary containers before / after (all) unit tests. The library also >> provides >> an elegant way to start and use browsers in remote web containers. >> >> Therefore, I decided to implement a PoC which provides an alternative and >> more stable way to test the HAL management console. Tests are self >> contained and can run in CI environments such as GH Actions or Teamcity. >> >> At the same time the new approach has to be compatible with the existing >> test >> suite. Tests should be runnable w/o major code changes using the new >> approach. >> >> The result is [1]. If you're interested, feel free to take a look and >> let me >> know what you think! >> >> --- >> [1] https://github.com/hpehl/manatoko >> [2] https://www.testcontainers.org/ >> [3] http://arquillian.org/arquillian-graphene/ >> [4] https://github.com/hal/testsuite.next >> _______________________________________________ >> wildfly-dev mailing list -- wildfly-dev(a)lists.jboss.org >> To unsubscribe send an email to wildfly-dev-leave(a)lists.jboss.org >> %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s > > > > -- > Jan (Honza) Kasik > Quality Engineer, EAP QE > _______________________________________________ > wildfly-dev mailing list -- wildfly-dev(a)lists.jboss.org > To unsubscribe send an email to wildfly-dev-leave(a)lists.jboss.org > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s _______________________________________________ wildfly-dev mailing list -- wildfly-dev(a)lists.jboss.org To unsubscribe send an email to wildfly-dev-leave(a)lists.jboss.org %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s

On Tue, Feb 1, 2022 at 8:43 AM Jan Kašík <jkasik(a)redhat.com&gt; wrote: > > I took a look at the PoC and tried to run it with Podman (IIRC officially unsupported by TestContainers so mileage may vary). It took up to 30 seconds to initialize containers already downloaded from quay.io for a test class. I'm also not a big fan of a common parent class for each test class and I would prefer to have an additional few lines of code in each test class. Otherwise it runs fine. Once the containers have been downloaded from quay.io, the startup times are pretty fast on my machine. The browser and HAL container are started just once for all tests, which takes about 20 seconds. The WildFly instance which is started once for each test takes about another 20 seconds.

Since the test should run on a CI environment, the startup times don't really matter IMO.

I agree that we should avoid putting too much logic in common test classes. But I think there are useful for several reasons: - place common annotations - start a clean WildFly instance

> Thanks to having multiple Maven modules with tests it is currently possible to parallelize the run of individual modules in Jenkins matrix configuration. Is it possible to download all needed containers in a preparation step to not have to do this in every matrix configuration? Initialization like this would add some execution time to the test run. I'm not sure if that's possible. > IMHO it is hard to say if stability will improve in the usual large scale HAL tests are running in. Jenkins nodes we use are virtualized. I can imagine that using TestContainers may improve stability in terms of dependencies - now we would be sure browser and VNC are always present and properly configured but that's not a common problem from my experience. How would using TestContainers improve stability? When I run the tests locally, the tests are not very stable and I often run into timeouts. Also the tests interfere with other applications. This might not be a problem in a CI environment though. > If all the changes you mentioned are aimed only at upstream (since you mention GH actions and TeamCity), would it be possible to still run tests without TestContainers? > Does the TestContainers solution suppose to replace existing testsuite? Currently the tests depend on Testcontainers and would not run without it. However I'd like the tests to be independent of Testcontainers. They should just depend on the pages, fragments, helper classes, Arquillian and JUnit. I don't think it makes sense to rewrite the current test suite. We put too much effort into it. If you think the new approach makes sense, we should rather create a branch and apply the new approach into the existing test suite. WDYT? -- Harald Pehl Senior Software Engineer hpehl(a)redhat.com

On Fri, Feb 4, 2022 at 11:44 AM Jan Kašík <jkasik(a)redhat.com&gt; wrote: > > I observed initialization taking place with each test class. > The WildFly container should be started / stopped before / after each unit test, but the browser and HAL container should be started only once. If your experience is different, let me know, the I should take a closer look. > > IMO it can add up to tens of minutes to the run. > On GitHub Actions the WildFly container starts in under 10 seconds, which is ok for me. > > It always starts with a few lines of code and ends up in a pile of mess. :) I'm really not a fan of inheritance in case of test classes since I spent some time trying to resolve issues caused by this. > I think it's a trade-off. If you really don't like it, I can try to use composition over inheritance. > > There's also another thing which I thought of. The host OS on CI machines might be RHEL 8 where docker is not available. Do you also plan to document within this effort how to run tests with podman? > I'll try to find resources on how to run w/o docker. I know there are some discussions, issues and PRs on the Testcontainers repo.