[gatein-dev] Selenium tests

In order to run selenium tests with jboss I have to do the following modification to current pom.xml: Index: testsuite/selenium-snifftests/pom.xml =================================================================== --- testsuite/selenium-snifftests/pom.xml (revision 4045) +++ testsuite/selenium-snifftests/pom.xml (working copy) @@ -13,12 +13,12 @@ <properties> <org.selenium.server.version>1.0.1</org.selenium.server.version> - <selenium.port>4444</selenium.port> + <selenium.port>8444</selenium.port> <selenium.browser>firefox</selenium.browser> <selenium.timeout>10000</selenium.timeout> <selenium.speed>300</selenium.speed> <selenium.host>localhost</selenium.host> - <org.selenium.maven-plugin.version>1.0</org.selenium.maven-plugin.version> + <org.selenium.maven-plugin.version>1.0.1</org.selenium.maven-plugin.version> </properties> <dependencies> I've been running these a lot the last two days. They aren't very useful at the moment for doing pre-commit checks to catch any introduction of systemic issues primarily for three reasons. - one, it takes 1 hour 40 mins on my laptop to run the suite. If I want a 'before my change', and 'after my change' I have to run it twice so I can see a diff in test failures. The name 'snifftests' gives an impression that this is a quick testsuite to be run before doing a commit :) - two, at the moment many tests are failing - my last run: Tests run: 248, Failures: 74, Errors: 21, Skipped: 0 which makes it difficult to determine if some might be due to my code changes. The same tests sometimes fail with a 'Failure', and sometimes with 'Error', so the end report always looks different making finding effective differences a challenge even with a diff tool. - three, some of the tests seem to fail randomly - more likely they are sensitive to initial conditions which can change if some other test fails to do a proper cleanup. It can also happen by killing the test in the middle of execution. The situation is exacerbated by the fact that the tests are run in random order ... Point number one could be addressed by making a set of simple to maintain tests that perform a few operations that touch many aspects of the portal. These would go into 'snifftests' module. The exhaustive mass of other detailed tests - which are undoubtedly also a burden to maintain, would go into 'alltests'. Point number two could be addressed by creating some kind of final report that would throw 'failures' and 'errors' in a single set and sort it alphabetically. For point number three, a slow workaround is to run another packaging/pkg/mvn install. Maybe we could add some cleanup mechanism, so each test can define the tear down sequence which would go through all the steps ignoring any errors. Removing test artifacts created during a test is already part of the existing tests. But if it was separated it could be run explicitly: mvn -Pselenium-cleanup integration-test Otherwise thumbs up for the sheer number of these tests, and the systematic approach ... - marko