[gatein-dev] Selenium tests

Fri Sep 10 09:26:40 EDT 2010

On Wed, Sep 8, 2010 at 7:54 PM, Arthur Peltier <arthur.peltier at gmail.com>wrote:

> Some remarks in-line
>
> 2010/9/7 Marko Strukelj <marko.strukelj at gmail.com>
>
>
>> In order to run selenium tests with jboss I have to do the following
>> modification to current pom.xml:
>>
>> Index: testsuite/selenium-snifftests/pom.xml
>> ===================================================================
>> --- testsuite/selenium-snifftests/pom.xml       (revision 4045)
>> +++ testsuite/selenium-snifftests/pom.xml       (working copy)
>> @@ -13,12 +13,12 @@
>>
>>         <properties>
>>
>>  <org.selenium.server.version>1.0.1</org.selenium.server.version>
>> -               <selenium.port>4444</selenium.port>
>> +               <selenium.port>8444</selenium.port>
>>                 <selenium.browser>firefox</selenium.browser>
>>                 <selenium.timeout>10000</selenium.timeout>
>>                 <selenium.speed>300</selenium.speed>
>>                 <selenium.host>localhost</selenium.host>
>> -
>> <org.selenium.maven-plugin.version>1.0</org.selenium.maven-plugin.version>
>> +
>> <org.selenium.maven-plugin.version>1.0.1</org.selenium.maven-plugin.version>
>>
> Could you explain why you need this maintenance version exactly ?
>

Without this version upgrade Firefox windows deadlock when opening up so
they fail to open, which causes all the tests to fail. I see this on
Windows, didn't try on unixes.

>
>          </properties>
>>
>>         <dependencies>
>>
>>
>> I've been running these a lot the last two days.
>>
>> They aren't very useful at the moment for doing pre-commit checks to catch
>> any introduction of systemic issues primarily for three reasons.
>>
>>  - one, it takes 1 hour 40 mins on my laptop to run the suite. If I want a
>> 'before my change', and 'after my change' I have to run it twice so I can
>> see a diff in test failures. The name 'snifftests' gives an impression that
>> this is a quick testsuite to be run before doing a commit :)
>>
>  - two, at the moment many tests are failing - my last run: Tests run:
>> 248, Failures: 74, Errors: 21, Skipped: 0
>> which makes it difficult to determine if some might be due to my code
>> changes. The same tests sometimes fail with a 'Failure', and sometimes with
>> 'Error', so the end report always looks different making finding effective
>> differences a challenge even with a diff tool.
>>
>>  - three, some of the tests seem to fail randomly - more likely they are
>> sensitive to initial conditions which can change if some other test fails to
>>  do a proper cleanup. It can also happen by killing the test in the middle
>> of execution. The situation is exacerbated by the fact that the tests are
>> run in random order ...
>>
>> Point number one could be addressed by making a set of simple to maintain
>> tests that perform a few operations that touch many aspects of the portal.
>> These would go into 'snifftests' module. The exhaustive mass of other
>> detailed tests - which are undoubtedly also a burden to maintain, would go
>> into 'alltests'.
>>
>
> In fact there are differences in Functional tests and sniff tests.
> - Sniff tests have "SNF" in the name of the  test
> - These tests can be run specifically by using "*-Dtest=Test*SNF* *" with
> your maven command
> - We will be implementing shortly what we call "Smoke tests" that will test
> in <20 steps most functionalities of GateIn (these will be for testing
> installations on many different configurations)
> - The name of the folder and how to run the different tests will be also be
> changed in SELEGEN 1.1 (expected in a week or so...see more improvements in
> the next points)
>

I gave this a try. SNF tests take 40 mins to run for me. In my opinion
that's still a bit too long for smoke tests. I feel like 15 min is a max.
Running it twice then makes it 30 min.

>
>> Point number two could be addressed by creating some kind of final report
>> that would throw 'failures' and 'errors' in a single set and sort it
>> alphabetically.
>>
> - This will become a reality with SELEGEN 1.1 (we will use a DOXIA maven
> framework to create html reports)...
> - see attached "surefire-report.html" for an old example (i've been using
> it on my own for some time). The format is not perfect but better than
> nothing for now.
>

This is better (alphabetical ordering), but not ideal. There is data that
changes on every run (Time column). Ideally there would also be a .txt file
with proper formatting so it looks like a table.
Then, when doing a diff it's all very cleary visible. Now if I do copy paste
of the html into a text file, then do diff, all the lines have been changed
due to different Time. Even if I use a more advanced diff tool that gives me
precise inline diff in the text form formatting is all skewed and hard to
read...

>
>> For point number three, a slow workaround is to run another
>> packaging/pkg/mvn install.
>> Maybe we could add some cleanup mechanism, so each test can define the
>> tear down sequence which would go through all the steps ignoring any errors.
>> Removing test artifacts created during a test is already part of the
>> existing tests. But if it was separated it could be run explicitly:
>>
> - We have a problem using testsuites with our Maven approach, that is why
> each script is independent and cleans after himself.
> - This also permits us to keep the artifacts of failed tests to better
> analyse what happened
> - we would need to pass a specific scenario between each test... (any ideas
> on this would be helpful)
>

Actually I was thinking more like having complementary test files:

Test_SNF_PRL_03_ChangeDisplayLanguage.html
Test_CLEAN_SNF_PRL_03_ChangeDisplayLanguage.html

And then run: -Dtest=Test_CLEAN*

The CLEAN would be automatically executed right after the complementary
non-CLEAN. But you can also run all the CLEAN directly. And these would not
be sensitive to errors. They would run all the way through on a best-effort
basis.

>
>> mvn -Pselenium-cleanup integration-test
>>
>>
>> Otherwise thumbs up for the sheer number of these tests, and the
>> systematic approach ...
>>
> *For that we can thank Hang Nguyen for her work on this ! *
>
> I attached 2 docs that can give you more visibility on how we work in eXo
> vietnam
>
>
>> - marko
>>
>>
>>
>> _______________________________________________
>> gatein-dev mailing list
>> gatein-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/gatein-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/gatein-dev/attachments/20100910/f48941c5/attachment.html 