Some remarks in-line<br><br><div class="gmail_quote">2010/9/7 Marko Strukelj <span dir="ltr">&lt;<a href="mailto:marko.strukelj@gmail.com" target="_blank">marko.strukelj@gmail.com</a>&gt;</span><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">

<div><br></div>In order to run selenium tests with jboss I have to do the following modification to current pom.xml:<div><br></div><div><div>Index: testsuite/selenium-snifftests/pom.xml</div><div>===================================================================</div>


<div>--- testsuite/selenium-snifftests/pom.xml       (revision 4045)</div><div>+++ testsuite/selenium-snifftests/pom.xml       (working copy)</div><div>@@ -13,12 +13,12 @@</div><div><br></div><div>        &lt;properties&gt;</div>


<div>                &lt;org.selenium.server.version&gt;1.0.1&lt;/org.selenium.server.version&gt;</div><div>-               &lt;selenium.port&gt;4444&lt;/selenium.port&gt;</div><div>+               &lt;selenium.port&gt;8444&lt;/selenium.port&gt;</div>


<div>                &lt;selenium.browser&gt;firefox&lt;/selenium.browser&gt;</div><div>                &lt;selenium.timeout&gt;10000&lt;/selenium.timeout&gt;</div><div>                &lt;selenium.speed&gt;300&lt;/selenium.speed&gt;</div>


<div>                &lt;selenium.host&gt;localhost&lt;/selenium.host&gt;</div><div>-               &lt;org.selenium.maven-plugin.version&gt;1.0&lt;/org.selenium.maven-plugin.version&gt;</div><div>+               &lt;org.selenium.maven-plugin.version&gt;1.0.1&lt;/org.selenium.maven-plugin.version&gt;</div>

</div></blockquote><div>Could you explain why you need this maintenance version exactly ? <br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">

<div>
<div>        &lt;/properties&gt;</div><div><br></div><div>        &lt;dependencies&gt;</div></div><div><br></div><div><br></div><div>I&#39;ve been running these a lot the last two days. </div><div><br></div><div>They aren&#39;t very useful at the moment for doing pre-commit checks to catch any introduction of systemic issues primarily for three reasons.</div>


<div><br></div><div> - one, it takes 1 hour 40 mins on my laptop to run the suite. If I want a &#39;before my change&#39;, and &#39;after my change&#39; I have to run it twice so I can see a diff in test failures. The name &#39;snifftests&#39; gives an impression that this is a quick testsuite to be run before doing a commit :)</div>

</blockquote><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div> - two, at the moment many tests are failing - my last run: <span style="font-family: Arial; font-size: 15px; white-space: pre-wrap;">Tests run: 248, Failures: 74, Errors: 21, Skipped: 0</span></div>


<div>which makes it difficult to determine if some might be due to my code changes. The same tests sometimes fail with a &#39;Failure&#39;, and sometimes with &#39;Error&#39;, so the end report always looks different making finding effective differences a challenge even with a diff tool.</div>


<div><br></div><div> - three, some of the tests seem to fail randomly - more likely they are sensitive to initial conditions which can change if some other test fails to  do a proper cleanup. It can also happen by killing the test in the middle of execution. The situation is exacerbated by the fact that the tests are run in random order ...</div>


<div><br></div><div>Point number one could be addressed by making a set of simple to maintain tests that perform a few operations that touch many aspects of the portal. These would go into &#39;snifftests&#39; module. The exhaustive mass of other detailed tests - which are undoubtedly also a burden to maintain, would go into &#39;alltests&#39;.</div>

</blockquote><div> </div><div><div>In fact there are differences in Functional tests and sniff tests.<br>- Sniff tests have &quot;SNF&quot; in the name of the  test <br>- These tests can be run specifically by using &quot;<b>-Dtest=Test*SNF* </b>&quot; with your maven command<br>


- We will be implementing shortly what we call &quot;Smoke tests&quot; that will test in &lt;20 steps most functionalities of GateIn (these will be for testing installations on many different configurations)<br>- The name of the folder and how to run the different tests will be also be changed in SELEGEN 1.1 (expected in a week or so...see more improvements in the next points)<br>

</div></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div><br></div><div>Point number two could be addressed by creating some kind of final report that would throw &#39;failures&#39; and &#39;errors&#39; in a single set and sort it alphabetically.</div></blockquote><div>- This will become a reality with SELEGEN 1.1 (we will use a DOXIA maven framework to create html reports)...<br>

- see attached &quot;surefire-report.html&quot; for an old example (i&#39;ve been using it on my own for some time). The format is not perfect but better than nothing for now.<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">

<div><br></div><div>For point number three, a slow workaround is to run another packaging/pkg/mvn install.</div>
<div>Maybe we could add some cleanup mechanism, so each test can define the tear down sequence which would go through all the steps ignoring any errors. Removing test artifacts created during a test is already part of the existing tests. But if it was separated it could be run explicitly:</div>

</blockquote><div>- We have a problem using testsuites with our Maven approach, that is why each script is independent and cleans after himself.<br>- This also permits us to keep the artifacts of failed tests to better analyse what happened<br>

- we would need to pass a specific scenario between each test... (any ideas on this would be helpful) <br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">


<div><br></div><div>mvn -Pselenium-cleanup integration-test</div><div><br></div><div><br></div><div>Otherwise thumbs up for the sheer number of these tests, and the systematic approach ... </div></blockquote><div><b>For that we can thank Hang Nguyen for her work on this ! </b><br>

<br>I attached 2 docs that can give you more visibility on how we work in eXo vietnam<br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">

<div><br></div><font color="#888888"><div>- marko</div>
<div><br></div><div><br></div>
</font><br>_______________________________________________<br>
gatein-dev mailing list<br>
<a href="mailto:gatein-dev@lists.jboss.org" target="_blank">gatein-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/gatein-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/gatein-dev</a><br>
<br></blockquote></div><br>