[wildfly-dev] Pull request building improvements

[Jaikiran Pai]

I'll let Tomaz answer that question, but I'll add some points where I 
think TeamCity helped a lot recently when compared to Jenkins.

Just before 8.0.0.Alpha2 of WildFly was released, we noticed that our 
testsuite was in a very bad shape. Too many intermittent failures. 
Although, the intermittent failures weren't a new thing, the frequency 
and the number had both grown to an extent where we felt that we had to 
start looking into each of those tests and investigate the problems. I 
hadn't used TeamCity earlier but while looking into these tests, I 
decided to take a look at the instance maintained by Tomaz. From an user 
point of view, I found the following features extremely handy and in 
fact these features did help me with better investigating the failures 
and also not losing interest in trying to track down those test failures:

1) "Investigations" feature -  TeamCity has this feature called 
"investigations" which allows you to mark a (failed) test or an entire 
build for investigation. The investigation can be assigned to a specific 
user. Investigations can be auto resolved (the next time the build/test 
succeeds) or can be manually resolved after investigating that failure. 
This feature allowed me to keep track of a bunch of failing tests and 
monitor their resolution over time. This is one step between finding a 
failing test and creating a JIRA, since this intermediate step allowed 
me to spend some time on that test to really understand what needs to be 
fixed/changed for that test to pass. Once I knew what was needed, I 
could then either fix it or file the JIRA assigned to the relevant 
component/person.This also was one way of saying that this specific test 
failure is a "known issue which is being investigated on by person X". 
This way someone else can spend their time on some other test failure 
investigation.

Investigations also allow "notes" to be attached to them which allowed 
me to make a note of what I have investigated so far and what might be 
the issue.

2) Immediate report and logs of failed tests - Unlike Jenkins where you 
have to wait for the entire testsuite to finish (which can take a hour 
and a half) to know how many and which tests failed in that run, 
TeamCity shows the progress of the build and reports the number of 
failed tests at that point in time in the build. Furthermore, it shows 
logs and the failure details of such tests immediately and you don't 
have to wait for the run to complete. I found this extremely useful 
since I didn't have to wait for the entire run to complete. In the past, 
when I've seen intermittent failing tests on Jenkins, I haven't had the 
determination to try out certain things and check the results since the 
thought of having to wait for another hour and a half would just switch 
off my interest on that issue.

3) Inline logs/stacktrace - I'm not sure why I like this so much but I 
really do like this feature of TeamCity. This and #2 in themselves are 
the reasons which kept me interested in tracking down a majority of the 
failures. This specific feature is really simple. When a bunch of tests 
fail in a build, the build report page shows all those failed test names 
and also for each failed tests allows you to hide/show stacktrace inline 
under the testcase name (see this for example 
http://teamcity.cafe-babe.org/viewLog.html?buildId=5666&tab=buildResultsDiv&buildTypeId=pr 
- click on that test link and it will show up the logs inline and you 
can then hide the logs if you want to). This allowed me to view all 
those failed tests and their logs on the same page and hide whichever 
ones I didn't want to view. Of course, in Jenkins, you can view these 
logs on a separate page/tab for each failed test, but I find the 
TeamCity way, much more usable than the Jenkins way.

-Jaikiran

On Tuesday 02 July 2013 11:50 PM, Vojtech Juranek wrote:
> Hi,
>
>> it works very well.
>> In many cases much better than what we had with jenkins on lightning.
> could you be more specific please? (not going to try to persuade you to stay
> with Jenkins, just wondering what you see as Jenkins weak points and where is
> TC better)
>
> Thanks
> Vojta
>
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/wildfly-dev/attachments/20130703/0ef089c8/attachment.html