One of the other things was that Jenkins uses stats that are relative to the last good
build, which makes them useless for determining how often an intermittent test fails. With
TC we immediately know what the top abusers are.
On Jul 3, 2013, at 8:33 AM, Jaikiran Pai <jpai(a)redhat.com> wrote:
I'll let Tomaz answer that question, but I'll add some points
where I think TeamCity helped a lot recently when compared to Jenkins.
Just before 8.0.0.Alpha2 of WildFly was released, we noticed that our testsuite was in a
very bad shape. Too many intermittent failures. Although, the intermittent failures
weren't a new thing, the frequency and the number had both grown to an extent where we
felt that we had to start looking into each of those tests and investigate the problems. I
hadn't used TeamCity earlier but while looking into these tests, I decided to take a
look at the instance maintained by Tomaz. From an user point of view, I found the
following features extremely handy and in fact these features did help me with better
investigating the failures and also not losing interest in trying to track down those test
failures:
1) "Investigations" feature - TeamCity has this feature called
"investigations" which allows you to mark a (failed) test or an entire build for
investigation. The investigation can be assigned to a specific user. Investigations can be
auto resolved (the next time the build/test succeeds) or can be manually resolved after
investigating that failure. This feature allowed me to keep track of a bunch of failing
tests and monitor their resolution over time. This is one step between finding a failing
test and creating a JIRA, since this intermediate step allowed me to spend some time on
that test to really understand what needs to be fixed/changed for that test to pass. Once
I knew what was needed, I could then either fix it or file the JIRA assigned to the
relevant component/person. This also was one way of saying that this specific test failure
is a "known issue which is being investigated on by person X". This way someone
else can spend their time on some other test failure investigation.
Investigations also allow "notes" to be attached to them which allowed me to
make a note of what I have investigated so far and what might be the issue.
2) Immediate report and logs of failed tests - Unlike Jenkins where you have to wait for
the entire testsuite to finish (which can take a hour and a half) to know how many and
which tests failed in that run, TeamCity shows the progress of the build and reports the
number of failed tests at that point in time in the build. Furthermore, it shows logs and
the failure details of such tests immediately and you don't have to wait for the run
to complete. I found this extremely useful since I didn't have to wait for the entire
run to complete. In the past, when I've seen intermittent failing tests on Jenkins, I
haven't had the determination to try out certain things and check the results since
the thought of having to wait for another hour and a half would just switch off my
interest on that issue.
3) Inline logs/stacktrace - I'm not sure why I like this so much but I really do like
this feature of TeamCity. This and #2 in themselves are the reasons which kept me
interested in tracking down a majority of the failures. This specific feature is really
simple. When a bunch of tests fail in a build, the build report page shows all those
failed test names and also for each failed tests allows you to hide/show stacktrace inline
under the testcase name (see this for example
http://teamcity.cafe-babe.org/viewLog.html?buildId=5666&tab=buildResu...
- click on that test link and it will show up the logs inline and you can then hide the
logs if you want to). This allowed me to view all those failed tests and their logs on the
same page and hide whichever ones I didn't want to view. Of course, in Jenkins, you
can view these logs on a separate page/tab for each failed test, but I find the TeamCity
way, much more usable than the Jenkins way.
-Jaikiran
On Tuesday 02 July 2013 11:50 PM, Vojtech Juranek wrote:
> Hi,
>
>
>> it works very well.
>> In many cases much better than what we had with jenkins on lightning.
>>
> could you be more specific please? (not going to try to persuade you to stay
> with Jenkins, just wondering what you see as Jenkins weak points and where is
> TC better)
>
> Thanks
> Vojta
>
>
>
>
> _______________________________________________
> wildfly-dev mailing list
>
> wildfly-dev(a)lists.jboss.org
>
https://lists.jboss.org/mailman/listinfo/wildfly-dev
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat