[wildfly-dev] Pull request building improvements

Mon Jul 8 10:54:40 EDT 2013

Hi,

to add few other observations I had with Jenkins vs TeamCity.

Biggest issue from my point of view is that Jenkins has a big performance
issue when running more than one job same in concurrently on different
nodes.
What happens is that when you have lets say 5 same jobs (with different
parameters) running and first job would get to the end it would get
stuck/wait in "recording test results" until all jobs are done and then all
of them would finish in matter of minutes.

That causes big issue as build slaves don't do noting for in some cases
hours as they are waiting for other jobs to complete. And new jobs are
waiting for avalibale slaves, this result in hardware not being utilized as
much as it could be.

Related recorded issues for this
https://issues.jenkins-ci.org/browse/JENKINS-10234
https://issues.jenkins-ci.org/browse/JENKINS-10175

Issues are open for few years now and the answer to it was "this is a
feature not a bug"
everyone is recommending using xunit instead of junit plugin, but trick is
that xunit plugin has totally different semantics and as such does not
resolve our issue.

Other problem is general performance of building and amount it loads
hardware. In general it takes 10-15% more time to do same job on jenkins
then on TC when running on same hardware.

This are just few deep technical issues I have found.

In any case i think everyone agrees that UI in general is much better with
TC and also TC has nice integration with IDEs (eclipse, intellij, VS.NET)
which allow you to monitor/debug/start/review builds as they are executing.
That add a nice feature for developers to debug why their code is failing
just one one build slave for example.

--
tomaz

On Wed, Jul 3, 2013 at 3:33 PM, Jaikiran Pai <jpai at redhat.com> wrote:

>  I'll let Tomaz answer that question, but I'll add some points where I
> think TeamCity helped a lot recently when compared to Jenkins.
>
>  Just before 8.0.0.Alpha2 of WildFly was released, we noticed that our
> testsuite was in a very bad shape. Too many intermittent failures.
> Although, the intermittent failures weren't a new thing, the frequency and
> the number had both grown to an extent where we felt that we had to start
> looking into each of those tests and investigate the problems. I hadn't
> used TeamCity earlier but while looking into these tests, I decided to take
> a look at the instance maintained by Tomaz. From an user point of view, I
> found the following features extremely handy and in fact these features did
> help me with better investigating the failures and also not losing interest
> in trying to track down those test failures:
>
>  1) "Investigations" feature -  TeamCity has this feature called
> "investigations" which allows you to mark a (failed) test or an entire
> build for investigation. The investigation can be assigned to a specific
> user. Investigations can be auto resolved (the next time the build/test
> succeeds) or can be manually resolved after investigating that failure.
> This feature allowed me to keep track of a bunch of failing tests and
> monitor their resolution over time. This is one step between finding a
> failing test and creating a JIRA, since this intermediate step allowed me
> to spend some time on that test to really understand what needs to be
> fixed/changed for that test to pass. Once I knew what was needed, I could
> then either fix it or file the JIRA assigned to the relevant
> component/person. This also was one way of saying that this specific test
> failure is a "known issue which is being investigated on by person X". This
> way someone else can spend their time on some other test failure
> investigation.
>
>  Investigations also allow "notes" to be attached to them which allowed
> me to make a note of what I have investigated so far and what might be the
> issue.
>
>  2) Immediate report and logs of failed tests - Unlike Jenkins where you
> have to wait for the entire testsuite to finish (which can take a hour and
> a half) to know how many and which tests failed in that run, TeamCity shows
> the progress of the build and reports the number of failed tests at that
> point in time in the build. Furthermore, it shows logs and the failure
> details of such tests immediately and you don't have to wait for the run to
> complete. I found this extremely useful since I didn't have to wait for the
> entire run to complete. In the past, when I've seen intermittent failing
> tests on Jenkins, I haven't had the determination to try out certain things
> and check the results since the thought of having to wait for another hour
> and a half would just switch off my interest on that issue.
>
>  3) Inline logs/stacktrace - I'm not sure why I like this so much but I
> really do like this feature of TeamCity. This and #2 in themselves are the
> reasons which kept me interested in tracking down a majority of the
> failures. This specific feature is really simple. When a bunch of tests
> fail in a build, the build report page shows all those failed test names
> and also for each failed tests allows you to hide/show stacktrace inline
> under the testcase name (see this for example
> http://teamcity.cafe-babe.org/viewLog.html?buildId=5666&tab=buildResultsDiv&buildTypeId=pr- click on that test link and it will show up the logs inline and you can
> then hide the logs if you want to). This allowed me to view all those
> failed tests and their logs on the same page and hide whichever ones I
> didn't want to view. Of course, in Jenkins, you can view these logs on a
> separate page/tab for each failed test, but I find the TeamCity way, much
> more usable than the Jenkins way.
>
>  -Jaikiran
>
>
> On Tuesday 02 July 2013 11:50 PM, Vojtech Juranek wrote:
>
> Hi,
>
>
>  it works very well.
> In many cases much better than what we had with jenkins on lightning.
>
>  could you be more specific please? (not going to try to persuade you to stay
> with Jenkins, just wondering what you see as Jenkins weak points and where is
> TC better)
>
> Thanks
> Vojta
>
>
>
>  _______________________________________________
> wildfly-dev mailing listwildfly-dev at lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/wildfly-dev
>
>
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/wildfly-dev/attachments/20130708/ee7fa12b/attachment-0001.html 